Slides used by Alexander Perathoner in his presentation of the paper "On the Impact of Hitting System Resource Limits on Test Flakiness". Presentation took place on 14th April 2024 at the International Flaky Test Workshop, co-located at the International Conference on Software Engineering (ICSE 24) in Lisbon. The paper was written by F. Leinen, A. Perathoner and Prof. Dr. A. Pretschner.

1. On the Impact of Hitting System
Resource Limits on Test Flakiness
FTW 2024
April 14, 2024
Fabian Leinen Alexander Perathoner Alexander
Pretschner Technical University Munich

2. Motivation
Intuition: “PC under load provokes lags and flakiness”
Study Gap: resources are not stressed, just measured
Hypothesis: “Reaching CPU limits promotes timing-related
flakiness”
2

3. Approach: UI Tests & macOS
Weaker Resources
More expensive
3

4. Identifying Potentially Flaky Tests
4

5. Does Hitting Natural Resource Limits Impact Flakiness?
We measure ratio of:
- time at resource limit
- total runtime
in executions of flaky / non-flaky tests
5

6. Does Hitting Natural Resource Limits Impact Flakiness?
We measure ratio of:
- time at resource limit
- total runtime
in passing / failing executions of flaky tests
6

7. Gathering Data
select:repo
(type:file content:XCUIApplication lang:Swift)
(repo:has.tag(macos) OR
repo:has.tag(osx) OR
repo:has.topic(macOS))
236 repositories
7

8. 236 repositories
Manual filtering
20 projects
Gathering Data
6.5k
Sparkle
8.1k
Maccy
8

9. tum-i4/workflow-telemetry-action
measuring:
CPU, RAM, network, disk
GitHub
Workflow
Gathering Data in the GitHub CI
9

10. 10
Gathering Data in the GitHub CI

11. 11
Gathering Data in the GitHub CI

12. How Does the Dataset Look?
12

13. Does Hitting Natural Resource Limits Impact Flakiness?
13

14. Does Hitting Natural Resource Limits Impact Flakiness?
14

15. Result Discussion
Alternative explanations for RQ1:
- Flakiness linked to complexity of tests (size) rather
than resource limits?
No quantitative explanations for RQ2
15

16. What to Take Away From This
We analyzed the impact of resource limits:
1. on tests being flaky → they do spend more time at
limits
2. on flaky failures → they do not
16
Upgrade the server to boost stability
But don't expect it to magically ace tests

17. Call for Action
● More projects!
● More resources!
● Less overhead!
Remember the GitHub Action!
17

18. Root Causes of Flaky Tests
18