Flaky tests that non-deterministically pass or fail are a major impediment to regression testing. Existing work on identifying flaky tests is mostly based on test re-executions, which may be a feasible solution for short-running unit tests, but can become prohibitively expensive for long-running integration and system tests. To alleviate the problem, we propose a novel heuristic for identifying flaky test results by analyzing run-length encodings of regression test histories. For flagging tests as flaky, the proposed heuristic solely requires knowledge about previous test results and neither requires test re-executions nor expensive model fitting. We apply this heuristic in the context of the CPAchecker project, which heavily relies on integration tests for regression testing.

Philipp Wendler

LMU Munich