CircleCI’s Test Insights Takes Aim at ‘Flaky’ Tests

The ability to drill down into so-called “flaky” test problems prior to software deployment is among the new features CircleCI has released for its new Test Insights tool that is part of its continuous integration and continuous delivery (CI/CD) platform.

While using CircleCI’s Insights dashboard for CI/CD analytics in the past — as well as with any other testing process — users have very often faced difficulties with flaky test results during the development process, the company said. These flaky results involve random test failures with no apparent indication of why they failed. The developer, for example, runs the same test on the same code and might inexplicably get five fails and five passes. Fixing these common testing abnormalities typically requires domain knowledge and the expertise of an expert other than the developer to fix, thus slowing down the development process.

“Flaky tests cost time and money and domain knowledge to fix. Someone new wouldn’t be able to figure out what’s happening or identify the problems, and then they may lose trust in the entire test suite if it keeps failing randomly,” CircleCI product manager Dawit Gebregziabher told The New Stack. “So, what we’re bringing is a way to identify these tests quickly and then offer a path to fixing them or removing them from your test suite.”

With the flaky-test analysis feature that CircleCI’s Test Insights tool offers, the model detects the problem test component to then “get to a point where it is really learning the intricacies and the nuances of your system” to offer possible fixes for, or replacements of, the test, Gebregziabher said. “Tests with unpredictable behavior are also usually flagged before the testing process begins,” Gebregziabher said.

👁 CircleCI's test insights

According to CircleCI data, the average failed workflow wastes approximately 30 minutes. With more than 4,000 flaky tests identified per day, CircleCI users can now save more than 2,000 otherwise wasted hours per day, collectively, through eliminating flaky test failures, the company said.

👁 The results of CircleCI's tests

Other new features Test Insights offers that CircleCI communicated include:

• Improved velocity. Identification of the longest-running and most-failed tests for more rapid fixes.

• Lookback periods to understand performance changes over time. Changes in behavior and performance are pinpointed with a diagnosis of “the culprits” behind the changes in order to improve efficiency.

• Ability to make actionable improvements with insight trends from CircleCI’s Benchmarks database. Available datasets related to testing include throughput, duration, mean time to recovery, success rates and other metrics.

CircleCI’s ongoing development of its testing and debugging tools will continue to involve bringing observability into test performance, Gebregziabher said. After releasing the Insights dashboard about a year ago, the most requested feature has been test-performance data.

“The trend now is for developers to follow their code into production, so, you are responsible for testing your own code, as developers seek opportunities for test optimizations to improve developer productivity. We are thus trying to improve the efficiency of your pipelines with our testing tools,” Gebregziabher said. “We make use of test metadata, so whatever testing tool you use, we take the results and then parse, process and display in an easy and understandable way. Before this, folks were doing that themselves.”