Tandem Diabetes Care develops and manufactures insulin pumps for use by people with diabetes. These pumps deliver a precise amount of insulin directly into the body and are one of the few on the market that can be controlled by a mobile app. Our customers need our products and services to be reliable and performant at all times. We have little margin for error and need to ship the highest quality code we can.
As a member of the mobile infrastructure team, I focus on developer automation and tooling of Tandem’s mobile apps. One of the newer tools we’ve been using at Tandem is Codecov. We love that it can track and report on our code coverage in total and on pull requests.
Codecov helps us understand which parts of our codebase are being tested by our unit tests and also helps us keep track of how the percentage of our codebase being tested changes over time. For example, if a pull request would introduce a greater than 1% change to our total code coverage, merging it will be blocked until that is addressed.
For software teams, Sentry is essential for monitoring application code health. From Error tracking to Performance monitoring, developers can see clearer, solve quicker, and learn continuously about their applications — from frontend to backend.
The Problem
One of my focuses recently was finding ways to optimize our CI workflows. I noticed that for our iOS app, the Codecov step of our workflow was taking much longer than expected, so I decided to find out why and if there was anything we could do to improve it.
Apple’s integrated development environment Xcode collects code coverage data and can display it in the IDE for developers. It can also be exported as an
.xcresult file when using x
codebuild from the command line, like so:
xcrun xcodebuild test \
-project tconnect.xcodeproj \
-scheme tconnect \
-testPlan tconnect \
-destination "platform=iOS Simulator,name=iPhone 14" \
-derivedDataPath DerivedData \
-resultBundlePath artifacts/ResultBundle.xcresult
These
xcresult files are great and can be useful in lots of different ways, but like many things Apple, they can be difficult to use outside the Apple ecosystem. While they can be converted into a JSON representation using the
xccov binary included with Xcode, the resulting JSON is not in the standard coverage formats that Codecov can ingest. It also has been known to change without warning with new Xcode releases.
And so, in order for Codecov to use the coverage results from Xcode, they have to be converted into another format.
Codecov’s official GitHub Action can do the conversion, but it handles this conversion by analyzing the coverage for each file one by one, which can take up to a second for each file. This is a fine enough approach for some projects, but when working with a large codebase like ours, that can take quite some time.
Enter xcresultparser
The open source Swift tool
xcresultparser can parse
xcresult files and convert them into various other formats. One of these formats is
Cobertura XML, which Codecov supports.
The big advantage xcresultparser brings is that, because it is a compiled program and not a script, it can use multiple threads to do the conversion. This speeds up the conversion process immensely.
After running the
xcodebuild command above to generate the
xcresult file, we tell xcresultparser to convert it like so:
xcresultparser \
--output-format cobertura \
"artifacts/ResultBundle.xcresult" >"artifacts/coverage.xml"
And finally, we tell the Codecov GitHub Action to upload that XML file instead of the
xcresult file.
Results
So, just how much time savings are we seeing?
👁 Image
We run these builds in parallel, so the total real-time savings for each build is the delta of the longer packages unit tests build: about seven minutes. This might not seem like much, but when you factor in that we’re running these builds upwards of 20 times a day, it’s a considerable time (and cost) savings. That’s over two hours of total developer time saved per day, almost 12 hours per week!
Bonus
While implementing xcresultparser for our project, I learned that it can also print a summary of test results to the command line. For our packages unit test build, we run eight separate test suites in serial, so if a test fails further up the log, it can be difficult to find it.
Now, at the end of each test run, we print out a summary like so:
xcresultparser \
--output-format cli \
--failed-tests-only \
"artifacts/ResultBundle.xcresult"
This produces output that looks like this:
👁 Image
Now, it’s very easy for our developers to see which specific tests had failures just by looking at the end of the test log in CI.
Conclusion
Using xcresultparser has improved the lives of our developers quite a bit. And the fact that it is open source means that we as a developer community can help improve it for the benefit of ourselves and others.
For software teams, Sentry is essential for monitoring application code health. From Error tracking to Performance monitoring, developers can see clearer, solve quicker, and learn continuously about their applications — from frontend to backend.
