Code Coverage Carryforward

Overview

In large repositories, CI pipelines often run only the subset of test jobs that are relevant to the files changed in a pull request. For example, a PR that touches only frontend code may skip backend unit tests and integration tests. When that happens, only some of the coverage reports are uploaded for the commit, and the total coverage metric drops because the data from the skipped jobs is missing.

Carryforward solves this problem. When a coverage report is missing for a commit, Datadog reuses the most recent matching report from an ancestor commit so that total coverage reflects what the result would have been if every CI job had run.

Carryforward works at the flag level. For each flag that has no uploaded report on the target commit, Datadog looks back through the commit’s ancestors and reuses the latest report tagged with that flag.

Prerequisites

Carryforward builds on top of Code Coverage flags. Do the following before you enable carryforward:

• Tag every coverage report with one or more flags using the --flags option on datadog-ci coverage upload. See Add flags to coverage reports.
• Use a stable set of flags across CI runs. The same flags should be produced whenever the corresponding tests or tested code change. If your CI distributes tests randomly across workers or generates a different set of reports on each run, carryforward results are not meaningful.

For the consistency requirement, assign one CI job per coverage report. For example, use separate jobs for unit tests, backend integration tests, and UI tests, with each job tagging its report with a stable flag.

Enable carryforward

Carryforward is configured in the code-coverage.datadog.yml file at the root of your repository.

Enable carryforward for all flags

Set the top-level carryforward field to true to enable carryforward for every flag in the repository:

schema-version:v1carryforward:true

Enable carryforward for specific flags

To enable carryforward for only a subset of flags, leave the top-level carryforward field unset (or false) and opt in per flag using the flags map:

schema-version:v1flags:unit-tests:carryforward:trueintegration-tests:carryforward:true

Disable carryforward for specific flags

When carryforward is enabled at the repository level, you can disable it for individual flags. This is useful for flaky test suites or report types where reusing ancestor data is not appropriate:

schema-version:v1carryforward:trueflags:nightly-tests:carryforward:false

In this example, carryforward applies to every flag except nightly-tests.

How carryforward works

When a new commit is pushed to a repository where carryforward is enabled, Datadog follows these steps:

1. Looks back through the commit’s ancestors.
2. For every flag with carryforward enabled, finds the most recent report tagged with that flag.
3. Reuses the coverage data from those ancestor reports for the new commit, until a fresh report for the flag is uploaded.

Carried-forward data is automatically replaced as soon as a real report is uploaded for the same flag on the new commit. The merged total coverage on the commit always reflects a combination of the following:

• Fresh reports uploaded for the commit
• Carried-forward reports for flags where no fresh report was uploaded

Use with PR Gates

While carryforward is in Preview, Datadog recommends starting with a non-blocking PR Gate to observe how carryforward affects evaluation before enforcing it as a merge requirement.

Further reading

Request a personalized demo