Troubleshooting

Overview

If you experience issues setting up or running Datadog Experiments, use this page to troubleshoot. If you continue to have trouble, contact Datadog support.

Experiment results do not appear

If experiment results are missing after you launch an experiment, start by checking whether the experiment is assigning users. Then, navigate to the appropriate troubleshooting step.

Step 1: Confirm the experiment is assigning users

On the Experiments page, select your experiment to open its detail page. Hover over the values of the metric scorecard:

If the subject assignment count for each variant is zero, go to Step 2 to debug traffic.
If the subject assignment count is greater than zero, but the metric values are zero, skip to Step 3.

In the following example, the subject assignment count is 12,427 for Variant A and 12,573 for Variant B.

If your metric scorecard shows N/A for metric values or subject assignment counts, this means the analysis has not run yet. Wait for it to run, then continue with the troubleshooting steps as needed.

Step 2: Confirm the experiment is receiving traffic

Verify that your feature flag is enabled and evaluates in the correct environment. Then, confirm that traffic reaches the experiment’s targeting rule.

On the Experiments page, select your experiment to open its detail page. Hover over the Feature flag label (for example, new-product-photos) next to the experiment status.
Note the Environment where the experiment is running, then click Go to Flag.
On the Feature Flags page, select the correct environment tab and confirm that the flag is Enabled. If the flag is disabled, enable it before proceeding.
In the Real-time metric overview section, confirm that the bar chart shows exposure events.

Based on what you see in the Real-time metric overview, follow the appropriate path:

The flag is not receiving traffic

If the Real-time metric overview section shows no exposure events, the flag is not receiving traffic from your application.

Confirm the flag is enabled in the environment where your application runs. You can manage environments on the Environments page. See the Getting Started with Feature Flags guide for details on environments.

After you enable the flag, check the Real-time metric overview for incoming exposure events. Then, return to Step 1 to verify that the subject assignment count is increasing.

The flag is receiving traffic but experiment assignments are zero

If the flag shows exposures but the metric scorecard shows zero assignments, traffic is not reaching the experiment’s targeting rule.

The Targeting Rules & Rollouts section displays a list of targeting rules that the flag evaluates from top to bottom. Rules above the experiment’s targeting rule, such as rules that exclude internal users or specific organizations, can capture traffic before it reaches the experiment.

Check the following and edit the targeting rule and traffic exposure as needed:

Targeting rule order: Are targeting rules above the experiment capturing traffic before it reaches the experiment rule?
Targeting rule filters: Does incoming traffic match the filters in the experiment’s targeting rule?
Traffic exposure: Is the traffic exposure for the targeting rule set correctly?

After making the necessary changes, return to Step 1 to verify that the subject assignment count is increasing.

Step 3: Confirm metric events are firing

If users are assigned to the experiment but the metric values are missing, check whether the assigned users have associated metric events.

Work through the following checks in order. Each builds on the previous one, so continue to the next if the issue persists.

A metric event must meet two criteria for Datadog to include it in experiment results:

The event must come from a user with at least one experiment exposure event.
The event must occur after the user's first experiment exposure.

If the issue persists after completing all checks, contact the Datadog support team.

URL: https://docs.datadoghq.com/experiments/troubleshooting/