Adaptive Sampling

Overview

Datadog adaptive sampling helps you capture more relevant traces while remaining close to a specific budget (ingested gigabytes).

When you choose adaptive sampling as your sampling strategy, you select a target monthly volume for trace ingestion for one or more services. This ensures that the consumption of these services matches the target volume at the end of the month, while keeping visibility over their endpoints.

Adaptive sampling uses remote configuration plus the existing sampling rules mechanisms to dynamically adjust sampling rates for each environment, service, and resource combination. This helps you to:

• Match your specified monthly budget.
• Ensure visibility for low-traffic services and endpoints by capturing at least one trace for each combination of service, resource, and environment every 5 minutes.

To configure services to use adaptive sampling, follow the instructions listed below.

Requirements

• Datadog Agent 7.53.0 or higher.
• Remote Configuration enabled for your Agent.
• APM Remote Configuration Write permission.
  Note: If you don’t have this permission, ask your Datadog admin to update your permissions from your organization settings.

Tracing library versions

The following table lists minimum SDK versions required for adaptive sampling:

Limitations

Limits apply to service and environment combinations depending on the sampling configuration:

Adaptive sampling

• The maximum number of service/env combinations onboarded to adaptive sampling is 800.
• Each unique service/env pair configured for adaptive sampling counts toward this limit.

Remote sampling configuration

• The maximum number of service/env combinations using remote sampling configuration is 1000.
• This limit applies regardless of how many sampling rules are defined for each service.
• Each unique service/env pair with remote sampling enabled counts once toward this limit.

Services using both adaptive and remote sampling

• If a service/env combination uses both adaptive sampling and remote sampling configuration, it counts once toward each respective limit (once toward the 800 adaptive sampling limit and once toward the 1000 remote sampling limit).
• It does not count twice within either individual limit.

Configure the adaptive sampling target

To get started with adaptive sampling, you first need to pick a target strategy setting:

• Set Budget by Number of APM Hosts: Configure a budget that is proportional to your allotment and the number of services onboarded (for example, based on the number of APM hosts)
• Set Budget by Data Volume: Configure a fixed target in gigabytes per month

To set the adaptive sampling monthly target:

1. Navigate to the Ingestion Control page.
2. Click Manage Adaptive Sampling Target.

3. Choose a target strategy for sampling:
   • Set budget by number of APM hosts
   • Set budget by data volume
4. Click Apply.

Set budget by number of APM hosts (Recommended)

Set your monthly target to a percentage of your allotment. At the bottom of the page, you are provided with a more complete explanation of how that percentage is converted in a monthly target volume. It is the product of:

• The global allotment: 150GB * number_of_APM_hosts + 50GB * number_of_traced_serverless_invocations (if applicable) + 10GB * number_of_fargate_tasks (if applicable)
• The percentage of allotment configured above
• The contribution of onboarded services to the allotment. For example, if the services onboarded to adaptive sampling contribute to 10% of the total ingested volume, Datadog targets 10% of the global allotment. This number increases with the number of services onboarded.

That monthly target volume is recomputed every 30 minutes.

Set budget by data volume

If you are configuring the first service to adaptive sampling, ensure that the ingestion volume target is >0. For subsequent services, you should increase the allocated budget after the new service is onboarded to account for the new volume.

Configure adaptive sampling for a service

View sampling rates by resource for a service

Before you configure adaptive sampling for a service, you can view the current ingestion configuration for the service.

To see configured sampling rates:

1. Navigate to the Ingestion Control page.
2. Click a service to view the Service Ingestion Summary.
3. View the table listing the applied sampling rates by resource of the service.

The table includes:

• Ingested bytes: Ingested bytes from spans of the service and resource.
• Downstream bytes: Ingested bytes from spans where the sampling decision starts from that service and resource, including downstream services.
• Configuration: Source of the resource sampling rate:
  • AUTOMATIC: Default head-based sampling mechanism from the Agent.
  • CONFIGURED LOCAL: Sampling rule set locally in the SDK.
  • CONFIGURED REMOTE: Remote sampling rule set from the Datadog UI.
  • ADAPTIVE REMOTE: Adaptive sampling rules set by Datadog.

Once a service is onboarded to adaptive sampling, the sampling rates are adjusted and recomputed every 10 minutes.

Onboard a service to adaptive sampling

To onboard a service to adaptive sampling:

1. Navigate to the Ingestion Control page.
2. Click a service to view the Service Ingestion Summary.
3. Click Manage Ingestion Rate.
4. Choose Datadog adaptive sampling rates as your service’s sampling strategy.
5. (Optional) Configure explicit sampling rates for specific resources, for which you would like to capture more (for example, 100% of GET /checkout endpoints) or less (for example, 0.1% of /health requests) data.
6. Click Apply.

The configuration should take effect in 5-6 minutes, the time it takes for Datadog to observe the service’s traffic pattern, compute, then apply the sampling rates. Resources that have been configured remotely display as Configured Remote in the Configuration column.

Permissions

By default, only users with the Datadog Admin role can modify adaptive sampling configurations or onboard services to adaptive sampling.

If your organization uses custom roles, assign your user to a custom role that includes APM Remote Configuration Write and APM Service Ingest Write permissions.

Restrict access

Use granular access controls to manage who can modify a service’s adaptive sampling configuration. You can restrict access based on roles, teams, or individual users.

To restrict access:

Note: Only users with the remote_config_write permission can restrict access to the adaptive sampling configuration of individual services.

1. Open the Permissions section in the Ingestion Control side panel of the service.

2. Click Restrict access.

3. Select the teams, roles, or users to grant access to.

4. Click Add.

Further reading

Request a personalized demo