Remote Configuration

Overview

Remote Configuration is a Datadog capability that allows you to remotely configure and change the behavior of select product features in Datadog components such as Agents, SDKs, and Observability Pipelines Workers deployed in your infrastructure. Use Remote Configuration to apply configurations to Datadog components in your environment on demand, decreasing management costs, reducing friction between teams, and accelerating issue resolution times.

For Datadog security products, App and API Protection and Workload Protection, Remote Configuration-enabled Agents and compatible SDKs provide real-time security updates and responses, enhancing security posture for your applications and cloud infrastructure.

How it works

When Remote Configuration is enabled, Datadog components such as the Datadog Agent securely poll the configured Datadog site for configuration changes that are ready to apply. Pending changes are then automatically applied to Datadog components. For example, after you submit configuration changes in the Datadog UI for a Remote Configuration-enabled product feature, the changes are stored in Datadog.

The following diagram illustrates how Remote Configuration works:

1. You configure select product features in the Datadog UI.
2. The product feature configurations are securely stored within Datadog.
3. Remote-configuration enabled Datadog components in your environments securely poll, receive, and automatically apply configuration updates from Datadog. Tracing libraries that are deployed in your environments communicate with Agents to request and receive configuration updates from Datadog instead of directly polling Datadog.

Supported environments

Remote Configuration works in environments where supported Datadog components are deployed. Supported Datadog components include:

• Agents
• Tracers (indirectly)
• Observability Pipeline Workers
• Private action runners and serverless container cloud services such as AWS Fargate.

Remote Configuration does not support serverless container managed apps, such as AWS App Runner, Azure Container Apps, Google Cloud Run; or functions deployed with container packaging, such as AWS Lambda, Azure Functions, and Google Cloud Functions.

Supported products and features

The following products and features are supported with Remote Configuration.

Fleet Automation

• Send flares directly from the Datadog site. Seamlessly troubleshoot the Datadog Agent without directly accessing the host.

• Upgrade your Agents.

App and API Protection (AAP)

• 1-click AAP activation: Enable AAP in 1-click from the Datadog UI.

• In-App attack patterns updates: Receive the newest Web Application Firewall (WAF) attack patterns automatically as Datadog releases them, following newly disclosed vulnerabilities or attack vectors.

• Protect: Block attackers’ IPs, authenticated users, and suspicious requests that are flagged in AAP Security Signals and Traces temporarily or permanently through the Datadog UI.

Application Performance Monitoring (APM)

• Configuration at runtime: Change a service’s trace sampling rate, Log Injection enablement, and HTTP header tags from within the Catalog UI, without having to restart the service. Read Configuration at Runtime for more information.

• Remotely set Agent sampling rate: Remotely configure the Datadog Agent to change its trace sampling rates and set rules to scale your organization’s trace ingestion according to your needs, without needing to restart your Datadog Agent.

Dynamic Instrumentation

• Send critical metrics, traces, and logs from your live applications with no code changes.

Workload Protection

• Automatic default Agent rule updates: Automatically receive and update the default Agent rules maintained by Datadog as new Agent detections and enhancements are released. See Setting Up Workload Protection for more information.

• Automatic deployment of custom Agent rules: Automatically deploy your custom Agent rules to designated hosts (all hosts or a defined subset of hosts).

Observability Pipelines

• Remotely deploy and update Observability Pipelines Workers (OPW): Build and edit pipelines in the Datadog UI, rolling out your configuration changes to OPW instances running in your environment.

Autoscaling

• Remotely manage autoscaling cluster and workload scaling configurations for your containerized environments. See Autoscaling for more information.

Private action runner

• Run Datadog workflows and apps that interact with services hosted on your private network without exposing your services to the public internet. For more information, see Private Actions.

Feature Flags

• Deliver flag configurations (targeting and assignment rules) to server-side SDKs for synchronous variant assignment based on evaluation context. See Feature Flags for more information.

Security considerations

Datadog implements the following safeguards to protect the confidentiality, integrity, and availability of configurations received and applied by your Datadog components:

• Remote Configuration enabled Datadog components deployed in your infrastructure request configurations from Datadog.
  Some components like private action runners are always remote configuration enabled. Others, like Agents, can be enabled or disabled using in-disk configuration options.
• Datadog never sends configuration changes unless requested by Datadog components. If it does send configuration changes, Datadog only sends changes relevant to the requesting component.
• The configuration requests are initiated from your infrastructure to Datadog over HTTPS (port 443). This is the same port that the Agent uses by default to send observability data.
• The communication between your datadog components and Datadog is encrypted using HTTPS and is authenticated and authorized using your Datadog API key except in the case of private action runners where a JWT token is used instead.
• Only users with the api_keys_write permission are authorized to enable or disable Remote Configuration capability on API keys and use the supported product features.
• Your configuration changes submitted through the Datadog UI are signed and validated by the requesting Datadog component, verifying the integrity of the configuration.
• When Remote Configuration is enabled, components periodically perform a connectivity test to Datadog over TLS on port 8042/TCP. This is a raw TLS connection rather than HTTPS. This test is used for protocol development; it includes no customer data and no customer-identifying information. This connection is used only while Remote Configuration is enabled and can be safely blocked without affecting functionality.

Role-based access

Enabling Remote Configuration impacts the following products. Each product defines a set of role-based access controls that need to be granted to their users. For general information on access management, see Access Control.

Enable Remote Configuration

In most cases, Remote Configuration is enabled by default for your organization. You can check if Remote Configuration is enabled on your organization from the Remote Configuration settings page. If you need to enable it:

1. Ensure your RBAC permissions include org_management, so you can enable Remote Configuration for your organization.
2. From your Organization Settings page, enable Remote Configuration. This enables Datadog components across your organization to receive configurations from Datadog.
3. Follow the product-specific configuration guidance below to finish setting up Remote Configuration.

Product-specific configuration

Consult the documentation below for instructions specific to the product you’re configuring.

Best practices

Datadog Audit Trail

Use Datadog Audit Trail to monitor organization access and Remote Configuration enabled events. Audit Trail allows your administrators and security teams to track the creation, deletion, and modification of Datadog API and application keys. After Audit Trail is configured, you can view events related to Remote Configuration enabled features and who has requested these changes. Audit Trail allows you to reconstruct sequences of events, and establish robust Datadog monitoring for Remote Configuration.

Monitors

Configure monitors to receive notifications when an event of interest is encountered.

Opting out of Remote Configuration

Instead of disabling Remote Configuration globally, Datadog recommends opting out for specific Datadog products. For more information, see the documentation for the relevant product.

Further Reading

Request a personalized demo