Datadog Logs Destination

Overview

Use Observability Pipelines’ Datadog Logs destination to send logs to Datadog Log Management. You can also use AWS PrivateLink to send logs from Observability Pipelines to Datadog.

Setup

Configure the Datadog Logs destination when you set up a pipeline. You can set up a pipeline in the UI, using the API, or with Terraform. The steps in this section are configured in the UI.

Optional settings

After you select the Datadog Logs destination in the pipeline UI, you can configure these optional settings.

Route logs to multiple Datadog organizations

You can route logs to multiple Datadog organizations. After routing has been set up, you can view metrics for the component or specific organizations to which you are routing logs.

Note: You can route up to 100 Datadog organizations.

Click Route to Multiple Organizations to set up routing to multiple Datadog organizations.

• If you haven’t added any organizations yet, enter organization details as described in the Add a Datadog organization section.
• If you have already added organizations, you can:
  • Click on an organization in the table to edit or delete it.
  • Use the search bar to find a specific organization by name, filter query, or Datadog site, and then select the organization to edit or delete it.
  • View metrics for an organization.
  • Click Add organization to route to another Datadog organization.

Note: If you don’t set up routing to multiple Datadog organizations, logs are routed to the default Datadog organization, which is the organization that is tied to the API key when you install the Worker.

Add an organization

1. Enter a name for the organization.
   • Note: The name does not have to correspond to the actual name of the Datadog organization.
2. Define a filter query. Only logs that match the specified filter query are sent to the organization. See Observability Pipelines Search Syntax for more information on writing filter queries.
3. Select the Datadog organization’s site.
4. Enter the identifier for the API key for that Datadog organization.
   • Note: Only enter the identifier for the API key. Do not enter the actual API key.
5. Click Save.

Buffering

Toggle the switch to enable Buffering Options. Enable a configurable buffer on your destination to ensure intermittent latency or an outage at the destination doesn’t create immediate backpressure, and allow events to continue to be ingested from your source. Disk buffers can also increase pipeline durability by writing data to disk, ensuring buffered data persists through a Worker restart. See Destination buffers for more information.

• If left unconfigured, your destination uses a memory buffer with a capacity of 500 events.
• To configure a buffer on your destination:
  1. Select the buffer type you want to set (Memory or Disk).
  2. Enter the buffer size and select the unit.
     1. Maximum memory buffer size is 128 GB.
     2. Maximum disk buffer size is 500 GB.
  3. In the Behavior on full buffer dropdown menu, select whether you want to block events or drop new events when the buffer is full.

Secret defaults

Note: If you entered identifiers for yours secrets and then choose to use environment variables, the environment variable is the identifier entered prepended with DD_OP. For example, if you entered PASSWORD_1 for the a password identifier, the environment variable for the password is DD_OP_PASSWORD_1.

View metrics for the component or specific organizations

You can view metrics at the component level or organization level.

Component-level metrics

To view metrics for the overall Datadog Logs destination:

1. Navigate to Observability Pipelines.
2. Select your pipeline.
3. Click the cog on the Datadog Logs destination and select View details.

Note: The Data dropped (intentional) metric shows logs that didn’t match any of the organizations’ filters.

Organization-level metrics

To view metrics for a specific Datadog organization:

1. Navigate to Observability Pipelines.
2. Select your pipeline.
3. Click the Datadog Logs destination so the organizations show up.

4. Click the organization you want to see metrics for.
5. Click View Health Metrics.

Alternatively, you can click on Review Configured Organizations in the Datadog Logs destination, and click the graph icon in the Metrics column for the organization you are interested in.

How the destination works

Event batching

A batch of events is flushed when one of these parameters is met. See event batching for more information.

Request a personalized demo