Pipeline Simulation

Overview

When you configure or edit a pipeline in Observability Pipelines, you often have to update filter queries, sampling rules, or Packs that transform your telemetry. These changes can impact downstream monitors, dashboards, and detection rules. Therefore, it’s important to test and validate how your changes affect your production data before you deploy those changes.

Use Pipeline Simulation to preview how your processors, volume control rules, and Packs modify your log data before deploying your pipeline configuration. This helps ensure your processors target the right data and modify your logs as expected. You can test your configuration with live logs sent through the pipeline or import your own sample data.

To use Pipeline Simulation:

1. Capture a snapshot of your data.
2. Run a simulation with that data for your processor configuration.
3. Inspect the data the processor group received and the resulting output. Confirm the processed data is what you expect. If it is not, update your processors and run another simulation.
4. After you validate that the processed data is what you expect, deploy the changes to production.

This document goes over each of those steps in detail.

The following example of Pipeline Simulation shows an unparsed log that a processor group receives (Entry) and the parsed output after the log is processed and tagged (Exit).

Permissions

Only users with the Observability Pipelines Live Capture Write permission can set up captures. Users with the Observability Pipelines Live Capture Read permission can only view the events that have already been captured. See Observability Pipelines Permissions for a list of permissions for Observability Pipelines assets.

Datadog Admins have read and write permissions by default. Standard users have only read permission by default. See Access Control for more information about default Datadog roles and how to create custom roles.

Add domains to firewall allowlist

If you are using a firewall, add these domains to the allowlist:

• api.
• obpipeline-intake.
• config.

Capture data for a pipeline simulation

Before running a simulation, you need to capture sample log data to test against. Pipeline Simulation lets you capture live data from an active pipeline, reuse previously captured samples, or import your own data.

1. Navigate to Observability Pipelines.
2. Set up a pipeline or select a pipeline and click Edit Pipeline on the top right side of the page.
3. On the pipeline’s edit page, you can expand a processor in a processor group to view its read-only configuration. To edit any processors, capture data, and run a simulation, click Edit on a processor group.

The status of a pipeline determines whether a simulation can be run with live data, imported data, or both.

Note: For inactive pipelines or pipelines in draft mode, you must import data the first time you run a simulation for the pipeline.

Capture live logs from your pipeline to run a simulation with production data

For active pipelines, Observability Pipelines automatically runs a Live Capture if a previously saved capture, whether from imported data or a live capture, is not available.

To run a capture for a specific set of data:

1. Click Live Capture.
2. (Optional) Enter a query to specify which events you want to capture. The default query is *.
   • Use a capture query, such as service:cloudtrail env:prod, to target a specific set of logs. This helps to distinguish between different sets of logs you use for your simulations. For example, you can capture a set of logs for custom app log processors and another for the Cloudflare Pack.
   • For more information on creating queries, see Search Syntax for Logs or Search Syntax for Metrics.
3. (Optional) Enter a capture duration (in seconds or minutes) for how long you want events to be captured.
   • Minimum duration (default, if no duration is specified): 30 seconds
   • Maximum duration: 300 seconds (5 minutes)
4. Click Capture.

Import your own sample data for a simulation

To import sample data for your pipeline simulation:

1. For an inactive or draft pipeline, if the pipeline does not have saved data, click Import Data. Otherwise, click the down arrow next to Live Capture and select Import Sample Data.
2. In the modal, drag and drop your sample data files or browse for them, or click the Paste tab to paste your sample data. Note: Sample data must be in a JSON or JSONL file. See Sample file import specifications for more details.
3. Click Confirm.

Sample file import specifications

JSON sample data format

When you import sample data in JSON format, the data must be:

• Structured as a set of JSON objects wrapped in an array
• Comma-separated between each log event

The following is an example set of logs in JSON you can import for a simulation:

[
 {
 "id": "abcd",
 "content": {
 "timestamp": "2026-05-27T19:41:46.609Z",
 "tags": [
 "env:prod",
 "service:web-store",
 "source:ruby"
 ],
 "host": "123",
 "service": "web-store",
 "message": "UnhandledInternalServiceError: Payment service reported 500 error code.",
 "attributes": {
 "transaction_id": "abc123",
 "shopist": {
 "webstore": {
 "merchant": {
 "plan_type": "m2m",
 "tier": "enterprise",
 "store_name": "ElectroNova"
 }
 },
 "customer": {
 "name": "Pam Jones",
 "location": "San Francisco, California"
 }
 },
 "level": "error",
 "logger": {
 "name": "ShoppingCartController"
 },
 "status": "error"
 }
 }
 }
]

JSONL sample data format

A JSONL file must contain multiple JSON objects, where each object is on its own line, separated by a newline.

The following is an example JSONL file you can import for a simulation:

{"name":"process_payload_ms.count","tags":{"host":"i-abcd123","version":"7.77.3"},"timestamp":"2026-05-08T20:16:10Z","interval_ms":10000,"kind":"incremental","counter":{"value":0.0}}
{"name":"max_eps.max_rate","tags":{"host":"i-efg123","version":"7.77.3"},"timestamp":"2026-05-08T20:16:10Z","interval_ms":10000,"kind":"absolute","gauge":{"value":200.0}}
{"name":"service_checks","tags":{"client_version":"5.8.3","host":"i-hijk123"},"timestamp":"2026-05-08T20:16:10Z","interval_ms":10000,"kind":"incremental","counter":{"value":0.0}}
{"name":"aggregated_context_by_type","tags":{"client_version":"5.8.1","xyz123":"macbook","metrics_type":"distribution"},"timestamp":"2026-05-08T20:16:10Z","interval_ms":10000,"kind":"incremental","counter":{"value":0.0}}

Reuse previously saved data for your simulation

After you use Live Capture to pull data from your pipeline, the results are stored for 72 hours in the Datadog platform and can be used for simulations. To use previously captured data for your pipeline simulation:

1. Click the down arrow next to Live Capture and select See Saved Samples.
2. Click Use as Sample Data for the live capture or previously imported data you want to use.

Run a pipeline simulation

After you have captured data for your pipeline simulation, edit your pipeline processors and run a simulation.

1. Navigate to Observability Pipelines.
2. Select your pipeline and click Edit Pipeline on the top right side of the page.
3. On the pipeline edit page, click Edit on a processor group to add or edit processors in your pipeline.
4. In the Processor Group dropdown menu on the left side of the Pipeline Simulation page, select the processor group with the processors you want to update and test. You can hover over Processor Group to see a snapshot of where the processors are in the pipeline.

5. Add and update your processors. You can update multiple processors for one simulation. Note: For a pipeline canvas, there is a limit of 25 processor groups and a total of 150 processors.
6. Choose a simulation scenario:
   • Run a simulation up to a specific processor group: Keep the End-to-End Preview toggle disabled to see the result of logs transformed by all processors in the current processor group and the prior processor groups. See Run a simulation up to a specific processor group for more information.
   • Run an end-to-end preview simulation: Enable the End-to-End Preview toggle to run a simulation through all processor groups and see the data that is sent to the destination. See End-to-end preview for more information.

7. Click Preview at the bottom of the processors panel to run the simulation.

8. After the simulation is complete, you can:
   • See the log data that the processor received and sent out.
   • Enter a free-text search query to find specific logs that were captured. The query searches all attributes and tags in the logs.
   • Use the All events dropdown menu to view events by status:
     • Modified only shows only modified events.
     • Unmodified only shows only unmodified events.
     • Rerouted only shows only logs rerouted to an overflow destination by the Quota processor.
     • Created only shows only:
       • Metrics generated from logs with the Generate Logs-Based Metrics processor.
       • Logs created by the Split Array processor.
     • Dropped only shows only dropped events.
   • Use the All telemetry dropdown menu to view Logs only or Metrics only events. Metrics only shows events only from the Generate Logs-Based Metrics processor.

9. Click Save at the bottom right side of the page to save your changes.
10. Click Back to Pipeline on the top right side of the page.
11. To deploy your updates to production, do one of the following:
    • Deploy updates using the UI: Click Next: Deploy Changes on the pipeline page.
    • Use the API to deploy changes: See Export and Import Configuration for information on exporting pipeline configurations to JSON.
    • Use Terraform to deploy changes: See Export and Import Configuration for information on exporting pipeline configurations to Terraform.

Note: For the Enrichment Table processor, only the Reference Table option supports Pipeline Simulation.

Simulation scenarios

When you run a simulation, you can choose to send your data to a specific processor group or through all processor groups to get an end-to-end preview.

Run a simulation up to a specific processor group

On the Pipeline Simulation page, you can choose which set of processors you want to test and validate. The simulation sends data through the previous processor groups and the selected group. For example, if you run a simulation for processor group 2, the processed data you see is the result from sending the logs through groups 1 and 2. Similarly, if you run a simulation for group 3, the result is after processing the log data through processor groups 1, 2, and 3.

End-to-end preview

When you toggle End-to-End Preview and run a simulation, the entry event shown is the log sent from the source before any processing is done. The exit log is the result after all processors have been applied and sent to the destination. This lets you visualize how a log is transformed after it goes through the entire pipeline.

Further reading

Request a personalized demo