Pipelines and Processors

Overview

Datadog Event Management Pipelines allow you to process and manage events from various sources efficiently. With pipelines, you can apply various processing rules and filters to customize how events are handled. Pipelines make it easier to manage large volumes of incoming events in a structured way.

Use pipelines to:

• Enrich events: Pipelines can add additional context or information to events, such as tagging, adding custom attributes, or correlating events with relevant metadata to make them more informative and actionable.
• Normalize event tags: You can set up rules to remap tags so that your events all have standardized tags.
• Parse messages and attributes as tags: Create custom grok rules to parse the full message or a specific attribute of your raw event. Translate the context into trackable tags and attributes that can be referenced during investigation.

Datadog Event Management Pipelines help organizations simplify their monitoring processes, enhance the clarity of their operational insights, and ultimately respond more effectively to incidents and alerts.

How it works

With pipelines, events are parsed and enriched by chaining them sequentially through processors. This extracts meaningful information or attributes from semi-structured text. Each event that comes through the pipelines is tested against every pipeline filter. If it matches a filter, then all the processors are applied sequentially before moving to the next pipeline.

Pipelines and processors can be applied to all events and can be configured in the Event Management Pipelines.

Create a pipeline

Create an Pipeline to filter to the events that you are interested in, for example, a source or a tag.

1. Navigate to Event Management Pipelines in Datadog.
2. Click Add a Pipeline.
3. Choose a filter from the dropdown menu or create your own filter query in the Event Management Explorer by selecting the </> icon. Use the filter to apply pipeline processors to specific events. Note: The pipeline filtering is applied before any of the pipeline’s processors. You cannot filter on an attribute that is extracted in the pipeline itself.
4. Name the pipeline.
5. Click Create.

Add a processor

You can add processors after you create a pipeline. The processors available are:

• Aggregation Key Processor
• Arithmetic Processor
• Date Remapper
• Category Processor
• Grok Parser
• Lookup Processor
• Remapper
• Service Remapper
• Status Remapper
• String Builder Processor

Pipeline permissions

Access to Event Management Pipelines is controlled by two permissions:

• events_write_pipelines: Allows you to create, edit, and delete pipelines, including modifying pipeline attributes such as filter queries and pipeline ordering.
• events_write_processors: Allows you to create, edit, and delete processors within a pipeline.

Administrators can assign these permissions to roles using Role Based Access Control to control who can modify pipeline and processor configurations.

Further reading

Request a personalized demo