VOOZH about

URL: https://docs.datadoghq.com/observability_pipelines/sources/logstash/

⇱ Logstash Source

For AI agents: A markdown version of this page is available at https://docs.datadoghq.com/observability_pipelines/sources/logstash.md. A documentation index is available at /llms.txt.

Logstash Source

This product is not supported for your selected Datadog site. ().
Available for:

Logs

Overview

Use Observability Pipelines’ Logstash source to receive logs from your Logstash agent.

You can also use the Logstash source to send logs to Observability Pipelines using Filebeat.

Prerequisites

To use Observability Pipelines’ Logstash source, you need the following information available:

  • Logstash address, such as 0.0.0.0:8088. The Observability Pipelines Worker listens on this bind address to receive logs from your applications. Later on, you configure your applications to send logs to this address.
  • The appropriate TLS certificates and the password you used to create your private key, if your forwarders are globally configured to enable SSL.

Setup

Set up this source when you set up a pipeline. You can set up a pipeline in the UI, using the API, or with Terraform. The instructions in this section are for setting up the source in the UI.

For Secrets Management: Only enter the identifiers for the Logstash address and, if applicable, the TLS key pass. Do not enter the actual values.
If you enter secret identifiers and then choose to use environment variables, the environment variable is the identifier entered and prepended with DD_OP_. For example, if you entered PASSWORD_1 for a password identifier, the environment variable for that password is DD_OP_PASSWORD_1.

After you select the Logstash source in the pipeline UI, enter the identifier for your Logstash address. If you leave it blank, the default is used.

Optional TLS settings

Toggle the switch to Enable TLS.

  • If you are using Secrets Management, enter the identifier for the key pass. See Set secrets for the default used if the field is left blank.
  • The following certificate and key files are required:
    • Server Certificate Path: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
    • CA Certificate Path: The path to the certificate file that is your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
    • Private Key Path: The path to the .key private key file that belongs to your Server Certificate Path in DER, PEM, or CRT (PKCS #8) format.
    • Notes:
      • The configuration data directory /var/lib/observability-pipelines-worker/config/ is automatically appended to the file paths. See Advanced Worker Configurations for more information.
      • The file must be readable by the observability-pipelines-worker group and user.
  • (Optional) Toggle Verify certificate to require connecting clients to present a valid client certificate. This enforces mutual TLS (mTLS), where the Worker verifies the identity of each connecting client.

Secret defaults

These are the defaults used for secret identifiers and environment variables.

  • Logstash address identifier:
    • References the address on which the Observability Pipelines Worker listens for incoming log messages.
    • The default identifier is SOURCE_LOGSTASH_ADDRESS.
  • Logstash TLS passphrase identifier (when TLS is enabled):
    • The default identifier is SOURCE_LOGSTASH_KEY_PASS.
  • Logstash address and port:
    • The Observability Pipelines Worker listens on this address, such as 0.0.0.0:9997, for incoming log messages.
    • The default environment variable is DD_OP_SOURCE_LOGSTASH_ADDRESS
  • Logstash TLS passphrase:
    • The default environment variable is DD_OP_SOURCE_LOGSTASH_KEY_PASS.

Send logs to the Observability Pipelines Worker over Logstash

To configure Logstash to send logs to the Observability Pipelines Worker, use the following output configuration:

output {
 http {
 url => "http://127.0.0.1:9997"
 http_method => "post"
 format => "json"
 }
}

Note: Logstash requires SSL to be configured.