Databricks (Zerobus) Destination

Overview

Use Observability Pipelines’ Databricks (Zerobus) destination to send logs to a Databricks Unity Catalog table. The destination streams logs to the Zerobus Ingest API and authenticates to Databricks with an OAuth service principal.

Prerequisites

Before you configure the Databricks (Zerobus) destination, you must:

• Set up a Unity Catalog schema and table that the Observability Pipelines Worker writes logs to.
• Set up a service principal that the Worker uses to authenticate to Databricks. The service principal needs permission to read and write to the table.

Set up a schema and table

The SQL examples in this section use the following placeholders:

Note: The GRANT commands must be run by a Databricks workspace admin.

In the Databricks workspace:

1. If you’re not a Databricks workspace admin, have an admin run the following command to grant your user permission to create a schema:

   GRANTCREATESCHEMAONCATALOG<CATALOG_NAME>TO<USER>;

2. Create the schema:

   CREATESCHEMAIFNOTEXISTS<CATALOG_NAME>.<SCHEMA_NAME>MANAGEDLOCATION'<YOUR_MANAGED_LOCATION>';

   • Note: MANAGED LOCATION is optional. See Databricks’ Create Schemas documentation for more information.

3. If you’re not an admin user, have an admin run the following command to grant your user permission to create a table on the schema:

   GRANTCREATETABLEONSCHEMA<CATALOG_NAME>.<SCHEMA_NAME>TO<USER>;

4. Run the following command to create the table that Observability Pipelines writes log data to:

   CREATETABLE<CATALOG_NAME>.<SCHEMA_NAME>.<TABLE_NAME>(hostSTRING,messageSTRING,serviceSTRING,source_typeSTRING,timestampTIMESTAMP);

   • See Databricks’ Create a Unity Catalog Managed Table documentation for more information.

The fully qualified table name is catalog.schema.table, for example main.obs_pipelines.apache_common_logs. This is the value you enter for Table Name when you set up the Observability Pipelines Databricks destination.

Set up a service principal

The Databricks Zerobus Ingest API uses OAuth authentication. When you create the service principal, the OAuth client secret is generated and the OAuth client ID is the service principal’s UUID.

To create a service principal:

1. In your Databricks workspace, navigate to User Settings > Identity and access > Service principals.
2. Click Add service principal.
3. After the service principal is created, generate an OAuth secret for it.
   • Take note of the service principal’s Application ID (client ID) and the OAuth client secret. You need both of them when you configure the Observability Pipelines Databricks destination.
4. Run this SQL in Databricks to grant the service principal access to the catalog, schema, and table. Replace <SERVICE_PRINCIPAL_UUID> with the service principal’s application ID from the previous step:

   GRANTUSECATALOGONCATALOG<CATALOG_NAME>TO<SERVICE_PRINCIPAL_UUID>;GRANTUSESCHEMAONSCHEMA<CATALOG_NAME>.<SCHEMA_NAME>TO<SERVICE_PRINCIPAL_UUID>;GRANTSELECT,MODIFYONTABLE<CATALOG_NAME>.<SCHEMA_NAME>.<TABLE_NAME>TO<SERVICE_PRINCIPAL_UUID>;

See Databricks’ Add service principals to your account and Grant permissions on an object documentation for more information.

Setup

Configure the Databricks (Zerobus) destination when you set up a pipeline. You can set up a pipeline in the UI, using the API, or with Terraform. The steps in this section are configured in the UI.

Note: Log fields that are not present in the table schema are dropped. For example, if a log has the fields id, name, and host, and the table schema only contains the columns name and host, then the id field is dropped and not written to the table.

After you select the Databricks (Zerobus) destination in the pipeline UI:

1. Enter the Ingestion Endpoint for your Databricks workspace, such as https://<workspace_id>.zerobus.<region>.cloud.databricks.com. The Worker sends logs to this endpoint.
2. Enter the Table Name in the format catalog.schema.table, such as main.obs_pipelines.apache_common_logs.
3. Enter the Unity Catalog Endpoint for your Databricks workspace, such as https://<workspace>.cloud.databricks.com. The Worker uses this endpoint to read the table’s schema.
4. In the Auth - Client ID field, enter the application ID of the service principal, such as abcdefgh-1234-5678-abcd-ef0123456789.
5. In the Auth - Client Secret field, enter the identifier for your OAuth client secret. If you leave it blank, the default is used.

Optional settings

Buffering

Toggle the switch to enable Buffering Options. Enable a configurable buffer on your destination to ensure intermittent latency or an outage at the destination doesn’t create immediate backpressure, and allow events to continue to be ingested from your source. Disk buffers can also increase pipeline durability by writing data to disk, ensuring buffered data persists through a Worker restart. See Destination buffers for more information.

• If left unconfigured, your destination uses a memory buffer with a capacity of 500 events.
• To configure a buffer on your destination:
  1. Select the buffer type you want to set (Memory or Disk).
  2. Enter the buffer size and select the unit.
     1. Maximum memory buffer size is 128 GB.
     2. Maximum disk buffer size is 500 GB.
  3. In the Behavior on full buffer dropdown menu, select whether you want to block events or drop new events when the buffer is full.

Convert string timestamps to timestamp format

If your logs have timestamps in string format and your Databricks table has a timestamp column declared as a TIMESTAMP type, you must convert the string to timestamp format before sending logs to the Databricks (Zerobus) destination. Databricks (Zerobus) can only convert the timestamp format to its TIMESTAMP type.

If you do not convert the string timestamp, the Worker throws an error similar to:

Protobuf encoding failed: Error converting timestamp field: Can't convert '2012-04-23T10[41]15Z' to i64: invalid digit found in string

To convert timestamps in string format to timestamp format:

1. Add a Custom Processor to your pipeline.
2. Add a function with the following custom script:

   .timestamp = parse_timestamp!(.timestamp, format: "%+")
   

   See parse_timestamp for more information.

Secret defaults

These are the defaults used for secret identifiers and environment variables.

How the destination works

Event batching

A batch of events is flushed when one of these parameters is met. See event batching for more information.

Request a personalized demo