Reference Tables

Overview

Reference Tables allow you to combine custom metadata with information already in Datadog. You can define new entities like customer details, service names and information, or IP addresses by uploading a CSV file containing a table of information. The entities are represented by a primary key in a Reference Table and the associated metadata.

For example, you can:

Enrich logs and security data for faster investigations: Correlate logs, traces, and security events with up-to-date business context—such as customer names, account owners, threat intelligence, or error code descriptions—to accelerate troubleshooting and analysis.
Segment users and resources for targeted analytics and cost management: Group users, customers, or cloud resources into meaningful segments (like user tiers, teams, or business units) for deeper product analytics and precise cost attribution using tools like Tag Pipelines.
Enhance data for advanced querying and reporting: Join external data from Reference Tables in Sheets, DDSQL Editor, or Notebooks to perform complex queries, aggregations, and build custom reports without technical expertise.

Create a Reference Table

Datadog supports the following data sources, including integrations and manual CSV upload:

Click New Reference Table +, then upload a CSV file, name the appropriate columns, and define the primary key for lookups.

Note: The manual CSV upload method supports files up to 4MB.

Create reference tables programmatically using the Datadog API.

Use the Create Reference Table endpoint to create reference tables from cloud storage or local files.

For cloud storage sources (S3, Azure, GCS), provide access_details in file_metadata pointing to a CSV file in cloud storage.
For local files, call POST /api/latest/reference-tables/uploads to get an upload ID and upload your CSV data. Then, call the Create Reference Table endpoint with the upload_id in file_metadata.

Note: The API supports the same file size limits as cloud storage uploads. See Reference Table limits for details.

This Reference Table can be used to add additional attributes to logs with the Lookup Processor.

Validation rules

Reference Table names and column headers are validated using the following naming conventions and automatically updated or normalized, if necessary.

Rule	Normalization
Names and headers cannot be duplicated.	Duplicated names are enumerated. For example, if `fileid` is used twice as a name, the first instance becomes `fileid1` and the second instance becomes `fileid2`. If a name or header is enumerated and it exceeds the 56 characters, it is rejected and needs to be renamed.
Names and headers cannot contain uppercase letters.	Names with uppercase letters are converted to lowercase. This conversion may result in duplicate names, which are then enumerated. For example, `Fileid` and `FileID` both become `fileid` and are enumerated to `fileid1` and `fileid2` respectively.
Names and headers cannot contain spaces.	Spaces other than leading and trailing spaces are replaced with underscore `_` characters. Leading and trailing spaces are removed. For example, `customer names` is replaced with `customer_names`.
Names and headers must start with a lowercase letter.	Uppercase characters are converted to lowercase. Non-letter leading characters are removed. For example, `23Two_three` becomes `two_three`.
Names and headers support only lowercase letters, numbers, and the `_` character.	Unsupported characters are replaced with the underscore `_` character, unless it breaks one of the rules above. In that case, the unsupported characters are normalized by the respective rule.
Names and headers must be 56 characters or less.	No normalization is done. Names and headers that have more than 56 characters are rejected and need to be renamed.

Modify a Reference Table

To modify an existing Reference Table with new data, select a table and click Update Config on the top right corner. The selected CSV is upserted into the table, meaning that:

All existing rows with the same primary key are updated
All new rows are added
All old rows that are not in the new file are deleted

Once the table is saved, the upserted rows are processed asynchronously and updated in the preview. It may take up to 10 minutes for the update to complete.

Export a Reference Table

To export a Reference Table, select a table and click Query in DDSQL Editor. From there, you can use the DDSQL Editor to export to CSV, Dashboard, and more.

Delete a Reference Table

To delete a Reference Table, select a table, click the gear icon in the top right corner, and then click Delete Table. The table and all associated rows is deleted.

If there is a Lookup Processor using a Reference Table for Log enrichment, then the enrichment stops. It may take up to 10 minutes for the enrichment to stop.

Monitor Reference Table Activity

You can monitor reference table activity with Audit Trail or Change Events. To view the audit trail and change events for a specific reference table, select the table and click the Settings icon next to Update Config. You need org management permissions to view the audit trail.

Audit Trail

Use the audit trail for reference tables to track user-triggered actions. Audit trail events are sent when a user initially uploads or imports a CSV file, or when a user creates, modifies, or deletes a reference table.

The reference_table_file Asset Type displays import/upload events and the reference_table Asset Type displays reference table events. The audit trail provides observability into the content of a reference table.

Change Events

Use change events for reference tables to track automated or user-triggered actions. They are sent when a cloud file is imported from a user or automatic refresh. (Uploading a local file does not generate a change event.) While events can track user-triggered actions, they are mainly used to track triggered imports when a reference table automatically pulls a new CSV file.

Events contain information about the success status, path, and table name of the import. If an error occurs, information about the error type is provided.

Alerting

To be alerted on errors encountered during imports, use Event Monitors for reference table change events. Reference table change events are sent from the reference_tables source.

You can create monitors from the Monitors tab, or click on the Settings icon next to New Reference Table + to generate a pre-filled monitor.

Reference Table limits

A reference table can have up to 50 columns
A single row cannot be larger than 500KiB
The size of a reference table file uploaded through the UI can be up to 200 MB
The size of a reference table file uploaded through a cloud bucket file can be up to 200 MB
The size of a reference table file uploaded through an integration can be up to 200 MB
You can have up to 100 reference tables per organization

Reach out to support if you have a use case that exceeds these limits.

Automatic update frequency

Reference Tables can be updated automatically, depending on the data source:

Cloud file storage (Amazon S3, Azure Storage, Google Cloud Storage): Every 5 minutes
Integrations: Every hour
CSV manual uploads: Automatic updates are not supported

Permissions

Role based access

To view Reference Tables, users require the reference_tables_read permission. To create or modify Reference Tables, users require the reference_tables_write permission.

For more information on permissions, see the RBAC documentation.

Granular access controls

Restrict access to individual tables by specifying a list of teams, roles, or users that are allowed to view or edit them.

Click on a table to open its detail page.
Click the cog icon in the upper-right corner.
Select Permissions from the menu.
Click Restrict Access.
Use the dropdown to select one or more teams, roles, or users.
Click Add.
Select either Editor or Viewer.
Click Save to apply changes.

URL: https://docs.datadoghq.com/reference_tables/