VOOZH about

URL: https://docs.datadoghq.com/data_observability/quality_monitoring/data_warehouses/bigquery/

⇱ BigQuery

For AI agents: A markdown version of this page is available at https://docs.datadoghq.com/data_observability/quality_monitoring/data_warehouses/bigquery.md. A documentation index is available at /llms.txt.
This product is not supported for your selected Datadog site. ().

Overview

The BigQuery integration connects Datadog to your Google Cloud project to sync metadata, query history, and table-level metrics. Use it to monitor data freshness, detect anomalies, and trace lineage across your data stack.

Prerequisites

If your Google Cloud project restricts network access by IP, add the Datadog webhook IPs to your allowlist. For the list of IPs, see the webhooks section of .

Configure the BigQuery integration in Datadog

To configure the BigQuery integration in Datadog:

  1. Navigate to Datadog Data Observability > Settings.
  2. Click the Configure button for the BigQuery option.

You can now choose to use a service account that you’ve already connected to Datadog or set up a new one specifically for Data Observability.

Using an existing service account

Choose this option if you’ve previously connected a service account to Datadog.

  1. Select the Use connected Service Account option.
  2. Select the service account from the dropdown list.
  3. Click Next.
  4. Turn on the Enable Data Observability toggle and click Add Account. Data Observability functionality requires additional roles and API access beyond the basic BigQuery integration.

Additional roles

Apply the following roles to the service account in your GCP IAM console.

RoleRole IDDescription
BigQuery Data Viewerroles/bigquery.dataViewerProvides visibility into datasets
BigQuery Resource Viewerroles/bigquery.resourceViewerProvides visibility into jobs
Job Userroles/bigquery.jobUserRequired to run data quality queries

Additional APIs

Make sure the following APIs have been enabled in the project associated with the service account.

APIAPI IDDescription
BigQuery APIbigquery.googleapis.comRequired to access BigQuery

Using a new service account

  1. Select the Add new Service Account option.
  2. Choose one of the setup methods. Quick Start is recommended for most users since it requires no manual permissions setup.

After you’ve completed the Quick Start flow, click Next to proceed to the next page and enable the Enable Data Observability toggle before clicking Add Account.

Manual setup

In your Google Cloud console’s IAM page, create a service account with the following roles:

RoleRole IDDescription
Monitoring Viewerroles/monitoring.viewerProvides read-only access to the monitoring data available in your Google Cloud environment
Cloud Asset Viewerroles/cloudasset.viewerProvides read-only access to cloud assets metadata
Browserroles/browserProvides read-only access to browse the hierarchy of a project
BigQuery Data Viewerroles/bigquery.dataViewerProvides visibility into datasets
BigQuery Resource Viewerroles/bigquery.resourceViewerProvides visibility into jobs
Job Userroles/bigquery.jobUserRequired to run data quality queries
Compute Viewerroles/compute.viewerProvides read-only access to get and list Compute Engine resources

In Datadog, after you’ve selected the Manual setup method:

  1. Copy the generated Datadog Principal to your clipboard. If this is your first time configuration GCP you’ll need to first click the Generate Principal button.
  2. In the Google Cloud console, under Service Accounts, find the service account you recently created and click on it.
  3. Navigate to the Principals with access tab and click the Grant access button.
  4. Paste the Datadog Principal into the New principals text box.
  5. Under the Assign roles section, select the Service Account Token Creator role. This will allow the Datadog Principal to impersonate the service account you just created.
  6. Click the Save button.

Finally, you need to enable the following APIs for every project you want to monitor, including the project where the service account has been created.

APIAPI IDDescription
Cloud Monitoring APImonitoring.googleapis.comAllows Datadog to query your Google Cloud metric data
Cloud Asset APIcloudasset.googleapis.comAllows Datadog to request Google Cloud resources and link relevant labels to metrics as tags
Compute Engine APIcompute.googleapis.comAllows Datadog to discover compute instance data
Cloud Resource Manager APIcloudresourcemanager.googleapis.comAllows Datadog to append metrics with the correct resources and tags
BigQuery APIbigquery.googleapis.comRequired to access BigQuery

Once the service account has been created and the necessary roles and APIs have been applied, you can return to Datadog.

  1. Click Verify and save account.
  2. Click Next.
  3. Turn on the Enable Data Observability toggle and click Add Account.

Next steps

After you configure the integration, Datadog begins syncing your information schema and query history in the background. Initial syncs can take several hours depending on the size of your BigQuery deployment.

After the initial sync completes, create a Data Observability monitor to start alerting on freshness, row count, column-level metrics, and custom SQL metrics.

Further reading