VOOZH about

URL: https://docs.datadoghq.com/llm_observability/monitoring/llm_observability_and_apm/

⇱ Correlating Agent Observability and APM

For AI agents: A markdown version of this page is available at https://docs.datadoghq.com/llm_observability/monitoring/llm_observability_and_apm.md. A documentation index is available at /llms.txt.

Correlating Agent Observability and APM

This product is not supported for your selected Datadog site. ().

Overview

This guide explains how you can use both Agent Observability and APM to correlate Agent Observability and APM spans in Datadog.

By instrumenting your LLM-specific operations with Agent Observability and your broader application with APM, you can accomplish the following:

  • Understand end-to-end visibility: Explore upstream and downstream requests of your LLM applications within the context of your entire application.
  • From APM, dive deeper into Agent Observability: Investigate whether or not an issue with your application is specific to LLM-specific applications, such as a call to OpenAI.

Setup

The Agent Observability SDK is built on APM’s dd-tracer. This allows you to use Agent Observability with Application Performance Monitoring (APM)

If you are using the Agent Observability SDK for Python along with APM’s dd-tracer, you can navigate between spans in Datadog APM and Agent Observability without additional setup.

If you are using the Agent Observability API with dd-tracer for APM:

  1. Use the appropriate method to obtain the span ID from the tracer (for example, using span.Context().SpanID() for the Go tracer).
  2. Include the captured span IDs in all of the Agent Observability API requests. This links APM and Agent Observability spans in Datadog.

Navigate between spans

By using this integration, you can correlate data across your application stack and understand how your LLM applications interact with other components. You can also resolve issues more quickly and optimize your application’s performance.

From Agent Observability to APM

To understand the broader context of your LLM operations within your application ecosystem, select an Agent Observability span in the Agent Observability Explorer and click APM span to navigate to the relevant APM span.

From APM to Agent Observability

To access LLM-specific insights, select an APM span in the Trace Explorer and click View Span in the Agent Observability section on the Info tab to navigate to the corresponding Agent Observability span.

Further Reading