VOOZH about

URL: https://docs.datadoghq.com/llm_observability/monitoring/prompt_tracking/

⇱ Prompt Tracking

For AI agents: A markdown version of this page is available at https://docs.datadoghq.com/llm_observability/monitoring/prompt_tracking.md. A documentation index is available at /llms.txt.

Prompt Tracking

This product is not supported for your selected Datadog site. ().

In Agent Observability, the Prompt Tracking feature links prompt templates and versions to LLM calls. Prompt Tracking works alongside Agent Observability’s traces, spans, and Playground.

Prompt Tracking enables you to:

  • See all prompts used by your LLM application or agent, with call volume and latency over time
  • Compare prompts or versions by calls, latency, tokens used, and cost
  • See detailed information about a prompt: review its version history, view a text diff, and jump to traces using a specific version
  • Filter Trace Explorer by prompt name, ID, or version to isolate impacted requests
  • Reproduce a run by populating Agent Observability Playground with the exact template and variables from any span

Set up Prompt Tracking

With structured prompt metadata

To use Prompt Tracking, you can submit structured prompt metadata (ID, optional version, template, variables).

Agent Observability Python SDK

If you are using the Agent Observability Python SDK (dd-trace v3.16.0+), attach prompt metadata to the LLM span using the prompt argument or helper. See the Agent Observability Python SDK documentation.

Agent Observability Node.js SDK

If you are using the Agent Observability Node.js SDK (dd-trace v5.83.0+), attach prompt metadata to the LLM span using the prompt option. See the Agent Observability Node.js SDK documentation.

Agent Observability API

If you are using the Agent Observability API intake, submit prompt metadata to the Spans API endpoint. See the Agent Observability HTTP API reference documentation.

OpenTelemetry instrumentation

If you are using OpenTelemetry instrumentation, you can attach prompt metadata to your LLM spans by setting the _dd.ml_obs.prompt_tracking attribute with a JSON string containing your prompt information.

Set the attribute on any LLM span:

import json
span.set_attribute("_dd.ml_obs.prompt_tracking", json.dumps({
 "name": "greeting-prompt",
 "version": "v1",
 "template": "Hello {{name}}, tell me about {{topic}}",
 "variables": {"name": "Alice", "topic": "weather"}
}))

span.setAttribute("_dd.ml_obs.prompt_tracking", JSON.stringify({
 name: "greeting-prompt",
 version: "v1",
 template: "Hello {{name}}, tell me about {{topic}}",
 variables: { name: "Alice", topic: "weather" }
}));

span.SetAttributes(attribute.String("_dd.ml_obs.prompt_tracking",`{"name":"greeting-prompt","version":"v1","template":"Hello {{name}}, tell me about {{topic}}","variables":{"name":"Alice","topic":"weather"}}`,))

The following fields are supported in the prompt tracking JSON:

FieldTypeRequiredDescription
templatestringYes (or chat_template)Template string for single-message prompts
chat_templatearrayYes (or template)List of {"role": "...", "content": "..."} message templates
idstringNoUnique identifier for the prompt. Defaults to {ml_app}_unnamed-prompt if omitted
namestringNoPrompt name. Used as a fallback for id if id is omitted
versionstringNoUser-supplied version tag
variablesobjectNoTemplate variable substitutions
rag_context_variablesarray of stringsNoNames of variables in variables that contain RAG context (ground truth). Used by RAG evaluators
rag_query_variablesarray of stringsNoNames of variables in variables that contain the user query. Used by RAG evaluators
If you are using prompt templates, Agent Observability can automatically attach version information based on prompt content.

With LangChain templates

If you are using LangChain prompt templates, Datadog automatically captures prompt metadata without code changes. IDs are derived from module or template names. To override these IDs, see Agent Observability Auto-instrumentation: LangChain.

Use Prompt Tracking in Agent Observability

View your app in Agent Observability and select Prompts on the left. The Prompts view features the following information:

  • Prompt Call Count: A timeseries chart displaying calls per prompt (or per version) over time
  • Recent Prompt Updates: Information about recent prompt updates, including time of last update, call count, average latency, and average tokens per call
  • Most Tokens Used: Prompts ranked by total (input or output) tokens
  • Highest Latency Prompts: Prompts ranked by average duration

Click on a prompt to open a detailed side-panel view that features information about version activity and various metrics. You can also see a diff view of two versions, open Trace Explorer pre-filtered to spans that use a selected version, or start a Playground session pre-populated with the selected version’s template and variables.

You can use the Agent Observability Trace Explorer to locate requests by prompt usage. You can use a prompt’s name, ID, and version as facets for both trace-level and span-level search. Click any LLM span to see the prompt that generated it.

Further Reading