 |
VOOZH | about |
|
VOOZH | about |
Runtime metrics provide insights into application performance, including memory usage, garbage collection, and parallelization. Datadog SDKs offer runtime metrics collection for each supported language, and OpenTelemetry (OTel) also collects compatible runtime metrics that can be sent to Datadog through the OpenTelemetry SDKs.
Datadog supports OpenTelemetry runtime metrics for the following languages:
For details about host and container metrics mapping, see OpenTelemetry Metrics Mapping.
Select your language to see instructions for configuring the OpenTelemetry SDK to send runtime metrics:
Note: Runtime metrics are only exported if a MeterProvider and metric exporter are configured. Set the OTEL_METRICS_EXPORTER environment variable or programmatically configure a metricReader in your SDK initialization. For Go, Node.js, and Python, the MeterProvider must be configured manually; Java and .NET auto-configure it via their auto-instrumentation agents.
If you use OpenTelemetry automatic instrumentation for Java applications, runtime metrics are enabled by default.
If you use OpenTelemetry manual instrumentation, follow the guides for your Java version:
Enabling experimental OpenTelemetry runtime telemetry may provide additional metric mappings between Datadog and OpenTelemetry for Java. To enable it, set the following environment variable:
OTEL_INSTRUMENTATION_RUNTIME_TELEMETRY_EMIT_EXPERIMENTAL_TELEMETRY=true
To collect additional JVM metrics beyond the default runtime instrumentation, install the OpenTelemetry JMX Metric Scraper. The scraper scrapes MBeans over JMX and emits them as OpenTelemetry metrics, which Datadog then maps to runtime metrics (see the JVM Contrib table in Data collected).
Configure the scraper with otel.jmx.target.source=legacy to collect these additional metrics. The scraper’s instrumentation source emits the same semantic-convention metrics already produced natively by the OpenTelemetry Java SDK, so it does not provide additional coverage.
Note: The minimum supported version of the OpenTelemetry Java agent is 2.0.0.
OpenTelemetry Go applications are instrumented manually. To enable runtime metrics, see the documentation for the runtime package.
Note: The minimum supported version of go.opentelemetry.io/contrib/instrumentation/runtime is v0.46.0, which also requires Go 1.20+ and OpenTelemetry Go SDK v1.21.0+.
If you use OpenTelemetry automatic instrumentation for .NET applications, runtime metrics are enabled by default.
If you use OpenTelemetry manual instrumentation, see the documentation for the OpenTelemetry.Instrumentation.Runtime library.
The default metric export interval for the .NET OTel SDK differs from the default for the Datadog .NET SDK. Datadog recommends setting the OTEL_METRIC_EXPORT_INTERVAL environment variable on your .NET service to match the default Datadog metric export interval:
OTEL_METRIC_EXPORT_INTERVAL=10000
Note: The minimum supported version of the .NET OpenTelemetry SDK is 1.5.0.
If you use OpenTelemetry automatic instrumentation for Node.js applications, runtime metrics are enabled by default through the @opentelemetry/instrumentation-runtime-node package.
If you use OpenTelemetry manual instrumentation, see the documentation for the @opentelemetry/instrumentation-runtime-node library.
Host-level metrics such as system CPU and memory usage are not included in OpenTelemetry automatic instrumentation. To collect these metrics:
Install and configure the @opentelemetry/host-metrics package:
npm install @opentelemetry/host-metrics
Initialize the package with your existing MeterProvider:
const { HostMetrics } = require('@opentelemetry/host-metrics');
const { metrics } = require('@opentelemetry/api');
const hostMetrics = new HostMetrics({
meterProvider: metrics.getMeterProvider(),
});
hostMetrics.start();
For the list of metrics collected by this package, see the Node.js Contrib Host table.
Note: The minimum supported version of @opentelemetry/instrumentation-runtime-node is 0.9.0.
Runtime metrics are not enabled by default for Python applications. Install the opentelemetry-instrumentation-system-metrics package:
pip install opentelemetry-instrumentation-system-metrics
If you use OpenTelemetry automatic instrumentation for Python applications, opentelemetry-instrument discovers and enables the package after installation.
If you use OpenTelemetry manual instrumentation, enable the package in your application:
from opentelemetry.instrumentation.system_metrics import SystemMetricsInstrumentor
SystemMetricsInstrumentor().instrument()
After setup is complete, you can view runtime metrics in:
The following tables list the OpenTelemetry runtime metrics collected for Datadog’s out-of-the-box in-app experiences, along with a description and any attribute conditions (Filter) that Datadog uses to identify the correct data point.
For Collector processor configuration required to make these metrics compatible with Datadog, see the Collector configuration instructions above.
These metrics are collected when using the latest version of the OpenTelemetry Java SDK.
| OTEL | DESCRIPTION | FILTER |
|---|---|---|
| jvm.buffer.count | Number of buffers in the pool. | jvm.buffer.pool.name: direct |
| jvm.buffer.count | Number of buffers in the pool. | jvm.buffer.pool.name: mapped |
| jvm.buffer.memory.limit | Measure of total memory capacity of buffers. | jvm.buffer.pool.name: direct |
| jvm.buffer.memory.limit | Measure of total memory capacity of buffers. | jvm.buffer.pool.name: mapped |
| jvm.buffer.memory.used | Measure of memory used by buffers. | jvm.buffer.pool.name: direct |
| jvm.buffer.memory.used | Measure of memory used by buffers. | jvm.buffer.pool.name: mapped |
| jvm.class.count | Number of classes currently loaded. | |
| jvm.cpu.recent_utilization | Recent CPU utilization for the process as reported by the JVM. Note: The value range is [0.0,1.0]. This utilization is not defined as being for the specific interval since last measurement (unlike system.cpu.utilization). Reference. | |
| jvm.file_descriptor.count | Number of open file descriptors as reported by the JVM. | |
| jvm.memory.used | Measure of memory used. | jvm.memory.pool.name: g1_old_gen, zgc_old_generation, tenured_gen, ps_old_gen, cms_old_genjvm.memory.type: heap |
| jvm.memory.used | Measure of memory used. | jvm.memory.pool.name: g1_eden_space, zgc_young_generation, eden_space, ps_eden_space, par_eden_spacejvm.memory.type: heap |
| jvm.memory.used | Measure of memory used. | jvm.memory.pool.name: g1_survivor_space, survivor_space, ps_survivor_space, par_survivor_spacejvm.memory.type: heap |
| jvm.memory.used | Measure of memory used. | jvm.memory.pool.name: metaspacejvm.memory.type: non_heap |
| jvm.system.cpu.utilization | Recent CPU utilization for the whole system as reported by the JVM. Note: The value range is [0.0,1.0]. This utilization is not defined as being for the specific interval since last measurement (unlike system.cpu.utilization). Reference. |
These metrics are collected when using the OpenTelemetry JMX Metric Scraper.
| OTEL | DESCRIPTION | FILTER |
|---|---|---|
| jvm.classes.loaded | The number of loaded classes | |
| jvm.gc.collections.count | The total number of garbage collections that have occurred | name: copy, ps_scavenge, parnew, g1_young_generation, zgc_minor_cycles, zgc_minor_pauses |
| jvm.gc.collections.count | The total number of garbage collections that have occurred | name: marksweepcompact, ps_marksweep, concurrentmarksweep, g1_mixed_generation, g1_old_generation, shenandoah_cycles, zgc_major_cycles, zgc_major_pauses |
| jvm.gc.collections.elapsed | The approximate accumulated collection elapsed time | name: copy, ps_scavenge, parnew, g1_young_generation, zgc_minor_cycles, zgc_minor_pauses |
| jvm.gc.collections.elapsed | The approximate accumulated collection elapsed time | name: marksweepcompact, ps_marksweep, concurrentmarksweep, g1_mixed_generation, g1_old_generation, shenandoah_cycles, zgc_major_cycles, zgc_major_pauses |
| jvm.memory.heap.committed | The amount of memory that is guaranteed to be available for the heap | |
| jvm.memory.heap.init | The initial amount of memory that the JVM requests from the operating system for the heap | |
| jvm.memory.heap.max | The maximum amount of memory can be used for the heap | |
| jvm.memory.heap.used | The current heap memory usage | |
| jvm.memory.nonheap.committed | The amount of memory that is guaranteed to be available for non-heap purposes | |
| jvm.memory.nonheap.init | The initial amount of memory that the JVM requests from the operating system for non-heap purposes | |
| jvm.memory.nonheap.max | The maximum amount of memory can be used for non-heap purposes | |
| jvm.memory.nonheap.used | The current non-heap memory usage | |
| jvm.memory.pool.used | The current memory pool memory usage | name: metaspace |
| jvm.memory.pool.used | The current memory pool memory usage | name: g1_old_gen, zgc_old_generation, tenured_gen, ps_old_gen, cms_old_gen |
| jvm.memory.pool.used | The current memory pool memory usage | name: g1_eden_space, zgc_young_generation, eden_space, ps_eden_space, par_eden_space |
| jvm.memory.pool.used | The current memory pool memory usage | name: g1_survivor_space, survivor_space, ps_survivor_space, par_survivor_space |
| jvm.threads.count | The current number of threads |
These metrics are collected when using OpenTelemetry Java SDK versions 1.32.0 and earlier.
| OTEL | DESCRIPTION | FILTER |
|---|---|---|
| process.runtime.jvm.buffer.count | Number of buffers in the pool. | pool: direct |
| process.runtime.jvm.buffer.count | Number of buffers in the pool. | pool: mapped |
| process.runtime.jvm.buffer.usage | Measure of memory used by buffers. | pool: direct |
| process.runtime.jvm.buffer.usage | Measure of memory used by buffers. | pool: mapped |
| process.runtime.jvm.classes.current_loaded | Number of classes currently loaded. | |
| process.runtime.jvm.cpu.utilization | Recent CPU utilization for the process. Note: The value range is [0.0,1.0]. This utilization is not defined as being for the specific interval since last measurement (unlike system.cpu.utilization). Reference. | |
| process.runtime.jvm.memory.usage | Measure of memory used. | pool: g1_old_gen, zgc_old_generation, tenured_gen, ps_old_gen, cms_old_gentype: heap |
| process.runtime.jvm.memory.usage | Measure of memory used. | pool: g1_eden_space, zgc_young_generation, eden_space, ps_eden_space, par_eden_spacetype: heap |
| process.runtime.jvm.memory.usage | Measure of memory used. | pool: g1_survivor_space, survivor_space, ps_survivor_space, par_survivor_spacetype: heap |
| process.runtime.jvm.memory.usage | Measure of memory used. | pool: metaspacetype: non_heap |
| process.runtime.jvm.system.cpu.utilization | Recent CPU utilization for the whole system as reported by the JVM. Note: The value range is [0.0,1.0]. This utilization is not defined as being for the specific interval since last measurement (unlike system.cpu.utilization). Reference. |
These metrics are collected by the OpenTelemetry Go runtime instrumentation package.
| OTEL | DESCRIPTION |
|---|---|
| go.config.gogc | Heap size target percentage configured by the user, otherwise 100. Note: The value range is [0.0,100.0]. Computed from /gc/gogc:percent. |
| go.goroutine.count | Count of live goroutines. Note: Computed from /sched/goroutines:goroutines. |
| go.memory.allocated | Memory allocated to the heap by the application. Note: Computed from /gc/heap/allocs:bytes. |
| go.memory.allocations | Count of allocations to the heap by the application. Note: Computed from /gc/heap/allocs:objects. |
| go.memory.gc.goal | Heap size target for the end of the GC cycle. Note: Computed from /gc/heap/goal:bytes. |
| go.memory.limit | Go runtime memory limit configured by the user, if a limit exists. Note: Computed from /gc/gomemlimit:bytes. This metric is excluded if the limit obtained from the Go runtime is math.MaxInt64. |
| go.processor.limit | The number of OS threads that can execute user-level Go code simultaneously. Note: Computed from /sched/gomaxprocs:threads. |
| process.runtime.go.cgo.calls | Number of cgo calls made by the current process |
| process.runtime.go.gc.count | Number of completed garbage collection cycles |
| process.runtime.go.gc.pause_total_ns | Cumulative nanoseconds in GC stop-the-world pauses since the program started |
| process.runtime.go.goroutines | Number of goroutines that currently exist |
| process.runtime.go.mem.heap_alloc | Bytes of allocated heap objects |
| process.runtime.go.mem.heap_idle | Bytes in idle (unused) spans |
| process.runtime.go.mem.heap_inuse | Bytes in in-use spans |
| process.runtime.go.mem.heap_objects | Number of allocated heap objects |
| process.runtime.go.mem.heap_released | Bytes of idle spans whose physical memory has been returned to the OS |
| process.runtime.go.mem.heap_sys | Bytes of heap memory obtained from the OS |
These metrics are emitted by the .NET runtime's built-in System.Runtime meter on .NET 9.0 and later. The OpenTelemetry SDK collects and exports them automatically.
| OTEL | DESCRIPTION | FILTER |
|---|---|---|
| dotnet.gc.last_collection.heap.size | The managed GC heap size (including fragmentation), as observed during the latest garbage collection. Note: Meter name: System.Runtime; Added in: .NET 9.0. This metric reports the same values as calling GC.GetGCMemoryInfo().GenerationInfo.SizeAfterBytes. | gc.heap.generation: gen0 |
| dotnet.gc.last_collection.heap.size | The managed GC heap size (including fragmentation), as observed during the latest garbage collection. Note: Meter name: System.Runtime; Added in: .NET 9.0. This metric reports the same values as calling GC.GetGCMemoryInfo().GenerationInfo.SizeAfterBytes. | gc.heap.generation: gen1 |
| dotnet.gc.last_collection.heap.size | The managed GC heap size (including fragmentation), as observed during the latest garbage collection. Note: Meter name: System.Runtime; Added in: .NET 9.0. This metric reports the same values as calling GC.GetGCMemoryInfo().GenerationInfo.SizeAfterBytes. | gc.heap.generation: gen2 |
| dotnet.gc.last_collection.heap.size | The managed GC heap size (including fragmentation), as observed during the latest garbage collection. Note: Meter name: System.Runtime; Added in: .NET 9.0. This metric reports the same values as calling GC.GetGCMemoryInfo().GenerationInfo.SizeAfterBytes. | gc.heap.generation: loh |
| dotnet.process.cpu.time | CPU time used by the process. Note: Meter name: System.Runtime; Added in: .NET 9.0. This metric reports the same values as accessing the corresponding processor time properties on System.Diagnostics.Process. | cpu.mode: user |
| dotnet.process.cpu.time | CPU time used by the process. Note: Meter name: System.Runtime; Added in: .NET 9.0. This metric reports the same values as accessing the corresponding processor time properties on System.Diagnostics.Process. | cpu.mode: system |
These metrics are collected by the OpenTelemetry.Instrumentation.Runtime package. On .NET 9.0 and later, these overlap with the System.Runtime metrics above.
| OTEL | DESCRIPTION | FILTER |
|---|---|---|
| process.runtime.dotnet.gc.heap.size | The heap size (including fragmentation), as observed during the | generation: gen0 |
| process.runtime.dotnet.gc.heap.size | The heap size (including fragmentation), as observed during the | generation: gen1 |
| process.runtime.dotnet.gc.heap.size | The heap size (including fragmentation), as observed during the | generation: gen2 |
| process.runtime.dotnet.gc.heap.size | The heap size (including fragmentation), as observed during the | generation: loh |
| process.runtime.dotnet.thread_pool.threads.count | The number of thread pool threads that currently exist. |
These metrics are collected by the OpenTelemetry.Instrumentation.Process package.
| OTEL | DESCRIPTION |
|---|---|
| process.thread.count | Process threads count. |
These metrics are emitted from auto-instrumentation with the latest version of the OpenTelemetry Node.js SDK.
| OTEL | DESCRIPTION |
|---|---|
| nodejs.eventloop.utilization | Event loop utilization. Note: The value range is [0.0, 1.0] and can be retrieved from performance.eventLoopUtilization([utilization1[, utilization2]]) |
| v8js.memory.heap.limit | Total heap memory size pre-allocated. Note: The value can be retrieved from value space_size of v8.getHeapSpaceStatistics() |
| v8js.memory.heap.space.available_size | Heap space available size. Note: Value can be retrieved from value space_available_size of v8.getHeapSpaceStatistics() |
| v8js.memory.heap.space.physical_size | Committed size of a heap space. Note: Value can be retrieved from value physical_space_size of v8.getHeapSpaceStatistics() |
| v8js.memory.heap.used | Heap Memory size allocated. Note: The value can be retrieved from value space_used_size of v8.getHeapSpaceStatistics() |
These metrics are collected by the @opentelemetry/host-metrics package. This package is not included in OpenTelemetry automatic instrumentation and must be installed and configured manually.
| OTEL | DESCRIPTION | FILTER |
|---|---|---|
| process.memory.usage | The amount of physical memory in use. | |
| system.memory.usage | Bytes of memory in use. | |
| system.memory.usage | Bytes of memory in use. | state: free, cached, buffered |
| system.memory.usage | Bytes of memory in use. | state: free |
| system.memory.usage | Bytes of memory in use. | system.memory.state: free |
These metrics are collected by the opentelemetry-instrumentation-system-metrics package. The following table lists the conceptual equivalences between OpenTelemetry and Datadog Python runtime metrics. There are no direct metric-name mappings because the metric types differ between the two systems.
| OTEL | Datadog | Description | Filter |
|---|---|---|---|
process.cpu.time | runtime.python.cpu.time.sys | Number of seconds executing in the kernel. | type: system |
process.cpu.time | runtime.python.cpu.time.user | Number of seconds executing outside the kernel. | type: user |
process.cpu.utilization | runtime.python.cpu.percent | CPU utilization percentage. OTel divides the raw value by 100 times the number of CPU cores. | |
process.context_switches | runtime.python.cpu.ctx_switch.voluntary | Number of voluntary context switches. | type: voluntary |
process.context_switches | runtime.python.cpu.ctx_switch.involuntary | Number of involuntary context switches. | type: involuntary |
process.runtime.{python_implementation}.gc_count | runtime.python.gc.count.gen0 | Number of generation 0 objects. | count: 0 |
process.runtime.{python_implementation}.gc_count | runtime.python.gc.count.gen1 | Number of generation 1 objects. | count: 1 |
process.runtime.{python_implementation}.gc_count | runtime.python.gc.count.gen2 | Number of generation 2 objects. | count: 2 |
process.memory.usage | runtime.python.mem.rss | Resident set memory. | |
process.thread.count | runtime.python.thread_count | Number of threads. |
OpenTelemetry runtime metrics are mapped to Datadog by metric name. Do not rename host metrics for OpenTelemetry runtime metrics as this breaks the mapping.
Additional helpful documentation, links, and articles:
| |
