Introduction​

This guide covers Netdata's advanced streaming and replication capabilities, which allow you to build centralized observability points across your infrastructure.

Streaming and replication work together to send metrics data from one Netdata Agent (Child) to another Netdata Agent (Parent). Streaming sends metrics in real-time, while replication ensures historical data is copied, as well, maintaining complete data integrity even after connection interruptions.

Understanding Streaming Architecture​

Before diving into configuration details, it's important to understand the key concepts behind Netdata's streaming architecture:

Parent-Child Relationship​

• Child nodes (data sources) collect metrics directly from systems they monitor
• Parent nodes (data centralization points) receive, store, and visualize metrics from multiple Child nodes
• A single Parent can receive data from many Children, creating a centralized monitoring hub
• Child nodes maintain independence and continue collecting metrics even if the connection to a Parent is lost

Data Flow​

1. Collection: Child nodes collect metrics from their local systems
2. Streaming: Child nodes send real-time metrics to Parent nodes
3. Replication: When a connection is established or restored, Child nodes replicate historical data to ensure Parents have complete history
4. Storage: Parent nodes store metrics based on their configured retention policies
5. Visualization: Users access the Parent node's dashboard to view metrics from all connected Child nodes

Benefits of This Architecture​

• Efficiency: Distribute collection workload across multiple nodes while centralizing visualization
• Resilience: Maintain data collection even during network disruptions or Parent node failures
• Scalability: Add more Child nodes or additional Parent nodes as your infrastructure grows
• Flexibility: Configure retention, alerts, and dashboards according to your specific needs

Protocol Architecture​

Netdata streaming uses a custom binary protocol over TCP, not HTTP/HTTPS. This is an important distinction:

• Custom binary protocol: Streaming uses Netdata's own protocol designed for efficient metrics transfer, not HTTP
• TLS encryption: When you enable :SSL in the destination, it adds TLS encryption as a security layer on top of the custom streaming protocol (this is not HTTPS)
• Port multiplexing: The same port (19999 by default) handles both web API requests and streaming connections—the server automatically detects which protocol is being used based on the initial handshake
• Destination requirement: This is why streaming requires a Netdata server at the destination, not just any HTTP server

Quick Reference​

Configuration Overview​

Netdata's streaming capabilities are configured through two key files:

• stream.conf – Controls streaming behavior, including Parent and Child configurations.
• netdata.conf – Contains global settings that can impact streaming.

To edit these files, navigate to your Netdata configuration directory (typically /etc/netdata) and run:

# Edit streaming configuration
sudo ./edit-config stream.conf

# Edit global Netdata settings
sudo ./edit-config netdata.conf

Configuring stream.conf​

The stream.conf file has three main sections:

1. [stream] – With these settings, you can configure how Child nodes send metrics.
2. [API_KEY] – Here you can define settings for authentication and access control between Parents and Children.
3. [MACHINE_GUID] – This area lets you customize settings for specific Child nodes by their unique ID.

Identifying a Node's GUID​

Each Netdata node has a unique identifier stored in:

/var/lib/netdata/registry/netdata.public.unique.id

This file is generated automatically the first time Netdata starts and remains unchanged.

Recommended Deployment Strategies​

For a production-ready streaming setup, consider the following best practices:

stream.conf Detailed Reference​

[stream] Section (Child Node Settings)​

With these settings, you can configure how your Child nodes send metrics to Parent nodes.

[API_KEY] Section (Parent Node Authentication)​

Here you can define settings for authentication and access control between Parents and Children.

[MACHINE_GUID] Section (Per-Node Customization)​

This area lets you customize settings for specific Child nodes by their unique ID.

Additional Settings​

destination​

Defines Parent nodes for streaming using the format:
[PROTOCOL:]HOST[%INTERFACE][:PORT][:SSL]

• PROTOCOL: tcp, udp, or unix (only tcp and unix are supported for Parents).
• HOST: IPv4, IPv6 (in brackets [ ]), hostname, or Unix domain socket path.
• INTERFACE (IPv6 only): Network interface to use.
• PORT: Port number or service name.
• SSL: Enables TLS/SSL encryption.

Example (TCP connection with SSL to 203.0.113.0 on port 20000):

[stream]
 # Send metrics securely to the Parent at 203.0.113.0:20000
 destination = tcp:203.0.113.0:20000:SSL

send charts matching​

Controls which charts are streamed.

• * (default) – Streams all charts.

• Specific charts:

  [stream]
   # Only send CPU application charts and all system charts
   send charts matching = apps.cpu system.*
  

• Exclude charts using !:

  [stream]
   # Send all charts except CPU application charts
   send charts matching = !apps.cpu *
  

allow from​

Defines which Child nodes (by IP) can connect.

• Allow a single IP:

  [API_KEY]
   # Only allow connections from 203.0.113.10
   allow from = 203.0.113.10
  

• Allow a range but exclude one:

  [API_KEY]
   # Allow all 10.*.*.* addresses except 10.1.2.3
   allow from = !10.1.2.3 10.*
  

db​

Defines the database mode:

• dbengine – Stores recent metrics in RAM and writes older data to disk.
• ram – Stores metrics only in RAM (lost on restart).
• none – No database.

[API_KEY]
 # Use disk-based database for all metrics
 db = dbengine

netdata.conf Settings Affecting Streaming​

The netdata.conf file is the primary configuration file for the Netdata agent. The following sections can impact streaming:

[global]​

This section defines global settings for the Netdata agent.

• hostname: The hostname used by the agent.
• error log file: Path to the file where error logs are saved.

[web]​

Configure the web interface settings here.

• bind to: Define the network address to which Netdata binds.
• port: Set the port for the web interface (default: 19999).
• disable SSL: Set to yes to disable SSL support.

[db]​

Manage database settings for data storage and retention.

• db (formerly memory mode): Choose between in-memory or disk-based storage (e.g., dbengine, ram, none).
• retention: Set how long to keep historical data.
• update every: The data collection frequency in seconds.

Complete Configuration Examples​

Basic Parent-Child Setup​

Parent node configuration (stream.conf):

# Generate a random UUID first: uuidgen
[11111111-2222-3333-4444-555555555555]
 type = api
 # Enable this API key
 enabled = yes
 # Allow all IPs to connect with this key
 allow from = *
 # Store data using dbengine for persistence
 db = dbengine

Child node configuration (stream.conf):

[stream]
 # Enable streaming on this node
 enabled = yes
 # Connect to Parent at 192.168.1.5 port 19999
 destination = 192.168.1.5
 # Use the same API key defined on the Parent
 api key = 11111111-2222-3333-4444-555555555555

High Availability Setup with Multiple Parents​

Parent nodes configuration (stream.conf on both Parents):

# Configuration for accepting metrics from Children
[11111111-2222-3333-4444-555555555555]
 type = api
 enabled = yes
 allow from = *
 db = dbengine

# Configuration for accepting metrics from other Parents
[22222222-3333-4444-5555-666666666666]
 type = api
 enabled = yes
 # Only allow the other Parent's IP
 allow from = 192.168.1.5 192.168.1.6
 db = dbengine

First Parent node's configuration for streaming to the second Parent:

[stream]
 enabled = yes
 destination = 192.168.1.6
 api key = 22222222-3333-4444-5555-666666666666

Second Parent node's configuration for streaming to the first Parent:

[stream]
 enabled = yes
 destination = 192.168.1.5
 api key = 22222222-3333-4444-5555-666666666666

Child node configuration:

[stream]
 enabled = yes
 # List both Parents for failover
 destination = 192.168.1.5 192.168.1.6
 api key = 11111111-2222-3333-4444-555555555555

Verifying Successful Connections​

If the streaming configuration is working correctly, you'll see logs similar to the following.

On the Parent side:

2017-03-09 09:38:52: netdata: INFO : STREAM [receive from [10.11.12.86]:38564]: new client connection.
2017-03-09 09:38:52: netdata: INFO : STREAM xxx [10.11.12.86]:38564: receive thread created (task id 27721)

On the Child side:

2017-03-09 09:38:28: netdata: INFO : STREAM xxx [send to box:19999]: connecting...
2017-03-09 09:38:28: netdata: INFO : STREAM xxx [send to box:19999]: established communication - sending metrics...

Both Parent and Child nodes log information in /var/log/netdata/error.log.

Troubleshooting​

FAQ​

Step-by-Step Setup Guide​

This guide will walk you through setting up Netdata streaming between nodes. By following these sequential steps, you'll create a basic streaming configuration that you can later customize based on your needs.

Quick Troubleshooting Tips​