[Updated 20 July, 2024]
- I've updated to .NET 8 & changes regarding otel collector
Overview
The Observability concept become the standard of almost system in recently. It helps team to troubleshoot what's happening inside the system. There are 3 pillars of Observability - Traces, Metrics and Logs.
OpenTelemetry is the most straightforward way to collect Traces and Metrics. Then export to Zipkin or Jaeger for tracings; Prometheus
Serilog is also the most straightforward way to collect logs and then export to Seq or ElasticSearch
Because of the various exporting ways, we have to consider one of these options when implementing
- Support all type of exporting then toggle via settings. For example, only export to Zipkin if it's enabled
- Or, just only export to Zipkin or Jaeger
Only one answer for the concerns
❓ Concerns
- Is there any way that just only one export for multiple consumers? Or,
- Is there any way that just only one export but change consumer without changing the code?
🌟 Only one answer
- And luckily, there is an elegant way to accomplish them - that is OpenTelemetry Collector (OTEL Collector in short)
Objectives
- Usability: Reasonable default configuration, supports popular protocols, runs and collects out of the box.
- Performance: Highly stable and performant under varying loads and configurations.
- Observability: An exemplar of an observable service.
- Extensibility: Customizable without touching the core code.
- Unification: Single codebase, deployable as an agent or collector with support for traces, metrics, and logs (future).
- An image more than thousand words
💻 Let our hand dirty
👉 The below steps are just the showcase of using OTEL Collector within .NET 8. The full implementation can be found at - .NET with OpenTelemetry Collector
👉 In which, we'll export the telemetry signals from application to OTEL Collector then they'll be exported to - Zipkin or Jaeger for tracings; Prometheus; and Loki for logs
Nuget packages Directory.Packages.props
<Project>
<PropertyGroup>
<ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
</PropertyGroup>
<ItemGroup>
<PackageVersion Include="Grpc.AspNetCore" Version="2.64.0" />
<PackageVersion Include="Microsoft.AspNetCore.OpenApi" Version="8.0.7" />
<PackageVersion Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.9.0" />
<PackageVersion Include="OpenTelemetry.Extensions.Hosting" Version="1.9.0" />
<PackageVersion Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.9.0" />
<PackageVersion Include="Serilog" Version="4.0.0" />
<PackageVersion Include="Serilog.AspNetCore" Version="8.0.1" />
<PackageVersion Include="Serilog.Sinks.OpenTelemetry" Version="3.0.0" />
<PackageVersion Include="Serilog.Sinks.PeriodicBatching" Version="5.0.0" />
<PackageVersion Include="Swashbuckle.AspNetCore" Version="6.6.2" />
</ItemGroup>
</Project>
Register OpenTelemetry, typically from Program.cs
var builder = WebApplication.CreateBuilder(args);
builder.Host.AddSerilog();
builder.Services
.AddOpenTelemetry()
.ConfigureResource(resource => resource.AddService(observabilityOptions.ServiceName))
.AddMetrics(observabilityOptions)
.AddTracing(observabilityOptions);
Configure Tracings
private static OpenTelemetryBuilder AddTracing(this OpenTelemetryBuilder builder, ObservabilityOptions observabilityOptions)
{
if (!observabilityOptions.EnabledTracing) return builder;
builder.WithTracing(tracing =>
{
tracing
.SetErrorStatusOnException()
.SetSampler(new AlwaysOnSampler())
.AddAspNetCoreInstrumentation(options =>
{
options.RecordException = true;
});
/* Add more instrument here: MassTransit, NgSql ... */
/* ============== */
/* Only export to OpenTelemetry collector */
/* ============== */
tracing
.AddOtlpExporter(_ =>
{
_.Endpoint = observabilityOptions.CollectorUri;
_.ExportProcessorType = ExportProcessorType.Batch;
_.Protocol = OpenTelemetry.Exporter.OtlpExportProtocol.Grpc;
});
});
return builder;
}
Configure for Metrics
private static OpenTelemetryBuilder AddMetrics(this OpenTelemetryBuilder builder, ObservabilityOptions observabilityOptions)
{
builder.WithMetrics(metrics =>
{
metrics
.AddAspNetCoreInstrumentation();
/* Add more instrument here */
/* ============== */
/* Only export to OpenTelemetry collector */
/* ============== */
metrics
.AddOtlpExporter(_ =>
{
_.Endpoint = observabilityOptions.CollectorUri;
_.ExportProcessorType = ExportProcessorType.Batch;
_.Protocol = OpenTelemetry.Exporter.OtlpExportProtocol.Grpc;
});
});
return builder;
}
Configure Logs
private static WebApplicationBuilder AddSerilog(this WebApplicationBuilder builder, ObservabilityOptions observabilityOptions)
{
var services = builder.Services;
var configuration = builder.Configuration;
services.AddSerilog((sp, serilog) =>
{
serilog
.ReadFrom.Configuration(configuration, new ConfigurationReaderOptions
{
SectionName = $"{nameof(ObservabilityOptions)}:{nameof(Serilog)}"
})
.ReadFrom.Services(sp)
.Enrich.FromLogContext()
.Enrich.WithProperty("ApplicationName", observabilityOptions.ServiceName)
.WriteTo.Console();
/* ============== */
/* Only export to OpenTelemetry collector */
/* ============== */
serilog
.WriteTo.OpenTelemetry(c =>
{
c.Endpoint = observabilityOptions.CollectorUrl;
c.Protocol = OtlpProtocol.Grpc;
c.IncludedData = IncludedData.TraceIdField | IncludedData.SpanIdField | IncludedData.SourceContextAttribute;
c.ResourceAttributes = new Dictionary<string, object>
{
{"service.name", observabilityOptions.ServiceName},
{"index", 10},
{"flag", true},
{"value", 3.14}
};
});
});
return builder;
}
The interesting here
1️⃣ - Refer to docker-compose.observability.yaml
2️⃣ - Refer to otel-collector.yaml to configure OTEL Collector
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
grpc:
endpoint: 0.0.0.0:4317
processors:
batch:
timeout: 1s
resource:
attributes:
- action: insert
key: loki.resource.labels
value: service.name, service.namespace
- action: insert
key: loki.format
value: json
exporters:
debug:
verbosity: normal
prometheus:
endpoint: 0.0.0.0:8889
namespace: test-space
resource_to_telemetry_conversion:
enabled: true
enable_open_metrics: true
otlp/jaeger:
endpoint: jaeger:4317
tls:
insecure: true
zipkin:
endpoint: "http://zipkin:9411/api/v2/spans"
format: proto
loki:
endpoint: http://loki:3100/loki/api/v1/push
default_labels_enabled:
exporter: false
job: true
extensions:
health_check:
pprof:
endpoint: :1888
zpages:
endpoint: :55679
service:
extensions: [pprof, zpages, health_check]
pipelines:
traces:
receivers: [otlp]
processors: [batch, resource]
exporters: [debug, otlp/jaeger, zipkin]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [debug, prometheus]
logs:
receivers: [otlp]
processors: [batch]
exporters: [debug, loki]
Let's 👍 OTEL Collector and take fully implementation at - .NET with OpenTelemetry Collector
Cheers!!! 🍻
For further actions, you may consider blocking this person and/or reporting abuse
