Unified Observability for .NET 8 Microservices with OpenTelemetry, Azure Monitor, and Grafana Loki

Why observability matters for .NET 8 microservices

Imagine a user places an order and the transaction disappears in the middle of the workflow. Without a unified view of traces, metrics, and logs, pinpointing the failure can take hours. In 2023, 62% of organizations reported that missing observability caused SLA breaches in distributed systems. For .NET 8 developers, the challenge is not just collecting data but correlating it across services that run on Kubernetes, Azure App Service, or container‑based platforms.

Setting up OpenTelemetry in a .NET 8 microservice

OpenTelemetry provides a vendor‑agnostic SDK that can emit traces, metrics, and logs to multiple back‑ends. Begin by adding the required NuGet packages to your project:

dotnet add package OpenTelemetry.Exporter.AzureMonitor
 dotnet add package OpenTelemetry.Exporter.Jaeger
 dotnet add package OpenTelemetry.Extensions.Hosting
 dotnet add package OpenTelemetry.Instrumentation.AspNetCore
 dotnet add package OpenTelemetry.Instrumentation.HttpClient
 dotnet add package OpenTelemetry.Exporter.Loki

Next, configure the SDK in Program.cs. The example below creates a single WebApplication builder, registers tracing, metrics, and log exporters, and tags every telemetry item with the service name OrderService:

using Microsoft.Extensions.DependencyInjection;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using OpenTelemetry.Logs;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .WithTracing(tracerProviderBuilder => tracerProviderBuilder
        .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("OrderService"))
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddJaegerExporter(options =>
        {
            options.AgentHost = "jaeger";
            options.AgentPort = 6831;
        })
        .AddAzureMonitorTraceExporter(o => o.ConnectionString = builder.Configuration["AzureMonitor:ConnectionString"]))
    .WithMetrics(metricsBuilder => metricsBuilder
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation());

builder.Logging.AddOpenTelemetry(options =>
{
    options.SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("OrderService"));
    options.AddConsoleExporter();
    options.AddAzureMonitorLogExporter(o => o.ConnectionString = builder.Configuration["AzureMonitor:ConnectionString"]);
    options.AddLokiExporter(o =>
    {
        o.Endpoint = new Uri("http://loki:3100/api/prom/push");
        o.Labels = new Dictionary { { "service", "OrderService" } };
    });
});

var app = builder.Build();
app.MapControllers();
app.Run();

The code above does three things: (1) instruments incoming HTTP requests and outgoing HttpClient calls, (2) exports traces to Azure Monitor and Jaeger (useful for local debugging), and (3) ships logs to both Azure Monitor and Grafana Loki.

Exporting traces to Azure Monitor

Azure Monitor’s Application Insights endpoint expects the OpenTelemetry ConnectionString. Store it securely in appsettings.json or Azure Key Vault:

{
  "AzureMonitor": {
    "ConnectionString": "InstrumentationKey=xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx;IngestionEndpoint=https://westus2-1.in.applicationinsights.azure.com/"
  }
}

When the service starts, the exporter batches spans and sends them every 10 seconds. In production you can tune the ExportProcessorType to Batch or Simple depending on latency requirements. Azure Monitor automatically creates a trace view where you can filter by operation name, status code, or custom attribute such as orderId.

Shipping logs to Grafana Loki

Grafana Loki works best when logs are labeled. The OpenTelemetry Loki exporter lets you attach static labels (service name) and dynamic ones (environment, version). Ensure the Loki service is reachable from the container network, typically at http://loki:3100. After deployment, open Grafana, add Loki as a data source, and run a query like:

{service="OrderService"} |= "Exception" |~ "Timeout"

This query returns every log line from OrderService that contains the word “Exception” and matches the regular expression “Timeout”. Because traces and logs share the same trace_id attribute, you can click a trace in Azure Monitor and jump directly to the related logs in Grafana, achieving true end‑to‑end visibility.

Correlating traces and logs across services

Distributed tracing assigns a unique trace_id to each request. OpenTelemetry propagates this identifier via the W3C Trace‑Context header. When a downstream service receives the request, its instrumentation automatically picks up the header and continues the span. To make correlation work in Loki, configure the log exporter to include the trace_id as a label:

options.AddLokiExporter(o =>
{
    o.Endpoint = new Uri("http://loki:3100/api/prom/push");
    o.Labels = new Dictionary
    {
        { "service", "OrderService" },
        { "trace_id", "${traceId}" }
    };
});

Now a Grafana dashboard can display a trace timeline alongside its logs, letting you answer questions like “Which request generated this error?” in seconds instead of minutes.

Practical tips for production readiness

1. **Sampling** – Enable ParentBasedSampler with a 10% rate for high‑traffic APIs to keep data volume manageable.
2. **Resource tagging** – Add environment (prod, staging) and version (v1.3.0) labels to every telemetry item; this simplifies cost allocation in Azure Monitor.
3. **Health checks** – Expose an endpoint (/healthz) that returns the status of the OpenTelemetry exporters; Azure Monitor can scrape it for alerts.
4. **Retention policies** – Azure Monitor retains traces for 90 days by default, while Loki can be configured with a 30‑day retention; align them to avoid gaps.
5. **Security** – Use managed identities for the Azure Monitor connection string and secure Loki with basic auth or OAuth.

Conclusion

Unified observability for .NET 8 microservices is no longer a piecemeal effort. By leveraging OpenTelemetry’s multi‑exporter model, Azure Monitor’s powerful trace analytics, and Grafana Loki’s flexible log querying, developers gain a single pane of glass that shortens incident resolution from hours to minutes. Implement the code snippets, apply the best‑practice checklist, and you’ll have a production‑grade observability stack that scales with your .NET ecosystem.

Sources

• Microsoft Docs – OpenTelemetry .NET
• Grafana Labs – Loki Exporter Guide
• Azure Monitor documentation – Application Insights

Author: Mahmut Sarıkaya — sarikayadev.com