Observability‑First Performance Tuning for .NET 8 with OpenTelemetry, Azure Monitor, and Grafana Loki

Why Observability Matters in .NET 8

Did you know that 70% of production incidents in modern cloud apps are traced back to missing telemetry? .NET 8 introduces performance improvements, but without a solid observability foundation, those gains can disappear under the weight of latency spikes and memory leaks. An observability‑first mindset forces you to instrument, collect, and analyze metrics before you start optimizing code.

Choosing the Right Toolchain: OpenTelemetry, Azure Monitor, and Grafana Loki

OpenTelemetry provides a vendor‑agnostic API for traces, metrics, and logs. Azure Monitor offers seamless integration with Azure resources and a powerful analytics engine, while Grafana Loki excels at log aggregation with low overhead. Combining the three gives you end‑to‑end visibility: traces for request flow, metrics for resource usage, and logs for contextual debugging.

Step‑by‑Step: Adding OpenTelemetry to a .NET 8 Web API

Start with a clean .NET 8 project. The following code registers tracing, metrics, and the exporters you need. Notice the use of ResourceBuilder to tag every signal with the service name, which simplifies cross‑system correlation in Azure Monitor and Loki.

using OpenTelemetry; using OpenTelemetry.Trace; using OpenTelemetry.Metrics; using OpenTelemetry.Resources; var builder = WebApplication.CreateBuilder(args); builder.Services.AddOpenTelemetryTracing(tracerProviderBuilder => tracerProviderBuilder .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("MyDotNetService")) .AddAspNetCoreInstrumentation() .AddHttpClientInstrumentation() .AddAzureMonitorTraceExporter(azureOptions => { azureOptions.ConnectionString = "InstrumentationKey=YOUR_KEY"; }) .AddLokiExporter(lokiOptions => { lokiOptions.Endpoint = new Uri("http://localhost:3100"); })); builder.Services.AddOpenTelemetryMetrics(metricProviderBuilder => metricProviderBuilder .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("MyDotNetService")) .AddAspNetCoreInstrumentation() .AddHttpClientInstrumentation() .AddAzureMonitorMetricExporter(azureOptions => { azureOptions.ConnectionString = "InstrumentationKey=YOUR_KEY"; })); var app = builder.Build(); app.MapGet("/", () => "Hello World!"); app.Run();

After publishing, you should see traces appear in Azure Monitor under "Application Insights" and logs flow into Loki at http://localhost:3100. Use Azure Log Analytics queries or Grafana dashboards to spot latency outliers.

Configuring Azure Monitor for Real‑Time Alerts

Azure Monitor lets you create metric alerts that trigger when CPU usage exceeds 80% for more than five minutes, or when the average request duration crosses 200 ms. In the portal, navigate to "Metrics" → select your resource → add a condition → set the threshold. Pair the alert with an Action Group that posts to a Teams channel, so developers receive immediate feedback.

Routing .NET Logs to Grafana Loki

Replace the default ILogger provider with a Loki sink. The Serilog.Sinks.Grafana.Loki package translates structured logs into Loki’s JSON format. Below is a minimal configuration for appsettings.json and the corresponding C# bootstrap code.

{ "Serilog": { "Using": [ "Serilog.Sinks.Grafana.Loki" ], "MinimumLevel": "Information", "WriteTo": [ { "Name": "Loki", "Args": { "Url": "http://localhost:3100", "BatchSizeLimit": 100, "Period": "00:00:02" } } ] } }

using Serilog; var builder = WebApplication.CreateBuilder(args); Log.Logger = new LoggerConfiguration() .ReadFrom.Configuration(builder.Configuration) .CreateLogger(); builder.Host.UseSerilog(); var app = builder.Build(); app.MapGet("/weather", (ILogger logger) => { logger.LogInformation("Weather endpoint hit"); return Results.Ok(new[] { "Sunny", "Rainy" }); }); app.Run();

With this setup, every log entry includes trace and span IDs, enabling you to jump from a Grafana log line directly to the corresponding OpenTelemetry trace in Azure Monitor.

Practical Performance Tuning Using Observability Data

After the observability pipeline is live, start a load test with k6 or ApacheBench. Examine the trace waterfall in Azure Monitor: look for spans that exceed the 95th percentile latency. In many .NET services, the System.Text.Json serializer becomes a bottleneck under high concurrency. Switching to Utf8Json or enabling JsonSerializerOptions.DefaultBufferSize can shave 10‑15 ms per request.

Metrics often reveal memory pressure. If the GC.HeapSize metric grows steadily, consider using ArrayPool for large buffers or enabling GCSettings.LatencyMode = GCLatencyMode.SustainedLowLatency during peak periods. Loki logs can confirm whether GC pauses correlate with error spikes.

Closing the Loop: Continuous Improvement

The key to observability‑first tuning is iteration. Deploy a new configuration, let the telemetry run for at least 15 minutes, then compare the baseline KPIs: average request duration, 99th‑percentile latency, and error rate. Document each change in a markdown file linked to your CI pipeline, so future developers inherit the performance story.

Author: Mahmut Sarıkaya — sarikayadev.com

Sources

• Microsoft Docs – OpenTelemetry for .NET
• Azure Monitor documentation
• Grafana Loki official guide