Observability‑First .NET 8 Microservices with OpenTelemetry, Prometheus & Grafana

Why observability matters for modern .NET 8 microservices

When a request traverses five independent services, a single latency spike can inflate response time by 300 ms, a figure that 70% of e‑commerce sites consider a conversion killer. Developers therefore need a unified view of traces, metrics and logs before an incident reaches production. An observability‑first approach embeds telemetry at the code level, enabling automatic detection of bottlenecks, failed calls and resource exhaustion.

Setting up a minimal .NET 8 microservice

Start with .NET 8 SDK (version 8.0.100 or later) and create a new Web API project using the minimal‑API pattern. The project structure stays lightweight, which is ideal for containerized deployments.

dotnet new web -n OrderService --framework net8.0

Navigate into the folder and add the OpenTelemetry packages that will later feed Prometheus.

dotnet add package OpenTelemetry.Exporter.Prometheus.AspNetCore
dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.HttpClient

After restoring, the Program.cs file becomes the single place to configure services, telemetry and endpoints.

Instrumenting the service with OpenTelemetry

OpenTelemetry offers both tracing and metrics APIs. The following code registers automatic ASP.NET Core instrumentation, HTTP client instrumentation, and two exporters: Jaeger for distributed tracing and Prometheus for metrics.

using OpenTelemetry.Trace;
using OpenTelemetry.Metrics;
var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .WithTracing(trace => trace
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddJaegerExporter(jaegerOptions =>
        {
            jaegerOptions.AgentHost = "jaeger";
            jaegerOptions.AgentPort = 6831;
        }))
    .WithMetrics(metrics => metrics
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddPrometheusExporter());

var app = builder.Build();

app.MapPrometheusScrapingEndpoint(); // exposes /metrics for Prometheus
app.MapGet("/orders/{id}", (int id) => new { Id = id, Status = "Processed" });
app.Run();

Note the MapPrometheusScrapingEndpoint call – it creates the /metrics endpoint that Prometheus will scrape every 15 seconds by default.

Exporting metrics to Prometheus

Deploy Prometheus as a Docker container or on a Kubernetes node. The configuration file must point to the microservice’s /metrics path.

global:
  scrape_interval: 15s
scrape_configs:
  - job_name: "order_service"
    static_configs:
      - targets: ["order-service:5000"]

Start Prometheus with the custom configuration:

docker run -d \
  -p 9090:9090 \
  -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus

Once running, open http://localhost:9090/targets to verify that the order_service target is up and being scraped.

Visualizing data with Grafana

Grafana connects to Prometheus as a data source and provides ready‑made dashboards for .NET metrics such as dotnet_threadpool_thread_count or custom HTTP request latency histograms. Install Grafana and add the Prometheus URL (http://localhost:9090) as a data source.

docker run -d \
  -p 3000:3000 \
  --name=grafana \
  -e "GF_SECURITY_ADMIN_PASSWORD=admin" \
  grafana/grafana

Import the community “ASP.NET Core Overview” dashboard (ID 6417) and set the instance variable to order-service:5000. You will instantly see request rate, error count, and 95th‑percentile latency trends.

Practical tips and common pitfalls

1. **Avoid double‑instrumentation** – adding both manual ActivitySource calls and the automatic ASP.NET Core instrumentation can produce duplicate spans. Stick to one method unless you need custom attributes.
2. **Set realistic retention** – Prometheus defaults to 15‑day retention; for high‑traffic services reduce it to 7 days to keep storage costs low.
3. **Secure the metrics endpoint** – expose /metrics only inside the cluster or behind a mutual‑TLS gateway; otherwise attackers can infer internal load patterns.
4. **Use environment variables for exporter endpoints** – this keeps Docker images immutable and allows CI/CD pipelines to inject staging or production URLs without code changes.

Conclusion

By embedding OpenTelemetry in a .NET 8 minimal API, exposing a Prometheus‑compatible endpoint, and wiring Grafana dashboards, developers gain end‑to‑end visibility without sacrificing performance. The same pattern scales from a single container on a laptop to a fleet of services orchestrated by Kubernetes, turning observability from an afterthought into a foundational design principle.

Sources

OpenTelemetry .NET Documentation, Prometheus Official Docs, Grafana Labs Dashboard Gallery

Author: Mahmut Sarıkaya — sarikayadev.com