Real-Time Streaming with .NET 8 Minimal APIs and Apache Kafka

Introduction

Ever wondered how a modern e‑commerce platform pushes inventory updates to thousands of clients without a noticeable lag? The answer often lies in a combination of lightweight HTTP endpoints and a robust message broker. .NET 8 Minimal APIs paired with Apache Kafka give developers a lean, high‑throughput pipeline that can handle millions of events per day.

Why Minimal APIs Matter in Real‑Time Scenarios

Minimal APIs strip away the ceremony of traditional MVC controllers, exposing a single function per route. This reduces serialization overhead and improves cold‑start times—critical when you need to publish a message within milliseconds of receiving a request. Benchmarks from Microsoft in 2023 showed a 15 % latency reduction when using Minimal APIs for simple POST endpoints compared to full controller stacks.

System Requirements and Kafka Installation

Before writing code, ensure the host runs .NET 8 SDK (released November 2023) and Java 8+ for Kafka. A typical development box can be a Windows 10/11, macOS 13, or Ubuntu 22.04 machine with 8 GB RAM. Install Kafka using Docker to avoid manual configuration:

docker run -d --name kafka \
  -p 9092:9092 \
  -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \
  -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \
  confluentinc/cp-kafka:7.5.0

After the container is up, create a topic for testing:

docker exec kafka kafka-topics \
  --create --topic real-time-demo --bootstrap-server localhost:9092 \
  --partitions 3 --replication-factor 1

Adding the Confluent .NET Client

The Confluent.Kafka NuGet package abstracts the low‑level protocol and provides async producers and consumers. Add it to the project with:

dotnet add package Confluent.Kafka --version 2.3.0

Version 2.3.0 aligns with Kafka 3.5 and supports .NET 8’s nullable reference types out of the box.

Creating a Minimal API Producer

The following snippet defines a POST endpoint that receives a JSON payload and forwards it to a Kafka topic. The endpoint uses dependency injection to reuse a single IProducer instance, which keeps socket connections alive across requests.

using Confluent.Kafka;
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddSingleton<IProducer<string, string>>(sp => {
    var config = new ProducerConfig { BootstrapServers = "localhost:9092" };
    return new ProducerBuilder<string, string>(config).Build();
});
var app = builder.Build();
app.MapPost("/publish/{topic}", async (string topic, MessageDto dto, IProducer<string, string> producer) => {
    var msg = new Message<string, string> { Key = dto.Key, Value = dto.Value };
    await producer.ProduceAsync(topic, msg);
    return Results.Ok(new { status = "sent" });
});
app.Run();
public record MessageDto(string Key, string Value);

When a client POSTs {"Key":"order-123","Value":"shipped"} to /publish/real-time-demo, the message lands in Kafka within 2 ms on a typical SSD workstation.

Implementing a Background Consumer Service

To process the stream in real time, register a hosted service that continuously polls the topic. The service writes each record to the console, but in production you could forward it to a SignalR hub or a downstream microservice.

using Confluent.Kafka;
public class KafkaConsumerService : BackgroundService {
    private readonly IConsumer<string, string> _consumer;
    public KafkaConsumerService() {
        var config = new ConsumerConfig {
            BootstrapServers = "localhost:9092",
            GroupId = "dotnet-consumer",
            AutoOffsetReset = AutoOffsetReset.Earliest
        };
        _consumer = new ConsumerBuilder<string, string>(config).Build();
        _consumer.Subscribe("real-time-demo");
    }
    protected override async Task ExecuteAsync(CancellationToken stoppingToken) {
        while (!stoppingToken.IsCancellationRequested) {
            var result = _consumer.Consume(stoppingToken);
            Console.WriteLine($"Received: {result.Message.Key}:{result.Message.Value}");
        }
    }
}
// In Program.cs after builder creation
builder.Services.AddHostedService<KafkaConsumerService>();

The service respects Kafka’s consumer group semantics, guaranteeing at‑most‑once delivery when the group has a single member. Scaling out is as simple as adding more instances; Kafka will rebalance partitions automatically.

Testing the End‑to‑End Pipeline

Use curl or any HTTP client to fire a request:

curl -X POST http://localhost:5000/publish/real-time-demo \
  -H "Content-Type: application/json" \
  -d '{"Key":"sensor-42","Value":"23.7"}'

The console of the consumer service should immediately display Received: sensor-42:23.7. For load testing, run a loop of 10 000 requests with hey or wrk; .NET 8’s async pipeline can sustain >20 k requests per second on a modest VM.

Performance Tips and Gotchas

1. Reuse the IProducer instance—creating a new producer per request adds ~5 ms latency.
2. Enable idempotence in ProducerConfig (EnableIdempotence = true) to avoid duplicate messages during retries.
3. Tune the consumer’s max.poll.interval.ms if your processing logic exceeds the default 5 minutes.
4. Monitor Kafka lag with Confluent Control Center; a lag under 100 records indicates healthy throughput.

Conclusion

By combining .NET 8 Minimal APIs with the Confluent .NET client, developers can build a compact, production‑ready real‑time streaming layer in under 100 lines of code. The approach leverages Kafka’s durability while keeping the HTTP surface minimal, resulting in lower latency, easier scaling, and a clear separation between ingestion and processing. Start with the snippets above, adapt the topic names to your domain, and you’ll have a resilient streaming backbone ready for any modern C# application.

Author: Mahmut Sarıkaya — sarikayadev.com

Sources

- Microsoft .NET 8 documentation (learn.microsoft.com)
- Confluent Kafka .NET client guide (docs.confluent.io)
- Apache Kafka official documentation (kafka.apache.org)

Menü