Real-time Event Streaming with .NET 8, Kafka, and Kubernetes

Why Real-Time Event Streaming Matters

Imagine a retail platform that must react to a surge of 10,000 orders per minute during a flash sale. Traditional request‑response pipelines quickly become bottlenecks, while a streaming architecture can process each event the instant it arrives. Real‑time event streaming turns latency from seconds into milliseconds, enabling features such as instant inventory updates, fraud detection, and personalized recommendations. The combination of .NET 8, Apache Kafka, and Kubernetes provides a proven stack for building such low‑latency pipelines.

Getting Started with .NET 8 and Kafka

.NET 8 introduces native support for async streams and improved performance for high‑throughput networking, making it a natural fit for Kafka producers. Begin by adding the Confluent.Kafka NuGet package to a minimal API project. The following snippet creates a simple producer that publishes order events to a topic named orders:

using Confluent.Kafka;

var config = new ProducerConfig { BootstrapServers = "kafka:9092" };
using var producer = new ProducerBuilder<string, string>(config).Build();
await producer.ProduceAsync("orders", new Message<string, string> { Key = "order-123", Value = "created" });
Console.WriteLine("Message sent");

The key "order-123" guarantees ordering for that specific order across partitions, while the value contains a JSON payload in a real implementation. Deploying this code on .NET 8 ensures you benefit from the latest JIT optimizations and reduced allocation overhead.

Consuming Kafka Streams in C#

On the consumer side, .NET 8’s IAsyncEnumerable lets you process messages in a back‑pressure‑aware loop. The example below shows a resilient consumer that automatically commits offsets after each successful handling:

using Confluent.Kafka;

var consumerConfig = new ConsumerConfig
{
    BootstrapServers = "kafka:9092",
    GroupId = "order-service",
    AutoOffsetReset = AutoOffsetReset.Earliest,
    EnableAutoCommit = false
};
using var consumer = new ConsumerBuilder<string, string>(consumerConfig).Build();
consumer.Subscribe("orders");

await foreach (var msg in consumer.ConsumeAsync(CancellationToken.None))
{
    Console.WriteLine($"Processing {msg.Message.Key}: {msg.Message.Value}");
    // Insert business logic here, e.g., update database
    consumer.Commit(msg);
}

Because the loop runs asynchronously, the application can scale to thousands of concurrent partitions without blocking threads, a crucial advantage when you run multiple replicas inside Kubernetes.

Deploying to Kubernetes

Kubernetes provides the orchestration layer that guarantees high availability and horizontal scaling. A typical deployment for the consumer service might look like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-consumer
spec:
  replicas: 3
  selector:
    matchLabels:
      app: kafka-consumer
  template:
    metadata:
      labels:
        app: kafka-consumer
    spec:
      containers:
      - name: consumer
        image: myregistry/consumer:latest
        env:
        - name: KAFKA_BOOTSTRAP_SERVERS
          value: "kafka:9092"
        resources:
          limits:
            cpu: "500m"
            memory: "256Mi"

The replica count of three matches the three‑node Kafka cluster used in many production setups, ensuring each pod can read from a distinct partition. Remember to expose Kafka via a ClusterIP service and configure network policies so only authorized pods can connect.

Scaling and Observability Tips

When traffic spikes, increase the replica count or add more partitions to the orders topic. .NET 8’s built‑in metrics (via System.Diagnostics.Metrics) can be scraped by Prometheus; expose them with the dotnet-monitor sidecar. Pair this with Kafka’s JMX metrics to correlate consumer lag with CPU usage. A practical alert is to fire when consumer lag exceeds 5,000 messages for more than two minutes—this usually indicates a downstream bottleneck.

Conclusion

By leveraging .NET 8’s async capabilities, Apache Kafka’s durable log, and Kubernetes’ scaling primitives, you can build a resilient real‑time event streaming pipeline that processes millions of events per day. The key is to start with a lightweight producer, adopt async consumption patterns, and let Kubernetes handle the elasticity. With proper observability, the system remains transparent and can be tuned as business demands evolve.

Sources

• Confluent Kafka .NET Client Documentation
• Microsoft .NET 8 Release Notes
• Kubernetes Official Documentation

Author: Mahmut Sarıkaya — sarikayadev.com