Self‑Healing .NET 8 Microservices with Kubernetes Operators and Polly

Why self‑healing matters for .NET 8 microservices

Imagine a retail API that processes 15,000 requests per minute during a flash sale. A single pod crash can drop the success rate below the SLA threshold of 99.9%. In .NET 8 the runtime offers faster startup and native AOT, but without an automated recovery loop the whole transaction chain stalls. Self‑healing architecture turns that failure into a brief hiccup by detecting the fault, restarting the service, and re‑routing traffic without human intervention.

Building resilience with Polly

Polly is a lightweight .NET library that lets you express retry, circuit‑breaker, timeout and fallback policies as fluent code. A typical retry‑with‑exponential‑backoff for HTTP calls looks like this:

var retryPolicy = Policy<HttpResponseMessage>.Handle<HttpRequestException>()
    .WaitAndRetryAsync(3, attempt => TimeSpan.FromSeconds(Math.Pow(2, attempt)),
        (exception, timespan, context) =>
        {
            Log.Warn($"Retry {context["RetryCount"]} after {timespan.TotalSeconds}s: {exception.Message}");
        });

Combine it with a circuit‑breaker to protect downstream services during prolonged outages:

var circuitBreaker = Policy<HttpResponseMessage>.HandleResult(r => !r.IsSuccessStatusCode)
    .CircuitBreakerAsync(5, TimeSpan.FromMinutes(1),
        onBreak: (outcome, breakDelay) => Log.Error($"Circuit opened for {breakDelay.TotalSeconds}s"),
        onReset: () => Log.Info("Circuit closed"));

These policies are registered in the DI container and injected wherever HttpClient is used, guaranteeing consistent behavior across all microservices.

Kubernetes Operators as the automation layer

Kubernetes Operators extend the control plane with custom resources that encapsulate domain‑specific logic. For a .NET 8 microservice, an operator can watch Deployment health, evaluate custom metrics, and issue a rolling restart when a readiness probe fails repeatedly.

apiVersion: apps.example.com/v1
kind: DotNetService
metadata:
  name: order‑processor
spec:
  image: myregistry.com/order‑processor:8.0.0
  replicas: 3
  healthCheck:
    failureThreshold: 5
    periodSeconds: 10

The operator’s controller (often written in Go or C# with the Operator SDK) implements a Reconcile loop that reads the failureThreshold. If the pod’s readiness probe fails five times, the controller patches the Deployment with a new annotation, forcing Kubernetes to recreate the pod. This pattern eliminates manual kubectl commands and aligns recovery with business‑level SLAs.

Putting it together: a step‑by‑step recipe

1. **Create a .NET 8 Web API** – use dotnet new webapi -f net8.0 and enable health checks via services.AddHealthChecks().
2. **Add Polly policies** – register IAsyncPolicy<HttpResponseMessage> in Program.cs and wrap outgoing HttpClient calls.
3. **Containerize** – write a multi‑stage Dockerfile that builds with dotnet publish -c Release -o /app and copies the trimmed output to a mcr.microsoft.com/dotnet/aspnet:8.0 runtime image.
4. **Deploy with a custom resource** – apply the YAML shown above; the Operator will create the underlying Deployment and Service automatically.
5. **Configure the Operator** – set the failureThreshold to match the retry count used in Polly, creating a feedback loop between application‑level resilience and cluster‑level self‑healing.
6. **Observe** – expose Prometheus metrics from both Polly (via Polly.Extensions.Logging) and the Operator (custom metrics endpoint). Alerting rules can trigger a Slack notification if the circuit breaker opens more than twice in ten minutes.

Monitoring and automatic remediation

Beyond the Operator’s restart logic, a sidecar that runs kubectl top pod can feed CPU and memory trends into a HorizontalPodAutoscaler (HPA). When the HPA scales up, the Operator re‑evaluates the health policy and disables the restart throttle to avoid thrashing. Combining these signals with Azure Monitor dashboards gives a single pane of glass for developers and SREs.

Sources

Microsoft Docs – .NET 8 Release Notes; Polly Project – Official Documentation; Kubernetes.io – Custom Resource Definitions and Operator Pattern.

Author: Mahmut Sarıkaya — sarikayadev.com