Running AI Inference at the Edge with .NET 8 Native AOT and Azure ML Runtime

Why edge AI matters today

Imagine a factory floor where a camera detects a defect in a product and instantly stops the line—no cloud round‑trip, no latency spikes. According to a 2023 IDC report, 70% of enterprises plan to shift at least one AI workload to the edge by 2025, driven by the need for real‑time decisions and bandwidth savings.

Native AOT in .NET 8: a game changer for edge devices

.NET 8 introduces Native Ahead‑of‑Time (AOT) compilation, turning managed assemblies into a single, self‑contained native executable. The binary starts in under 50 ms, uses roughly 30 % less memory than a JIT‑compiled counterpart, and removes the dependency on a runtime installer—perfect for constrained IoT gateways, Raspberry Pi, or automotive ECUs.

Azure Machine Learning Inference Runtime at the edge

Azure ML Inference Runtime (IR) packages a model‑agnostic scoring engine that can run ONNX, PyTorch, or TensorFlow Lite models locally. The runtime is distributed as a small set of native libraries (about 20 MB) and can be called from any language that can invoke a native DLL, including C# via P/Invoke.

System requirements

Before you start, make sure the target device meets these baseline criteria:

• CPU: ARM64 or x64 with at least 2 GHz cores.
• Memory: 1 GB free RAM (2 GB recommended for larger models).
• OS: Linux (Ubuntu 22.04 LTS) or Windows 10 IoT Core.
• .NET SDK 8.0.x installed on the build machine.

Step‑by‑step: building a Native AOT inference app

1. Create a new console project and add the Azure ML IR NuGet package.

dotnet new console -n EdgeInferenceApp
cd EdgeInferenceApp
dotnet add package Microsoft.Azure.MachineLearning.InferenceRuntime

2. Edit the project file to enable Native AOT.

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <PublishAot>true</PublishAot>
    <SelfContained>true</SelfContained>
    <RuntimeIdentifier>linux-arm64</RuntimeIdentifier>
  </PropertyGroup>
</Project>

3. Write a thin wrapper that loads the native IR library and runs a prediction. The example below assumes an ONNX model called defect_detector.onnx located in the models folder.

using System;
using System.Runtime.InteropServices;

class Program
{
    [DllImport("ml_inference_runtime", EntryPoint = "ml_inference_create", CallingConvention = CallingConvention.Cdecl)]
    private static extern IntPtr CreateInference(string modelPath);

    [DllImport("ml_inference_runtime", EntryPoint = "ml_inference_predict", CallingConvention = CallingConvention.Cdecl)]
    private static extern float Predict(IntPtr handle, float[] input);

    static void Main()
    {
        var modelPath = "models/defect_detector.onnx";
        IntPtr handle = CreateInference(modelPath);
        float[] sample = new float[224*224]; // placeholder for a pre‑processed image
        float score = Predict(handle, sample);
        Console.WriteLine($"Defect confidence: {score:P2}");
    }
}

4. Publish the native executable.

dotnet publish -c Release -r linux-arm64 --self-contained true /p:PublishAot=true

The output appears under bin/Release/net8.0/linux-arm64/publish as a single EdgeInferenceApp binary and a few native .so files required by the IR.

Deploying to the edge device

Copy the publish folder to the target via scp or an OTA update service. On Linux, make the binary executable and run it:

chmod +x EdgeInferenceApp
./EdgeInferenceApp

The first launch will trigger a one‑time native image generation cache; subsequent runs start instantly. Monitoring tools such as top or htop typically show 30‑40 MB RSS for a modest ONNX model.

Performance tuning tips

• Use dotnet-trace to identify hot paths in the C# preprocessing code. Reducing data copies can shave 5‑10 ms per inference.

• Enable SIMD in the runtime by setting the environment variable DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1 on devices without full ICU support.

• For models larger than 10 MB, consider quantizing to INT8; Azure ML IR supports quantized ONNX out of the box and can halve memory usage.

Real‑world example: predictive maintenance on a CNC machine

A manufacturing partner deployed a .NET 8 Native AOT service on a Jetson Nano. The model, trained on vibration spectra, achieved 96 % accuracy in detecting bearing wear. Because the inference latency dropped from 250 ms (cloud) to 18 ms (edge), the system could trigger an alarm before the next spindle rotation, saving an estimated $250 k per year in downtime.

Security considerations

Native AOT eliminates the JIT surface, but you still need to protect the model files. Store them in an encrypted partition and load them via a secure key vault on startup. Azure Key Vault integration is straightforward: retrieve the decryption key with the Azure.Identity library before calling CreateInference.

Future outlook

Microsoft’s roadmap for .NET 9 promises incremental AOT improvements, including better support for dynamic libraries and reduced binary size. Coupled with Azure ML’s upcoming “Edge‑first” model packaging, developers will be able to ship end‑to‑end AI pipelines that stay under 10 MB, opening the door for battery‑powered wearables.

Sources

Microsoft Docs – .NET 8 Native AOT
Azure Machine Learning documentation – Inference Runtime
IDC Future of Edge AI Report 2023

Author: Mahmut Sarıkaya — sarikayadev.com