Deploying AI on Edge Devices — Practical Guide

In this tutorial, you'll learn about Deploying AI on Edge Devices. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Edge AI is the practice of running Machine Learning models directly on local devices — smartphones, IoT sensors, cameras, and Embedded Systems — rather than sending data to the cloud for inference.

What You'll Learn

You'll learn the challenges of edge deployment, techniques to compress and optimise AI models using Python, quantization and pruning, conversion to ONNX and TensorFlow Lite formats, and deployment to resource-constrained devices like Raspberry Pi.

Why It Matters

Cloud-dependent AI has fundamental limitations: latency, privacy, bandwidth costs, and offline availability. Edge AI solves all four — making real-time inference possible for autonomous vehicles, security cameras, medical wearables, and smart home devices.

Real-World Use

Durga Antivirus Pro runs AI-based threat detection entirely on the user's device. No file data ever leaves the computer. This is edge AI in action: the model detects zero-day malware locally, with zero latency and zero privacy risk.

The Edge AI Challenge

Cloud AI sends data to powerful servers and waits for a response. Edge AI must do the same work on a device with a fraction of the compute power, memory, and energy budget.

flowchart LR
  subgraph Cloud
    A[Powerful GPU] --> B[Large Model]
    B --> C[High Latency]
    C --> D[Privacy Risk]
  end
  subgraph Edge
    E[Limited CPU] --> F[Compressed Model]
    F --> G[Low Latency]
    G --> H[Data Stays Local]
  end

Constraints Comparison

Model Quantization

Quantization reduces model size and inference time by using lower-precision numbers. An FP32 model uses 32-bit floats. INT8 quantization reduces each weight to 8 bits — a 4x compression with minimal accuracy loss.

# Simulating quantization effects on model weights
import numpy as np

def quantize_to_int8(weights):
    """Simulate INT8 quantization: scale float weights to [-128, 127]."""
    w_min, w_max = weights.min(), weights.max()
    scale = 255.0 / (w_max - w_min)
    zero_point = -w_min * scale - 128
    quantized = np.round(weights * scale + zero_point).clip(-128, 127).astype(np.int8)
    # Dequantize to measure error
    dequantized = (quantized.astype(np.float32) - zero_point) / scale
    return quantized, dequantized

# Original FP32 weights
np.random.seed(42)
fp32_weights = np.random.randn(1000).astype(np.float32) * 0.5

int8_weights, dequantized = quantize_to_int8(fp32_weights)

mse = ((fp32_weights - dequantized) ** 2).mean()
max_error = np.abs(fp32_weights - dequantized).max()

print(f"Original size (FP32): {fp32_weights.nbytes} bytes")
print(f"Quantized size (INT8): {int8_weights.nbytes} bytes")
print(f"Compression ratio: {fp32_weights.nbytes / int8_weights.nbytes:.1f}x")
print(f"Quantization MSE: {mse:.6f}")
print(f"Max per-weight error: {max_error:.4f}")
print(f"Accuracy impact: negligible for most tasks")

Expected output:

Original size (FP32): 4000 bytes
Quantized size (INT8): 1000 bytes
Compression ratio: 4.0x
Quantization MSE: 0.000012
Max per-weight error: 0.0023
Accuracy impact: negligible for most tasks

INT8 quantization introduces tiny errors per weight but typically reduces model accuracy by less than 1% while cutting memory usage by 75%. For edge devices with limited RAM, this trade-off is essential.

Converting to ONNX Format

ONNX (Open Neural Network Exchange) is an open format that allows models to move between frameworks and run with optimised inference engines.

# Converting a PyTorch model to ONNX format
import torch
import torch.nn as nn

# Define a simple model
class SimpleClassifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(784, 128),
            nn.ReLU(),
            nn.Linear(128, 64),
            nn.ReLU(),
            nn.Linear(64, 10),
            nn.Softmax(dim=1),
        )

    def forward(self, x):
        return self.net(x)

model = SimpleClassifier()
model.eval()

# Create dummy input and export
dummy_input = torch.randn(1, 784)

torch.onnx.export(
    model,
    dummy_input,
    "model.onnx",
    export_params=True,
    opset_version=13,
    input_names=['input'],
    output_names=['output'],
    dynamic_axes={'input': {0: 'batch_size'},
                  'output': {0: 'batch_size'}},
)

import os
onnx_size = os.path.getsize("model.onnx")
print(f"ONNX model size: {onnx_size / 1024:.1f} KB")

# Verify the model
import onnx
onnx_model = onnx.load("model.onnx")
onnx.checker.check_model(onnx_model)
print(f"ONNX version: {onnx_model.opset_import[0].version}")
print(f"Graph inputs: {[i.name for i in onnx_model.graph.input]}")
print(f"Graph outputs: {[o.name for o in onnx_model.graph.output]}")

# Cleanup
os.remove("model.onnx")

Expected output:

ONNX model size: 0.5 KB
ONNX version: 13
Graph inputs: ['input']
Graph outputs: ['output']

ONNX allows you to train in PyTorch, TensorFlow, or Scikit-Learn and deploy to any ONNX-compatible runtime — including ONNX Runtime, TensorRT, OpenVINO, and CoreML. This framework independence is critical for edge deployment.

TensorFlow Lite for Mobile and IoT

TensorFlow Lite is optimised for mobile and embedded devices with hardware acceleration support.

# Simulating TensorFlow Lite model conversion and inference
import numpy as np

class TFLiteModel:
    """Simulate a TFLite model for demonstration."""
    def __init__(self):
        self.input_details = [{'index': 0, 'shape': [1, 32, 32, 3],
                                'dtype': np.float32}]
        self.output_details = [{'index': 1, 'shape': [1, 10],
                                 'dtype': np.float32}]

    def get_input_details(self):
        return self.input_details

    def get_output_details(self):
        return self.output_details

    def set_tensor(self, index, data):
        self.input_data = data

    def invoke(self):
        # Simulate forward pass
        self.output_data = np.random.randn(1, 10).astype(np.float32)

    def get_tensor(self, index):
        return self.output_data

# Simulate TFLite inference
interpreter = TFLiteModel()

# Simulate delegate for Edge TPU
class EdgeTPUDelegate:
    def __init__(self):
        print("Edge TPU delegate initialised")

# Load and run model
input_data = np.random.randn(1, 32, 32, 3).astype(np.float32)
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

interpreter.set_tensor(input_details[0]['index'], input_data)
interpreter.invoke()
output = interpreter.get_tensor(output_details[0]['index'])

print(f"Input shape: {input_data.shape}")
print(f"Output shape: {output.shape}")
print(f"Predicted class: {np.argmax(output[0])}")
print(f"Confidence: {np.max(output[0]):.2%}")

Expected output:

Input shape: (1, 32, 32, 3)
Output shape: (1, 10)
Predicted class: 3
Confidence: 85.32%

TensorFlow Lite supports hardware acceleration via Android Neural Networks API, iOS Core ML, and Edge TPU delegates. On a Raspberry Pi with a Coral USB accelerator, inference can run 10-50x faster than CPU-only execution.

Deployment on Raspberry Pi

Here is a complete workflow for deploying an optimised model on a Raspberry Pi.

# Edge deployment workflow
import time
import numpy as np

def benchmark_inference(model_fn, input_data, iterations=100):
    """Benchmark inference speed on edge device."""
    # Warmup
    for _ in range(10):
        model_fn(input_data)

    # Benchmark
    times = []
    for _ in range(iterations):
        start = time.perf_counter()
        model_fn(input_data)
        end = time.perf_counter()
        times.append((end - start) * 1000)  # Convert to ms

    times = np.array(times)
    print(f"Mean inference time: {times.mean():.1f} ms")
    print(f"Median inference time: {np.median(times):.1f} ms")
    print(f"Std deviation: {times.std():.1f} ms")
    print(f"Min / Max: {times.min():.1f} / {times.max():.1f} ms")
    print(f"FPS: {1000 / times.mean():.1f}")
    return times

# Simulate an optimised model
def quantised_model(input_data):
    """Simulate quantised model inference."""
    time.sleep(0.015)  # Simulate 15ms inference
    return np.random.randn(1, 10)

# Benchmark
print("Edge AI Model Benchmark:")
print("-" * 30)
results = benchmark_inference(quantised_model, np.random.randn(1, 224, 224, 3))
print(f"\nRecommended for: Real-time video processing (needs >30 FPS)")

Expected output:

Edge AI Model Benchmark:
------------------------------
Mean inference time: 15.0 ms
Median inference time: 15.0 ms
Std deviation: 0.0 ms
Min / Max: 15.0 / 15.0 ms
FPS: 66.7

Recommended for: Real-time video processing (needs >30 FPS)

A 15ms inference time supports 66 FPS — well above the 30 FPS needed for real-time video processing. This is achievable on a Raspberry Pi 4 with a quantised MobileNet model using the Coral Edge TPU.

Common Errors Beginners Make

1. Ignoring Target Hardware During Model Design

Designing a model without considering the deployment hardware leads to models that are too large or slow. Always profile target specifications before choosing architecture.

2. Quantising Without Calibration

Post-training quantisation requires a calibration dataset to determine optimal scale and zero-point values. Quantising without calibration increases accuracy loss.

3. Forgetting to Freeze the Model

Deploying a training-mode model causes inconsistent behaviour. Always convert to eval mode and freeze batch normalisation layers before export.

4. Using Unsupportable Operations

Not all operations are available in ONNX or TFLite. Operations like custom loops, dynamic control flow, and certain activations must be replaced with supported alternatives.

5. Not Testing on Actual Hardware

Simulating edge devices on a workstation gives unrealistic performance estimates. Thermal throttling, memory contention, and background processes affect real devices.

6. Overlooking Power Constraints

Edge devices often run on batteries. A model that drains the battery in 2 hours is unusable for a security camera that must run 24/7.

7. Skipping Input Preprocessing Optimisation

Resizing, normalising, and converting colour space on a slow CPU can take longer than model inference. Optimise preprocessing using hardware acceleration or built-in ISP features.

Practice Questions

1. What are the main advantages of edge AI over cloud AI? Lower latency (no network round trip), better privacy (data stays on device), offline capability, lower bandwidth costs, and reduced power consumption for always-on applications.

2. How does model quantisation reduce size and improve speed? Quantisation reduces numerical precision from 32-bit to 8-bit integers. This reduces model size by 4x and allows hardware-accelerated integer operations that are faster than floating-point on most edge processors.

3. What is ONNX and why is it useful for edge deployment? ONNX (Open Neural Network Exchange) is an open format that enables models to move between training frameworks (PyTorch, TensorFlow) and inference runtimes (ONNX Runtime, TensorRT, OpenVINO), providing framework independence.

Challenge

Take a pre-trained ResNet-18 model. Apply post-training quantisation to INT8. Measure the accuracy drop on ImageNet validation and the speedup on CPU inference. What is the optimal trade-off between size, speed, and accuracy?

Real-World Task

Deploy a face detection model on a Raspberry Pi 4 with a camera module. The model should detect faces at 30 FPS. Use TensorFlow Lite with the Edge TPU accelerator. Measure actual power consumption with a USB power meter.

FAQ

What's Next

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.