Fine-Tuning Small Language Models (SLMs) for Low-Latency Edge Inference

Cloud LLM APIs introduce latency (often 800ms - 2500ms) and ongoing cost bottlenecks for high-frequency user actions. For domain-specific tasks like data classification or structured parsing, small fine-tuned models outperform generalized large models.

We detail our quantization protocol: converting PyTorch model weights to 4-bit and 8-bit representations, profiling execution via ONNX Runtime, and running on-device mobile inference in under 40 milliseconds.