Picking a quantization level for browser inference
Quantization is the single biggest lever on whether a model is usable in a browser. fp32 is the reference: largest download, always supported. q8 typically cuts the download by four with quality differences most users cannot detect on encoder models. q4 halves it again and is how a 1.7B chat model fits in a browser at all, at a visible cost in coherence. fp16 and q4f16 look attractive on paper and fail outright on adapters that do not expose the WebGPU "shader-f16" feature — which includes plenty of current desktop GPUs. This page explains what to choose per task and how the playground falls back safely.
onnx quantization comparisonq4 vs q8 qualityfp16 webgpu shader-f16model quantization browserint8 quantization onnxq4f16 not supportedreduce model size inference
Frequently asked questions
Why does a q4f16 model fail to load on my machine?
Because your WebGPU adapter does not advertise the "shader-f16" feature, so half-precision shaders cannot be compiled. This site probes the feature and never silently hands you an f16 build: models with a non-f16 option fall back to q4 or q8 automatically, and the two models published only as f16 are blocked before the download starts, with the reason named.
Which models here need shader-f16, and why only those?
Qwen3 4B and Phi-3.5 Mini. Every other model in the catalogue ships a q4 or q8 build that runs on any WebGPU adapter — but above about 3 billion parameters the publishers stop producing non-f16 ONNX exports, so f16 is the only build that exists. That is a packaging limit, not a hardware one.
Does quantization make inference faster or just smaller?
Both, but the download saving is the reliable part. On WebGPU the speed-up from int8 is real but smaller than on CPU, because the dequantization work partly offsets the reduced memory traffic.