Case study 02 / Edge inference / latency
Meteor Lake Latency Lab
Which processor on one laptop chip should run live instance segmentation inside a 33.3 ms frame budget? Flat-out, the Arc iGPU had the worse p99 of the two accelerators; on a live camera loop it had the better one.
The short version
- On a live webcam loop, INT8 on the Arc iGPU ran a 10.4 ms median and 13.3 ms p99, ahead of the NPU at 18.0 and 22.3 ms. It missed the 33.3 ms budget more often: 8 of 2,398 frames (0.33%) against 1 of 1,891 (0.05%).
- In a flat-out benchmark the tails reversed: the iGPU's p99 was 47.0 ms and the NPU's 18.0 ms. Which one has the better tail depends on how busy the accelerator is kept.
- The iGPU's flat-out tail came from CPU-side stalls, not the model. Without the stall frames its INT8 p99 is 17.1 ms, and inference alone was 9.5 ms at p99.
- Keeping the detection-head decode in float held the INT8 accuracy loss on the CPU to 0.0015 box mAP. Quantizing it too cost 0.0389.
Setup
The hardware is one laptop with an Intel Core Ultra 7 155H on mains power: the Intel AI Boost NPU, the Arc integrated GPU and the CPU, which share one package power budget. The software is OpenVINO 2026.3 and NNCF on Windows 11.
The model is YOLOv8n-seg at a fixed 640 × 640 input, exported to OpenVINO in FP32 and FP16, plus two INT8 builds made with NNCF post-training quantization on 300 COCO val2017 images. One keeps the detection-head decode in float and the other quantizes everything. Accuracy was measured on the other 4,700 images, so calibration and evaluation never overlap.
In the benchmark, every configuration discarded 100 warm-up frames and then timed 1,000 frames one at a time, with a 45 s cooldown between runs. End-to-end latency means preprocess, inference and the host-side postprocess (NumPy decode with OpenCV NMS); capture and drawing are not timed.
Code:run_all.py,bench.py,quantize.py,prepare_data.py,device_info.json
For the live loop, INT8 on the iGPU and on the NPU ran against a real webcam for one 60 s session each. The loop does not wait for a new camera frame, so the iGPU processed 2,398 frames in its session, about 40 a second: busier than a strict 30 fps camera, and far from flat-out.
Live loop:live_demo.py L164-172,live_results.json L42-51,README.md L181-181
Findings
The benchmark and the camera reversed the p99 order
Hollow dots are the flat-out benchmark (1,000 frames each); filled dots are the live webcam loop (2,398 iGPU frames, 1,891 NPU frames). The readout gives flat-out / live. The vertical line is the 33.3 ms frame budget.
Show the numbers as a table
| Processor | Regime | Frames | Median (ms) | p99 (ms) | Frames over 33.3 ms |
|---|---|---|---|---|---|
| Arc iGPU | Flat-out | 1,000 | 11.30 | 47.02 | 1.6% |
| Arc iGPU | Live | 2,398 | 10.42 | 13.31 | 0.33% |
| NPU | Flat-out | 1,000 | 13.93 | 17.98 | 0.2% |
| NPU | Live | 1,891 | 18.04 | 22.28 | 0.05% |
Flat-out, the iGPU had the better median but the worse tail of the two, and the NPU looked like the safe real-time choice. On the camera loop, where the iGPU spends much of each frame interval idle, its tail collapsed and it beat the NPU on both median and p99, although the NPU still missed the budget less often. The NPU got slower live, not faster. A study that reported only the benchmark would have described headroom, not this workload.
The iGPU tail was CPU-side stalls, not inference
The frame strip is drawn with JavaScript. The readout gives the whole run, and every frame is in the linked source file.
| Processor | Median (ms) | p99 (ms) | Median inference (ms) | Frames over 33.3 ms | Stall frames |
|---|---|---|---|---|---|
| Arc iGPU | 11.30 | 47.02 | 7.25 | 16 | 17 |
| NPU | 13.93 | 17.98 | 10.65 | 2 | 2 |
| CPU | 32.63 | 49.01 | 28.82 | 463 | 1 |
| Processor | Frame | Preprocess | Inference | Postprocess | End to end |
|---|---|---|---|---|---|
| Arc iGPU | 37 | 2.83 | 8.06 | 35.70 | 46.59 |
| Arc iGPU | 92 | 3.11 | 7.31 | 72.69 | 83.11 |
| Arc iGPU | 109 | 2.76 | 7.15 | 36.41 | 46.32 |
| Arc iGPU | 253 | 3.50 | 7.55 | 36.59 | 47.64 |
| Arc iGPU | 361 | 2.97 | 7.49 | 36.58 | 47.04 |
| Arc iGPU | 413 | 2.64 | 8.40 | 38.24 | 49.28 |
| Arc iGPU | 414 | 3.26 | 9.64 | 76.99 | 89.89 |
| Arc iGPU | 416 | 3.38 | 8.65 | 110.26 | 122.29 |
| Arc iGPU | 420 | 3.76 | 9.28 | 73.40 | 86.44 |
| Arc iGPU | 428 | 3.04 | 8.88 | 122.63 | 134.55 |
| Arc iGPU | 434 | 4.20 | 9.62 | 16.42 | 30.24 |
| Arc iGPU | 507 | 2.70 | 7.50 | 34.22 | 44.42 |
| Arc iGPU | 509 | 3.04 | 19.02 | 34.32 | 56.38 |
| Arc iGPU | 518 | 2.97 | 6.89 | 35.88 | 45.74 |
| Arc iGPU | 590 | 2.61 | 7.62 | 35.51 | 45.74 |
| Arc iGPU | 859 | 2.43 | 6.26 | 46.46 | 55.15 |
| Arc iGPU | 956 | 2.09 | 5.98 | 38.95 | 47.02 |
| NPU | 30 | 3.53 | 10.83 | 74.13 | 88.49 |
| NPU | 410 | 2.11 | 10.74 | 71.68 | 84.53 |
| CPU | 419 | 2.85 | 47.10 | 35.99 | 85.94 |
A stall is a frame whose CPU-side stages (preprocess plus postprocess) took more than 20 ms. Hollow dots include every frame; filled dots leave the stall frames out. The readout gives all frames / without stalls.
Show the numbers as a table
| Configuration | Stall frames (of 1,000) | p99, all frames (ms) | p99 without stalls (ms) | Inference-only p99 (ms) |
|---|---|---|---|---|
| Arc iGPU FP32 | 17 | 50.50 | 17.45 | 10.36 |
| Arc iGPU FP16 | 8 | 18.92 | 17.30 | 10.10 |
| Arc iGPU INT8 | 17 | 47.02 | 17.06 | 9.51 |
| Arc iGPU INT8 full | 21 | 49.00 | 15.05 | 8.22 |
| NPU FP32 | 0 | 17.94 | 17.94 | 12.79 |
| NPU FP16 | 2 | 19.01 | 18.92 | 13.68 |
| NPU INT8 | 2 | 17.98 | 17.85 | 12.21 |
| NPU INT8 full | 2 | 16.77 | 16.59 | 11.88 |
Across its four builds the iGPU produced 63 stall frames in 4,000, the NPU 6 and the CPU 1. Take the stall frames out and three of the four iGPU tails fall from around 50 ms to 15 to 17.5 ms, while inference-only p99 on the same runs stayed near 10 ms. Faster inference would not have fixed these frames.
Stall counts:stall_analysis.json L161-178
Why the iGPU runs stall is not isolated here. Display compositor contention, the shared package power budget and driver threads are all candidates, and separating them would need ETW tracing.
What INT8 cost
COCO val2017, 4,700 images not used for calibration. The FP32 reference was scored on the CPU, so the iGPU and NPU rows combine a precision change with a device change.
Show the numbers as a table
| Build | Scored on | Box mAP50-95 | Box change vs FP32 | Mask mAP50-95 | Mask change vs FP32 |
|---|---|---|---|---|---|
| FP32 | CPU | 0.3602 | reference | 0.3004 | reference |
| FP16 | CPU | 0.3602 | 0.0000 | 0.3005 | +0.0001 |
| INT8, head in float | CPU | 0.3587 | −0.0015 | 0.2985 | −0.0019 |
| INT8, fully quantized | CPU | 0.3213 | −0.0389 | 0.2872 | −0.0132 |
| INT8, head in float | Arc iGPU | 0.3578 | −0.0024 | 0.2990 | −0.0014 |
| INT8, head in float | NPU | 0.3579 | −0.0023 | 0.2990 | −0.0014 |
With the detection-head decode kept in float, INT8 cost 0.0015 box mAP on the CPU. Quantizing that decode as well cost 0.0389, about 26 times as much. What that bought in speed depended on the processor. On the NPU the fully quantized build's median was only 0.35 ms faster (13.58 against 13.93 ms). On the iGPU its median was 1.33 ms faster, but its flat-out p99 was worse (49.00 against 47.02 ms). On the CPU it cut the frames over budget from 46.3% to 4.8%. The same INT8 weights scored within 0.0009 of each other on the CPU, iGPU and NPU.
Latency by build:summary_main.json L50-97,summary_main.json L146-193,summary_main.json L242-289
What this does not show
- Power was not measured, so the efficiency argument for the NPU is not tested here.
- The stall mechanism is not attributed.
- One model and one input size, on one machine at room temperature and on mains power.
- Only the two INT8 configurations were run live, for one 60 s session each, in one lighting condition with one person in frame. A busier scene changes postprocess cost.
Limits stated in the repository:README.md L166-184
- The live loop does not wait for new camera frames, so its duty cycle sits between a 30 fps camera and flat-out rather than matching either.
- The repository includes a script that checks the hand-written postprocess against the Ultralytics decoder, but its output is not committed, so no result from it is quoted here.
Added for this write-up:live_demo.py,verify_postprocess.py