Case study 02 / Edge inference / latency

Meteor Lake Latency Lab

Which processor on one laptop chip should run live instance segmentation inside a 33.3 ms frame budget? Flat-out, the Arc iGPU had the worse p99 of the two accelerators; on a live camera loop it had the better one.

  • August 2026
  • Python, OpenVINO 2026.3, NNCF
  • 12 benchmark configurations and 2 live runs

The short version

  • On a live webcam loop, INT8 on the Arc iGPU ran a 10.4 ms median and 13.3 ms p99, ahead of the NPU at 18.0 and 22.3 ms. It missed the 33.3 ms budget more often: 8 of 2,398 frames (0.33%) against 1 of 1,891 (0.05%).
  • In a flat-out benchmark the tails reversed: the iGPU's p99 was 47.0 ms and the NPU's 18.0 ms. Which one has the better tail depends on how busy the accelerator is kept.
  • The iGPU's flat-out tail came from CPU-side stalls, not the model. Without the stall frames its INT8 p99 is 17.1 ms, and inference alone was 9.5 ms at p99.
  • Keeping the detection-head decode in float held the INT8 accuracy loss on the CPU to 0.0015 box mAP. Quantizing it too cost 0.0389.

Setup

The hardware is one laptop with an Intel Core Ultra 7 155H on mains power: the Intel AI Boost NPU, the Arc integrated GPU and the CPU, which share one package power budget. The software is OpenVINO 2026.3 and NNCF on Windows 11.

The model is YOLOv8n-seg at a fixed 640 × 640 input, exported to OpenVINO in FP32 and FP16, plus two INT8 builds made with NNCF post-training quantization on 300 COCO val2017 images. One keeps the detection-head decode in float and the other quantizes everything. Accuracy was measured on the other 4,700 images, so calibration and evaluation never overlap.

In the benchmark, every configuration discarded 100 warm-up frames and then timed 1,000 frames one at a time, with a 45 s cooldown between runs. End-to-end latency means preprocess, inference and the host-side postprocess (NumPy decode with OpenCV NMS); capture and drawing are not timed.

Code:run_all.py,bench.py,quantize.py,prepare_data.py,device_info.json

For the live loop, INT8 on the iGPU and on the NPU ran against a real webcam for one 60 s session each. The loop does not wait for a new camera frame, so the iGPU processed 2,398 frames in its session, about 40 a second: busier than a strict 30 fps camera, and far from flat-out.

Live loop:live_demo.py L164-172,live_results.json L42-51,README.md L181-181

Findings

The benchmark and the camera reversed the p99 order

Latency of INT8 on the iGPU and NPU, flat-out against live

Hollow dots are the flat-out benchmark (1,000 frames each); filled dots are the live webcam loop (2,398 iGPU frames, 1,891 NPU frames). The readout gives flat-out / live. The vertical line is the 33.3 ms frame budget.

Flat-out benchmarkLive webcam loop33.3 ms budget
Show the numbers as a table
ProcessorRegimeFramesMedian (ms)p99 (ms)Frames over 33.3 ms
Arc iGPUFlat-out1,00011.3047.021.6%
Arc iGPULive2,39810.4213.310.33%
NPUFlat-out1,00013.9317.980.2%
NPULive1,89118.0422.280.05%
Data:live_results.json L3-51

Flat-out, the iGPU had the better median but the worse tail of the two, and the NPU looked like the safe real-time choice. On the camera loop, where the iGPU spends much of each frame interval idle, its tail collapsed and it beat the NPU on both median and p99, although the NPU still missed the budget less often. The NPU got slower live, not faster. A study that reported only the benchmark would have described headroom, not this workload.

The iGPU tail was CPU-side stalls, not inference

Frame streamReplaying at 40 frames a secondStatic view: all 1,000 frames

The frame strip is drawn with JavaScript. The readout gives the whole run, and every frame is in the linked source file.

To inspect single frames, press the left and right arrow keys. Page Up and Page Down move 50 frames, Home and End go to the first and the latest frame shown, and Escape ends the inspection. Inspecting pauses the replay.
Flat-out benchmark, INT8, 1,000 timed frames per processorEach bar is one measured frame: its height is end-to-end latency, stacked as preprocess, inference and postprocess. The INT8 build keeps the detection-head decode in float. A stall is a frame whose preprocess plus postprocess, both on the CPU, took more than 20 ms. Bars taller than 70 ms are cut at the top and marked with a caret; where there is room, a label or a short tag gives the full time. In motion, frames replay at 40 a second, not at the rate each processor ran them, the readout counts only the frames shown so far, the replay stops at the end of the run, and switching processor keeps the frame number. With motion off all 1,000 frames are drawn at once, and a pixel column that covers several frames shows the tallest.Frames:latency_main.parquetArc iGPU checked against:summary_main.json L146-169,stall_analysis.json L82-94NPU checked against:summary_main.json L242-265,stall_analysis.json L134-146CPU checked against:summary_main.json L50-73,stall_analysis.json L30-42
Flat-out benchmark, INT8, 1,000 timed frames per processor: latency summary
ProcessorMedian (ms)p99 (ms)Median inference (ms)Frames over 33.3 msStall frames
Arc iGPU11.3047.027.251617
NPU13.9317.9810.6522
CPU32.6349.0128.824631
Every stall frame (preprocess plus postprocess over 20 ms), times in ms
ProcessorFramePreprocessInferencePostprocessEnd to end
Arc iGPU372.838.0635.7046.59
Arc iGPU923.117.3172.6983.11
Arc iGPU1092.767.1536.4146.32
Arc iGPU2533.507.5536.5947.64
Arc iGPU3612.977.4936.5847.04
Arc iGPU4132.648.4038.2449.28
Arc iGPU4143.269.6476.9989.89
Arc iGPU4163.388.65110.26122.29
Arc iGPU4203.769.2873.4086.44
Arc iGPU4283.048.88122.63134.55
Arc iGPU4344.209.6216.4230.24
Arc iGPU5072.707.5034.2244.42
Arc iGPU5093.0419.0234.3256.38
Arc iGPU5182.976.8935.8845.74
Arc iGPU5902.617.6235.5145.74
Arc iGPU8592.436.2646.4655.15
Arc iGPU9562.095.9838.9547.02
NPU303.5310.8374.1388.49
NPU4102.1110.7471.6884.53
CPU4192.8547.1035.9985.94
p99 latency with and without stall frames, flat-out benchmark

A stall is a frame whose CPU-side stages (preprocess plus postprocess) took more than 20 ms. Hollow dots include every frame; filled dots leave the stall frames out. The readout gives all frames / without stalls.

All framesWithout stall frames33.3 ms budget
Show the numbers as a table
ConfigurationStall frames (of 1,000)p99, all frames (ms)p99 without stalls (ms)Inference-only p99 (ms)
Arc iGPU FP321750.5017.4510.36
Arc iGPU FP16818.9217.3010.10
Arc iGPU INT81747.0217.069.51
Arc iGPU INT8 full2149.0015.058.22
NPU FP32017.9417.9412.79
NPU FP16219.0118.9213.68
NPU INT8217.9817.8512.21
NPU INT8 full216.7716.5911.88
Data:stall_analysis.json L56-159,analyze_stalls.py

Across its four builds the iGPU produced 63 stall frames in 4,000, the NPU 6 and the CPU 1. Take the stall frames out and three of the four iGPU tails fall from around 50 ms to 15 to 17.5 ms, while inference-only p99 on the same runs stayed near 10 ms. Faster inference would not have fixed these frames.

Stall counts:stall_analysis.json L161-178

Why the iGPU runs stall is not isolated here. Display compositor contention, the shared package power budget and driver threads are all candidates, and separating them would need ETW tracing.

What INT8 cost

Box mAP50-95 lost against FP32

COCO val2017, 4,700 images not used for calibration. The FP32 reference was scored on the CPU, so the iGPU and NPU rows combine a precision change with a device change.

Show the numbers as a table
BuildScored onBox mAP50-95Box change vs FP32Mask mAP50-95Mask change vs FP32
FP32CPU0.3602reference0.3004reference
FP16CPU0.36020.00000.3005+0.0001
INT8, head in floatCPU0.3587−0.00150.2985−0.0019
INT8, fully quantizedCPU0.3213−0.03890.2872−0.0132
INT8, head in floatArc iGPU0.3578−0.00240.2990−0.0014
INT8, head in floatNPU0.3579−0.00230.2990−0.0014
Data:accuracy.json

With the detection-head decode kept in float, INT8 cost 0.0015 box mAP on the CPU. Quantizing that decode as well cost 0.0389, about 26 times as much. What that bought in speed depended on the processor. On the NPU the fully quantized build's median was only 0.35 ms faster (13.58 against 13.93 ms). On the iGPU its median was 1.33 ms faster, but its flat-out p99 was worse (49.00 against 47.02 ms). On the CPU it cut the frames over budget from 46.3% to 4.8%. The same INT8 weights scored within 0.0009 of each other on the CPU, iGPU and NPU.

Latency by build:summary_main.json L50-97,summary_main.json L146-193,summary_main.json L242-289

What this does not show

  • Power was not measured, so the efficiency argument for the NPU is not tested here.
  • The stall mechanism is not attributed.
  • One model and one input size, on one machine at room temperature and on mains power.
  • Only the two INT8 configurations were run live, for one 60 s session each, in one lighting condition with one person in frame. A busier scene changes postprocess cost.

Limits stated in the repository:README.md L166-184

  • The live loop does not wait for new camera frames, so its duty cycle sits between a 30 fps camera and flat-out rather than matching either.
  • The repository includes a script that checks the hand-written postprocess against the Ultralytics decoder, but its output is not committed, so no result from it is quoted here.

Added for this write-up:live_demo.py,verify_postprocess.py