Case study 01 / LLM serving / reproducibility
bitwise-forensics
Does a serving engine give the same greedy tokens for a prompt when other requests share its engine steps? A measurement on OpenVINO GenAI's continuous-batching pipeline, on the CPU and Arc iGPU of an Intel laptop chip.
The short version
- Under each device's default settings, a batched run's tokens differed from the same prompt served alone in 95.6 to 100% of iGPU runs and 54.4 to 72.5% of CPU runs, within the first 128 generated tokens.
- Given its arrival schedule, the output was reproducible where tested: replaying the eight c8 schedules reproduced all 320 runs on each device, bit for bit.
- On the CPU, turning off dynamic activation quantization cut the share of changed runs at c8 from 71.2% to 21.2%. Adding an f32 KV cache brought it to 0 of 320 runs (95% upper bound 8.8% counting prompts), although logprobs still moved. No tested iGPU setting removed the changes.
- The two devices run different quantized defaults, so the gap between them is not a hardware comparison.
Setup
The engine is OpenVINO GenAI 2026.3's ContinuousBatchingPipeline serving Qwen2.5-Coder-0.5B-Instruct with INT4 weights, on the Arc iGPU and the CPU of an Intel Core Ultra 7 155H. Decoding is greedy, with up to 128 new tokens.
The prompts are 40 short instructions built from 4 templates and 10 LLM-serving topics. Each prompt is first served alone to get its batch-of-1 output. Then all 40 arrive on a Poisson schedule and share engine steps. A run counts as changed when its tokens differ from that batch-of-1 output at any position, and as not bitwise identical when its tokens or any chosen-token logprob differ.
Load is an arrival rate written c2 to c32: the average number of arrivals per maximum-length generation, on a virtual clock of 20 ms engine steps. It is not a number of concurrent users. Every device and option set sees the same arrival schedules, although batches at later steps can differ once outputs stop at different lengths, and prefill options change how requests are grouped into steps even when output lengths match. Each configuration ran 8 trials, so every row covers 320 runs: 12 configurations make 3,840 runs per device and 7,680 in total.
Both devices quantize by default. As compiled, the iGPU runs f16 activations with a 4-bit KV cache and dynamic quantization in groups of 32; the CPU runs f32 activations with an 8-bit KV cache and the same dynamic quantization.
Compiled properties:summary.json L93-107,summary.json L225-242
Findings
Sharing engine steps usually changed the tokens
Default options. Dots are the share of 320 runs. Lines are 95% intervals: a bootstrap over prompts and trials, widened to the exact interval on runs; where the share is 100% the line shows the exact interval counting prompts instead. Each device has its own panel because each runs its own quantized defaults.
Arc iGPU
CPU
Show the numbers as a table
| Arrival rate | iGPU | iGPU 95% interval | CPU | CPU 95% interval |
|---|---|---|---|---|
| c2 | 95.6% | 90.9 to 99.1 | 54.4% | 43.8 to 65.0 |
| c4 | 99.7% | 98.1 to 100.0 | 72.5% | 63.1 to 80.6 |
| c8 | 100.0% | 98.9 to 100.0 by run; 91.2 to 100.0 by prompt | 71.2% | 58.8 to 82.2 |
| c16 | 99.7% | 98.1 to 100.0 | 70.3% | 56.2 to 82.8 |
| c32 | 98.4% | 94.4 to 100.0 | 70.6% | 56.9 to 83.4 |
On the CPU the share rose between c2 and c4, by 18.1 points (95% CI 5.3 to 30.3). The two comparisons made beyond c4 (c16 against c4 and c32 against c8) detected no change, so these data do not show the share growing with load beyond that step.
Paired rate comparisons:readme_stats.md L11-18
CPU · Default options · c8
228of 320 runs changed
No properties passed. Compiled CPU defaults: f32 activations, u8 KV cache, dynamic quantization with group size 32. 39 of 40 prompts changed in at least one trial. 2 prompts gave the same completion in all 8 trials, and 1 of those matched batch-of-1.
40 prompts across · 8 trials down · one strip = one run, 128 tokens wide
- Same tokens as batch-of-1
- Copper from the first differing token on
- Batch-of-1 output continues
The 320 token strips are drawn with JavaScript. The counts shown are the final measured values.
Share of the 320 runs changed, by token position
Final state at token 128- Median first differing token
- 29
- position, over the 228 runs that changed
- Distinct completions per prompt
- 3.32
- mean over 40 prompts, across 8 trials
- Tokens or logprobs differ
- 100%
- of runs, counting chosen-token logprob changes too
| Device | Engine options | Runs changed | Within 16 tokens | Within 32 tokens | Within 64 tokens | Median first differing token | Distinct completions per prompt |
|---|---|---|---|---|---|---|---|
| CPU | Default options | 228 of 320 (71.2%) | 7.2% | 38.1% | 56.2% | 29 | 3.32 |
| CPU | Dynamic quantization off | 68 of 320 (21.2%) | 2.2% | 10% | 11.9% | 40 | 1.45 |
| CPU | Both: f32 KV cache + dynamic quantization off | 0 of 320 (0%) | 0% | 0% | 0% | none | 1.00 |
| Arc iGPU | Default options | 320 of 320 (100%) | 58.1% | 78.8% | 96.2% | 12 | 7.22 |
| Arc iGPU | Dynamic quantization off | 320 of 320 (100%) | 60.9% | 90.9% | 96.9% | 14 | 1.25 |
| Arc iGPU | f16 KV cache | 164 of 320 (51.2%) | 9.7% | 20% | 25.6% | 64.5 | 3.00 |
| Prompt | Trials changed | Earliest first differing token | Distinct completions |
|---|---|---|---|
| 0: Summarize the trade-offs of kernel fusion for a latency-sensitive service. | 6 of 8 | 65 | 6 |
| 1: Summarize the trade-offs of paged attention for a latency-sensitive service. | 8 of 8 | 29 | 2 |
| 2: Summarize the trade-offs of weight quantization for a latency-sensitive service. | 8 of 8 | 17 | 5 |
| 3: Summarize the trade-offs of KV eviction for a latency-sensitive service. | 8 of 8 | 50 | 3 |
| 4: Summarize the trade-offs of speculative decoding for a latency-sensitive service. | 8 of 8 | 79 | 2 |
| 5: Summarize the trade-offs of continuous batching for a latency-sensitive service. | 8 of 8 | 41 | 4 |
| 6: Summarize the trade-offs of tensor parallelism for a latency-sensitive service. | 8 of 8 | 40 | 5 |
| 7: Summarize the trade-offs of prefix caching for a latency-sensitive service. | 4 of 8 | 35 | 5 |
| 8: Summarize the trade-offs of guided decoding for a latency-sensitive service. | 1 of 8 | 84 | 2 |
| 9: Summarize the trade-offs of disaggregated serving for a latency-sensitive service. | 7 of 8 | 23 | 3 |
| 10: Explain kernel fusion to an engineer who only knows training. | 8 of 8 | 17 | 5 |
| 11: Explain paged attention to an engineer who only knows training. | 8 of 8 | 57 | 2 |
| 12: Explain weight quantization to an engineer who only knows training. | 7 of 8 | 13 | 2 |
| 13: Explain KV eviction to an engineer who only knows training. | 2 of 8 | 48 | 3 |
| 14: Explain speculative decoding to an engineer who only knows training. | 8 of 8 | 18 | 4 |
| 15: Explain continuous batching to an engineer who only knows training. | 7 of 8 | 18 | 2 |
| 16: Explain tensor parallelism to an engineer who only knows training. | 8 of 8 | 37 | 2 |
| 17: Explain prefix caching to an engineer who only knows training. | 8 of 8 | 14 | 2 |
| 18: Explain guided decoding to an engineer who only knows training. | 7 of 8 | 17 | 4 |
| 19: Explain disaggregated serving to an engineer who only knows training. | 3 of 8 | 19 | 4 |
| 20: List three failure modes of kernel fusion under bursty traffic. | 0 of 8 | none | 1 |
| 21: List three failure modes of paged attention under bursty traffic. | 1 of 8 | 57 | 2 |
| 22: List three failure modes of weight quantization under bursty traffic. | 7 of 8 | 7 | 5 |
| 23: List three failure modes of KV eviction under bursty traffic. | 8 of 8 | 15 | 5 |
| 24: List three failure modes of speculative decoding under bursty traffic. | 8 of 8 | 29 | 3 |
| 25: List three failure modes of continuous batching under bursty traffic. | 1 of 8 | 27 | 2 |
| 26: List three failure modes of tensor parallelism under bursty traffic. | 7 of 8 | 40 | 5 |
| 27: List three failure modes of prefix caching under bursty traffic. | 7 of 8 | 19 | 3 |
| 28: List three failure modes of guided decoding under bursty traffic. | 1 of 8 | 82 | 2 |
| 29: List three failure modes of disaggregated serving under bursty traffic. | 4 of 8 | 25 | 5 |
| 30: Compare kernel fusion across two hypothetical GPU clusters. | 7 of 8 | 26 | 5 |
| 31: Compare paged attention across two hypothetical GPU clusters. | 1 of 8 | 22 | 2 |
| 32: Compare weight quantization across two hypothetical GPU clusters. | 7 of 8 | 27 | 6 |
| 33: Compare KV eviction across two hypothetical GPU clusters. | 8 of 8 | 21 | 1 |
| 34: Compare speculative decoding across two hypothetical GPU clusters. | 1 of 8 | 71 | 2 |
| 35: Compare continuous batching across two hypothetical GPU clusters. | 3 of 8 | 51 | 4 |
| 36: Compare tensor parallelism across two hypothetical GPU clusters. | 8 of 8 | 23 | 4 |
| 37: Compare prefix caching across two hypothetical GPU clusters. | 7 of 8 | 59 | 4 |
| 38: Compare guided decoding across two hypothetical GPU clusters. | 8 of 8 | 19 | 2 |
| 39: Compare disaggregated serving across two hypothetical GPU clusters. | 2 of 8 | 32 | 3 |
The same schedule gives the same output
Replaying the eight c8 arrival schedules with default options, in the same process, reproduced every run on both devices. Batch-of-1 output was also bitwise identical across two passes before the trials and one after them, for every option set. So the output here is a fixed function of the arrival schedule, where that was tested: a different schedule can change it, and the same schedule gives it back.
Batch-of-1 checks:baseline_checks.csv L2-7,baseline_checks.csv L2-7
What changed the rate
Each row is measured against the same prompts served alone under that row's own settings. Lines are 95% intervals as in the chart above; rows at 0% or 100% show the interval counting prompts. The table adds paired differences against the default row.
Arc iGPU
CPU
Show the numbers as a table
| Setting | Device | Runs changed | 95% interval | Distinct outputs per prompt | Paired difference vs default, points (95% CI) |
|---|---|---|---|---|---|
| Default (f16 activations, 4-bit KV cache) | iGPU | 100.0% | 98.9 to 100.0 by run; 91.2 to 100.0 by prompt | 7.22 | reference |
| f16 KV cache | iGPU | 51.2% | 39.4 to 63.1 | 3.00 | −48.8 (−60.6 to −36.9) |
| Dynamic quantization off | iGPU | 100.0% | 98.9 to 100.0 by run; 91.2 to 100.0 by prompt | 1.25 | 0.0 (both rows at 100%) |
| Both changes (f16 KV cache, dynamic quantization off) | iGPU | 60.3% | 45.6 to 75.0 | 1.12 | −39.7 (−54.4 to −25.0) |
| 16-token step budget (shared with decode tokens) | iGPU | 99.7% | 98.1 to 100.0 | 6.68 | −0.3 (−1.9 to 0.0) |
| Default (f32 activations, 8-bit KV cache) | CPU | 71.2% | 58.8 to 82.2 | 3.32 | reference |
| f32 KV cache | CPU | 71.6% | 59.4 to 82.8 | 3.30 | +0.3 (−14.7 to +15.0) |
| Dynamic quantization off | CPU | 21.2% | 10.6 to 32.8 | 1.45 | −50.0 (−64.4 to −34.7) |
| Both changes (f32 KV cache, dynamic quantization off) | CPU | 0 of 320 | 0.0 to 1.1 by run; 0.0 to 8.8 by prompt | 1.00 | −71.2 (−82.2 to −58.8) |
| 16-token step budget (shared with decode tokens) | CPU | 90.0% | 82.8 to 95.6 | 5.80 | +18.8 (+5.3 to +32.2) |
On the CPU, turning off dynamic activation quantization was the main lever: the share at c8 fell by 50.0 points (paired 95% CI 34.7 to 64.4). An f32 KV cache on its own had no detectable effect on the share (+0.3 points, paired 95% CI −14.7 to +15.0), although its batch-of-1 outputs differed from the default's on 77.5% of prompts. With both changes, 0 of 320 runs had changed tokens at c8, and 0 of 320 at c32. The exact 95% upper bound is 8.8% counting prompts, the fairer unit because runs of the same prompt are correlated, and 1.1% counting runs.
That setting is still not batch-invariant: every one of those runs had chosen-token logprobs that differed from batch-of-1, by at most 2.9e-5.
Data:paired.csv L3-4,readme_stats.json L376-395,baseline_checks.csv L4
On the CPU at c8, capping each engine step at 16 tokens (a budget shared with decode tokens, so prompts are split across several steps) raised the share from 71.2% to 90.0% (paired difference +18.8 points, 95% CI 5.3 to 32.2). That option also changed batch-of-1 output on 36 of 40 prompts.
Data:paired.csv L7,baseline_checks.csv L6
On the iGPU no tested setting removed the changes. An f16 KV cache lowered the share to 51.2%. Turning off dynamic quantization left it at 100% but cut the distinct outputs per prompt from 7.22 to 1.25. There is no float32 row: none of the eight builds given the f32 inference precision hint served even one 2-token request in this OpenVINO GenAI version with this driver. Two did not compile, four failed on the first engine step and two on the second.
Float32 probe:README.md L72-106,f32_bare_request.json
Where the outputs fork
On the iGPU most changed runs forked early: 58.1% of c8 runs had already diverged within the first 16 tokens. On the CPU, forks came later.
| Row | N = 16 | N = 32 | N = 64 | N = 128 |
|---|---|---|---|---|
| iGPU default, c8 | 58.1% | 78.8% | 96.2% | 100.0% |
| CPU default, c2 | 1.9% | 12.5% | 26.9% | 54.4% |
| CPU default, c8 | 7.2% | 38.1% | 56.2% | 71.2% |
| iGPU f16 KV cache, c8 | 9.7% | 20.0% | 25.6% | 51.2% |
| CPU dynamic quantization off, c8 | 2.2% | 10.0% | 11.9% | 21.2% |
| CPU f32 KV cache and dynamic quantization off, c8 | 0.0% | 0.0% | 0.0% | 0.0% |
What this does not show
- One engine, one 0.5B model with INT4 weights, one laptop and greedy decoding. It does not compare OpenVINO with other engines.
- 40 short templated instructions that share their opening tokens. No code, long-context or multi-turn prompts.
- Output quality was not measured, so this says nothing about batched outputs being better or worse, and it does not identify which operation causes a change.
- "Bitwise" covers the tokens and the chosen token's logprob, not full logit vectors.
- Each load level is one burst of 40 requests on a virtual clock, not production traffic.
- Replay and repeat checks ran within one process. Across processes only batch-of-1 output was checked, and no other drivers or OpenVINO versions were tried.
- CPU thread count, core type and execution mode were not varied.
- Intervals are not adjusted for multiple comparisons, and option sets ran as separate blocks rather than interleaved in time.
- The iGPU and the CPU differ in hardware, activation precision and KV cache format at once.
How the statistics work
- Per-row intervals are a bootstrap over prompts and trials, widened to the exact Clopper-Pearson interval so they never have zero width.
- Differences between engine settings at the same arrival rate apply the same resampled prompts and trials to both rows, because both rows ran the same prompts on the same arrival schedules. Differences across arrival rates, such as CPU c4 against c2, resample prompts jointly but trials independently for each row, because those rows share prompts but not arrival schedules. Paired intervals are not widened.
- The README maps every result to its source. Most come from the scorecards and CSV files that
bitwise/report.pywrites from the committed run records, plus statistics thatanalysis/readme_stats.pycomputes from the same records; the rest come from a separate precision probe, the model's tokenizer and model file, the sweep manifests and the simulator runs ofanalysis/sim_seeds.py. 186 tests run in CI on Linux and Windows.
Code:report.py,readme_stats.py,README.md
How it started
The project began as a simulator. Across 12 simulator seeds, each changing the synthetic model, its noise streams and the arrival schedules, the c8 and c32 rates averaged 31.8% and 32.0% with standard deviations near 15 points, and the seeds split on the direction of the difference between them (6 lower, 5 higher, 1 tied). A load trend that flips with the seed is not a finding, so the simulator stays in the repository as a pipeline test and the published results come from the real engine.
Data:sim_seeds.md