Case study 01 / LLM serving / reproducibility

bitwise-forensics

Does a serving engine give the same greedy tokens for a prompt when other requests share its engine steps? A measurement on OpenVINO GenAI's continuous-batching pipeline, on the CPU and Arc iGPU of an Intel laptop chip.

  • Version 0.3.1, 15 Sep 2026
  • Python, OpenVINO GenAI 2026.3
  • 186 tests, CI on Linux and Windows

The short version

  • Under each device's default settings, a batched run's tokens differed from the same prompt served alone in 95.6 to 100% of iGPU runs and 54.4 to 72.5% of CPU runs, within the first 128 generated tokens.
  • Given its arrival schedule, the output was reproducible where tested: replaying the eight c8 schedules reproduced all 320 runs on each device, bit for bit.
  • On the CPU, turning off dynamic activation quantization cut the share of changed runs at c8 from 71.2% to 21.2%. Adding an f32 KV cache brought it to 0 of 320 runs (95% upper bound 8.8% counting prompts), although logprobs still moved. No tested iGPU setting removed the changes.
  • The two devices run different quantized defaults, so the gap between them is not a hardware comparison.

Setup

The engine is OpenVINO GenAI 2026.3's ContinuousBatchingPipeline serving Qwen2.5-Coder-0.5B-Instruct with INT4 weights, on the Arc iGPU and the CPU of an Intel Core Ultra 7 155H. Decoding is greedy, with up to 128 new tokens.

The prompts are 40 short instructions built from 4 templates and 10 LLM-serving topics. Each prompt is first served alone to get its batch-of-1 output. Then all 40 arrive on a Poisson schedule and share engine steps. A run counts as changed when its tokens differ from that batch-of-1 output at any position, and as not bitwise identical when its tokens or any chosen-token logprob differ.

Load is an arrival rate written c2 to c32: the average number of arrivals per maximum-length generation, on a virtual clock of 20 ms engine steps. It is not a number of concurrent users. Every device and option set sees the same arrival schedules, although batches at later steps can differ once outputs stop at different lengths, and prefill options change how requests are grouped into steps even when output lengths match. Each configuration ran 8 trials, so every row covers 320 runs: 12 configurations make 3,840 runs per device and 7,680 in total.

Both devices quantize by default. As compiled, the iGPU runs f16 activations with a 4-bit KV cache and dynamic quantization in groups of 32; the CPU runs f32 activations with an 8-bit KV cache and the same dynamic quantization.

Compiled properties:summary.json L93-107,summary.json L225-242

Findings

Sharing engine steps usually changed the tokens

Runs whose tokens changed, by arrival rate

Default options. Dots are the share of 320 runs. Lines are 95% intervals: a bootstrap over prompts and trials, widened to the exact interval on runs; where the share is 100% the line shows the exact interval counting prompts instead. Each device has its own panel because each runs its own quantized defaults.

Arc iGPU

CPU

Show the numbers as a table
Arrival rateiGPUiGPU 95% intervalCPUCPU 95% interval
c295.6%90.9 to 99.154.4%43.8 to 65.0
c499.7%98.1 to 100.072.5%63.1 to 80.6
c8100.0%98.9 to 100.0 by run; 91.2 to 100.0 by prompt71.2%58.8 to 82.2
c1699.7%98.1 to 100.070.3%56.2 to 82.8
c3298.4%94.4 to 100.070.6%56.9 to 83.4
Data:summary.csv L2-7,summary.csv L2-7,scorecard.md L17-22

On the CPU the share rose between c2 and c4, by 18.1 points (95% CI 5.3 to 30.3). The two comparisons made beyond c4 (c16 against c4 and c32 against c8) detected no change, so these data do not show the share growing with load beyond that step.

Paired rate comparisons:readme_stats.md L11-18

CPU · Default options · c8

228of 320 runs changed

71.2%by token 128 of 128

Device
Engine options

No properties passed. Compiled CPU defaults: f32 activations, u8 KV cache, dynamic quantization with group size 32. 39 of 40 prompts changed in at least one trial. 2 prompts gave the same completion in all 8 trials, and 1 of those matched batch-of-1.

40 prompts across · 8 trials down · one strip = one run, 128 tokens wide

  • Same tokens as batch-of-1
  • Copper from the first differing token on
  • Batch-of-1 output continues

The 320 token strips are drawn with JavaScript. The counts shown are the final measured values.

Share of the 320 runs changed, by token position

Final state at token 128
7.2%within the first 16 tokens
38.1%within the first 32 tokens
56.2%within the first 64 tokens
71.2%within the first 128 tokens
Median first differing token
29
position, over the 228 runs that changed
Distinct completions per prompt
3.32
mean over 40 prompts, across 8 trials
Tokens or logprobs differ
100%
of runs, counting chosen-token logprob changes too
Continuous batching changed greedy tokens, and engine options changed how oftenOpenVINO GenAI serving Qwen2.5-Coder-0.5B-Instruct INT4 on an Intel Core Ultra 7 155H, greedy decoding, arrival rate c8 (a Poisson rate of 8 arrivals per max-length generation on a virtual clock, not 8 users), 128 new tokens max, 40 prompts × 8 trials. A strip turns copper at the first token that differs from the same prompt served alone (batch-of-1) with the same options; each option set has its own batch-of-1 reference. Positions, lengths and counts are measured. All runs are replayed together by token position, not in arrival order, and the streaming speed is illustrative. Prompt, trial and token positions are 0-indexed, as in the data files.Data:runs.jsonl.gz,baselines.jsonl.gz,summary.csv L1,summary.csv L7Options:manifest.json,README.md L453
The six c8 option sets shown in the grid, 320 runs each
DeviceEngine optionsRuns changedWithin 16 tokensWithin 32 tokensWithin 64 tokensMedian first differing tokenDistinct completions per prompt
CPUDefault options228 of 320 (71.2%)7.2%38.1%56.2%293.32
CPUDynamic quantization off68 of 320 (21.2%)2.2%10%11.9%401.45
CPUBoth: f32 KV cache + dynamic quantization off0 of 320 (0%)0%0%0%none1.00
Arc iGPUDefault options320 of 320 (100%)58.1%78.8%96.2%127.22
Arc iGPUDynamic quantization off320 of 320 (100%)60.9%90.9%96.9%141.25
Arc iGPUf16 KV cache164 of 320 (51.2%)9.7%20%25.6%64.53.00
Per prompt, CPU, Default options: trials whose tokens changed, the earliest first differing token, and distinct completions across 8 trials
PromptTrials changedEarliest first differing tokenDistinct completions
0: Summarize the trade-offs of kernel fusion for a latency-sensitive service.6 of 8656
1: Summarize the trade-offs of paged attention for a latency-sensitive service.8 of 8292
2: Summarize the trade-offs of weight quantization for a latency-sensitive service.8 of 8175
3: Summarize the trade-offs of KV eviction for a latency-sensitive service.8 of 8503
4: Summarize the trade-offs of speculative decoding for a latency-sensitive service.8 of 8792
5: Summarize the trade-offs of continuous batching for a latency-sensitive service.8 of 8414
6: Summarize the trade-offs of tensor parallelism for a latency-sensitive service.8 of 8405
7: Summarize the trade-offs of prefix caching for a latency-sensitive service.4 of 8355
8: Summarize the trade-offs of guided decoding for a latency-sensitive service.1 of 8842
9: Summarize the trade-offs of disaggregated serving for a latency-sensitive service.7 of 8233
10: Explain kernel fusion to an engineer who only knows training.8 of 8175
11: Explain paged attention to an engineer who only knows training.8 of 8572
12: Explain weight quantization to an engineer who only knows training.7 of 8132
13: Explain KV eviction to an engineer who only knows training.2 of 8483
14: Explain speculative decoding to an engineer who only knows training.8 of 8184
15: Explain continuous batching to an engineer who only knows training.7 of 8182
16: Explain tensor parallelism to an engineer who only knows training.8 of 8372
17: Explain prefix caching to an engineer who only knows training.8 of 8142
18: Explain guided decoding to an engineer who only knows training.7 of 8174
19: Explain disaggregated serving to an engineer who only knows training.3 of 8194
20: List three failure modes of kernel fusion under bursty traffic.0 of 8none1
21: List three failure modes of paged attention under bursty traffic.1 of 8572
22: List three failure modes of weight quantization under bursty traffic.7 of 875
23: List three failure modes of KV eviction under bursty traffic.8 of 8155
24: List three failure modes of speculative decoding under bursty traffic.8 of 8293
25: List three failure modes of continuous batching under bursty traffic.1 of 8272
26: List three failure modes of tensor parallelism under bursty traffic.7 of 8405
27: List three failure modes of prefix caching under bursty traffic.7 of 8193
28: List three failure modes of guided decoding under bursty traffic.1 of 8822
29: List three failure modes of disaggregated serving under bursty traffic.4 of 8255
30: Compare kernel fusion across two hypothetical GPU clusters.7 of 8265
31: Compare paged attention across two hypothetical GPU clusters.1 of 8222
32: Compare weight quantization across two hypothetical GPU clusters.7 of 8276
33: Compare KV eviction across two hypothetical GPU clusters.8 of 8211
34: Compare speculative decoding across two hypothetical GPU clusters.1 of 8712
35: Compare continuous batching across two hypothetical GPU clusters.3 of 8514
36: Compare tensor parallelism across two hypothetical GPU clusters.8 of 8234
37: Compare prefix caching across two hypothetical GPU clusters.7 of 8594
38: Compare guided decoding across two hypothetical GPU clusters.8 of 8192
39: Compare disaggregated serving across two hypothetical GPU clusters.2 of 8323

The same schedule gives the same output

320 of 320
iGPU runs at c8 with identical tokens, chosen-token logprobs and batch sizes on replay
Data:readme_stats.json L802-808
320 of 320
CPU runs at c8 with identical tokens, chosen-token logprobs and batch sizes on replay
Data:readme_stats.json L943-949

Replaying the eight c8 arrival schedules with default options, in the same process, reproduced every run on both devices. Batch-of-1 output was also bitwise identical across two passes before the trials and one after them, for every option set. So the output here is a fixed function of the arrival schedule, where that was tested: a different schedule can change it, and the same schedule gives it back.

Batch-of-1 checks:baseline_checks.csv L2-7,baseline_checks.csv L2-7

What changed the rate

Runs whose tokens changed at c8, by engine setting

Each row is measured against the same prompts served alone under that row's own settings. Lines are 95% intervals as in the chart above; rows at 0% or 100% show the interval counting prompts. The table adds paired differences against the default row.

Arc iGPU

CPU

Show the numbers as a table
SettingDeviceRuns changed95% intervalDistinct outputs per promptPaired difference vs default, points (95% CI)
Default (f16 activations, 4-bit KV cache)iGPU100.0%98.9 to 100.0 by run; 91.2 to 100.0 by prompt7.22reference
f16 KV cacheiGPU51.2%39.4 to 63.13.00−48.8 (−60.6 to −36.9)
Dynamic quantization offiGPU100.0%98.9 to 100.0 by run; 91.2 to 100.0 by prompt1.250.0 (both rows at 100%)
Both changes (f16 KV cache, dynamic quantization off)iGPU60.3%45.6 to 75.01.12−39.7 (−54.4 to −25.0)
16-token step budget (shared with decode tokens)iGPU99.7%98.1 to 100.06.68−0.3 (−1.9 to 0.0)
Default (f32 activations, 8-bit KV cache)CPU71.2%58.8 to 82.23.32reference
f32 KV cacheCPU71.6%59.4 to 82.83.30+0.3 (−14.7 to +15.0)
Dynamic quantization offCPU21.2%10.6 to 32.81.45−50.0 (−64.4 to −34.7)
Both changes (f32 KV cache, dynamic quantization off)CPU0 of 3200.0 to 1.1 by run; 0.0 to 8.8 by prompt1.00−71.2 (−82.2 to −58.8)
16-token step budget (shared with decode tokens)CPU90.0%82.8 to 95.65.80+18.8 (+5.3 to +32.2)
Data:summary.csv L7-11,summary.csv L7-11,paired.csv L3-7,paired.csv L3-7,scorecard.md L19-27,scorecard.md L19-27

On the CPU, turning off dynamic activation quantization was the main lever: the share at c8 fell by 50.0 points (paired 95% CI 34.7 to 64.4). An f32 KV cache on its own had no detectable effect on the share (+0.3 points, paired 95% CI −14.7 to +15.0), although its batch-of-1 outputs differed from the default's on 77.5% of prompts. With both changes, 0 of 320 runs had changed tokens at c8, and 0 of 320 at c32. The exact 95% upper bound is 8.8% counting prompts, the fairer unit because runs of the same prompt are correlated, and 1.1% counting runs.

That setting is still not batch-invariant: every one of those runs had chosen-token logprobs that differed from batch-of-1, by at most 2.9e-5.

Data:paired.csv L3-4,readme_stats.json L376-395,baseline_checks.csv L4

On the CPU at c8, capping each engine step at 16 tokens (a budget shared with decode tokens, so prompts are split across several steps) raised the share from 71.2% to 90.0% (paired difference +18.8 points, 95% CI 5.3 to 32.2). That option also changed batch-of-1 output on 36 of 40 prompts.

Data:paired.csv L7,baseline_checks.csv L6

On the iGPU no tested setting removed the changes. An f16 KV cache lowered the share to 51.2%. Turning off dynamic quantization left it at 100% but cut the distinct outputs per prompt from 7.22 to 1.25. There is no float32 row: none of the eight builds given the f32 inference precision hint served even one 2-token request in this OpenVINO GenAI version with this driver. Two did not compile, four failed on the first engine step and two on the second.

Float32 probe:README.md L72-106,f32_bare_request.json

Where the outputs fork

On the iGPU most changed runs forked early: 58.1% of c8 runs had already diverged within the first 16 tokens. On the CPU, forks came later.

Share of runs whose first changed token fell within the first N generated tokens (320 runs per row)
RowN = 16N = 32N = 64N = 128
iGPU default, c858.1%78.8%96.2%100.0%
CPU default, c21.9%12.5%26.9%54.4%
CPU default, c87.2%38.1%56.2%71.2%
iGPU f16 KV cache, c89.7%20.0%25.6%51.2%
CPU dynamic quantization off, c82.2%10.0%11.9%21.2%
CPU f32 KV cache and dynamic quantization off, c80.0%0.0%0.0%0.0%
Data:readme_stats.md L140-147

What this does not show

  • One engine, one 0.5B model with INT4 weights, one laptop and greedy decoding. It does not compare OpenVINO with other engines.
  • 40 short templated instructions that share their opening tokens. No code, long-context or multi-turn prompts.
  • Output quality was not measured, so this says nothing about batched outputs being better or worse, and it does not identify which operation causes a change.
  • "Bitwise" covers the tokens and the chosen token's logprob, not full logit vectors.
  • Each load level is one burst of 40 requests on a virtual clock, not production traffic.
  • Replay and repeat checks ran within one process. Across processes only batch-of-1 output was checked, and no other drivers or OpenVINO versions were tried.
  • CPU thread count, core type and execution mode were not varied.
  • Intervals are not adjusted for multiple comparisons, and option sets ran as separate blocks rather than interleaved in time.
  • The iGPU and the CPU differ in hardware, activation precision and KV cache format at once.

How the statistics work

  • Per-row intervals are a bootstrap over prompts and trials, widened to the exact Clopper-Pearson interval so they never have zero width.
  • Differences between engine settings at the same arrival rate apply the same resampled prompts and trials to both rows, because both rows ran the same prompts on the same arrival schedules. Differences across arrival rates, such as CPU c4 against c2, resample prompts jointly but trials independently for each row, because those rows share prompts but not arrival schedules. Paired intervals are not widened.
  • The README maps every result to its source. Most come from the scorecards and CSV files that bitwise/report.py writes from the committed run records, plus statistics that analysis/readme_stats.py computes from the same records; the rest come from a separate precision probe, the model's tokenizer and model file, the sweep manifests and the simulator runs of analysis/sim_seeds.py. 186 tests run in CI on Linux and Windows.

Code:report.py,readme_stats.py,README.md

How it started

The project began as a simulator. Across 12 simulator seeds, each changing the synthetic model, its noise streams and the arrival schedules, the c8 and c32 rates averaged 31.8% and 32.0% with standard deviations near 15 points, and the seeds split on the direction of the difference between them (6 lower, 5 higher, 1 tied). A load trend that flips with the seed is not a finding, so the simulator stays in the repository as a pipeline test and the published results come from the real engine.

Data:sim_seeds.md