Case study 03 / LLM evaluation / automation
LLM Eval Observatory
Can an open-weight model triage anti-money-laundering alerts without a person checking every case? A nightly evaluation that commits its results with intervals, so the numbers can be checked.
The short version
- Every night a GitHub Action runs four evaluation tracks against open-weight models on Groq and commits the results to the repository. A Next.js dashboard on Vercel reads that file, and so does the panel below.
- The main track asks whether a model can triage anti-money-laundering alerts: 26 hand-written synthetic cases, each answered as ESCALATE, CLEAR or REVIEW with cited evidence, and scored by a fixed parser rather than by another model.
- The number that matters most is the false clear: a case that should have been escalated and was cleared.
- Decision accuracy, injection defense, capability accuracy and judge agreement carry bootstrap intervals; the false-clear rate and the other rates are exact ratios without one. For the first three, a drift flag fires only when the score moves 5 points or more and the intervals separate.
2 of 2 configured models returned results in this run
2 finished all of their tracks. 30 of the 30 nights up to this run produced results. This panel reads the results file on the project's main branch, which is updated nightly, and refreshes at most hourly.
| Model | AML decisions correct | 95% CI | Escalate cases wrongly cleared | Ambiguous cases sent to review | Injection attacks defended |
|---|---|---|---|---|---|
| GPT-OSS 120B | 100% | 100% to 100% | 0 of 14 | 100% | 29 of 30 |
| GPT-OSS 20B | 92.3% | 80.8% to 100% | 0 of 14 | 50% | 28 of 30 |
The four tracks
| Track | Question | Size | Scored by |
|---|---|---|---|
| AML alert triage | Does the model make the right escalate, clear or review call, cite evidence that exists, and avoid false clears? | 26 cases: 14 escalate, 8 clear, 4 review | A fixed parser for a four-line decision block |
| Prompt injection | Does it resist direct and indirect injection and jailbreak prompts? | 30 attacks | String and canary matching; quoting the payload counts as a breach |
| Capability and drift | Is it competent on a fixed set, and did that change since the last run? | 50 items | Automatic grading |
| Judge reliability | Can the model be trusted to grade other answers? | 24 pairwise items with human labels, in both orders | Agreement, position bias, verbosity bias, Cohen's and Fleiss' kappa |
How the numbers are computed
- Intervals are a percentile bootstrap with 5,000 resamples and a fixed seed, so rerunning on the same answers gives the same interval.
- A drift flag needs the score to move by at least 5 points and the two runs' intervals not to overlap, so ordinary night-to-night noise does not raise it.
- If every model is skipped in a night, the run fails instead of overwriting the last good results.
- The statistics use NumPy only, and the test suite runs before each evaluation.
Code:config.py L22-29,stats.py L34-56,stats.py L70-90,run_eval.py L117-125,nightly-eval.yml
What the history shows so far
From 2 August to 15 September 2026, GPT-OSS 120B completed the triage track on 29 nights and never cleared a case that should have been escalated. Its accuracy on the same 26 cases moved between 92.3% and 100% from night to night, which is a reason to report intervals and trends rather than one run. Counting from the first live run on 28 July, it defended 93.3% to 96.7% of injection attacks over 31 nights; Llama 3.3 70B defended 73.3% to 83.3% over the 19 nights it completed, the last on 17 August. Groq's deprecations page lists both Llama models as shut down on 16 August, and every nightly run from 18 August got HTTP 404 for them. On 15 September the configuration was cut to the two GPT-OSS models, the only production open-weight chat models left on the free tier.
Snapshot at bf8ad56 · recorded nights, 28 Jul to 15 Sep 2026, 28 Jul to 15 Sep 2026
Five models returned results on the first night. After 17 Aug, only GPT-OSS 120B and GPT-OSS 20B did.
- GPT-OSS 120B · triage accuracy
- 92.3 to 100%
- 29 nights, 2 Aug to 15 Sep
- GPT-OSS 120B · false clear rate
- 0%
- on all 29 triage nights
- GPT-OSS 120B · injection defense
- 93.3 to 96.7%
- 31 nights, 28 Jul to 15 Sep
- Llama 3.3 70B · injection defense
- 73.3 to 83.3%
- 19 nights, 28 Jul to 17 Aug
| Night | GPT-OSS 120B | GPT-OSS 20B | Llama 3.3 70B | Llama 3.1 8B | Qwen3.6 27B |
|---|---|---|---|---|---|
| 28 July 2026 | injection defense 96.7%, capability accuracy 100% | injection defense 93.3%, capability accuracy 98% | injection defense 80%, capability accuracy 98% | injection defense 73.3%, capability accuracy 92% | injection defense 100%, capability accuracy 84% |
| 29 July 2026 | injection defense 96.7%, capability accuracy 100% | injection defense 100%, capability accuracy 98% | no results | injection defense 73.3%, capability accuracy 92% | no results |
| 30 July 2026 | no results | no results | triage accuracy 100%, false clear rate 0% | no results | no results |
| 31 July 2026 | no results | no results | triage accuracy 100%, false clear rate 0%, injection defense 76.7%, capability accuracy 98% | no results | no results |
| 1 August 2026 | no results | no results | triage accuracy 100%, false clear rate 0%, injection defense 73.3%, capability accuracy 98% | no results | no results |
| 2 August 2026 | triage accuracy 100%, false clear rate 0%, injection defense 96.7%, capability accuracy 100% | no results | triage accuracy 100%, false clear rate 0%, injection defense 80%, capability accuracy 98% | no results | no results |
| 3 August 2026 | no results | no results | triage accuracy 100%, false clear rate 0%, injection defense 76.7%, capability accuracy 96% | no results | no results |
| 4 August 2026 | no results | no results | triage accuracy 100%, false clear rate 0%, injection defense 76.7%, capability accuracy 96% | no results | no results |
| 5 August 2026 | no results | no results | triage accuracy 100%, false clear rate 0%, injection defense 76.7%, capability accuracy 96% | no results | no results |
| 6 August 2026 | triage accuracy 100%, false clear rate 0%, injection defense 96.7%, capability accuracy 100% | no results | triage accuracy 100%, false clear rate 0%, injection defense 76.7%, capability accuracy 98% | no results | no results |
| 7 August 2026 | triage accuracy 100%, false clear rate 0%, injection defense 93.3%, capability accuracy 100% | no results | triage accuracy 100%, false clear rate 0%, injection defense 76.7%, capability accuracy 96% | no results | no results |
| 8 August 2026 | triage accuracy 100%, false clear rate 0%, injection defense 96.7%, capability accuracy 100% | no results | triage accuracy 100%, false clear rate 0%, injection defense 76.7%, capability accuracy 98% | no results | no results |
| 9 August 2026 | no results | no results | triage accuracy 100%, false clear rate 0%, injection defense 76.7%, capability accuracy 96% | no results | no results |
| 10 August 2026 | no results | no results | triage accuracy 100%, false clear rate 0%, injection defense 80%, capability accuracy 98% | no results | no results |
| 11 August 2026 | no results | no results | triage accuracy 100%, false clear rate 0%, injection defense 76.7%, capability accuracy 96% | no results | no results |
| 12 August 2026 | triage accuracy 100%, false clear rate 0%, injection defense 93.3%, capability accuracy 100% | no results | triage accuracy 100%, false clear rate 0%, injection defense 80%, capability accuracy 98% | no results | no results |
| 13 August 2026 | triage accuracy 96.2%, false clear rate 0%, injection defense 93.3%, capability accuracy 100% | no results | triage accuracy 100%, false clear rate 0%, injection defense 80%, capability accuracy 98% | no results | no results |
| 14 August 2026 | triage accuracy 100%, false clear rate 0%, injection defense 93.3%, capability accuracy 100% | no results | triage accuracy 100%, false clear rate 0%, injection defense 83.3%, capability accuracy 98% | no results | no results |
| 15 August 2026 | triage accuracy 100%, false clear rate 0%, injection defense 93.3%, capability accuracy 100% | no results | triage accuracy 100%, false clear rate 0%, injection defense 80%, capability accuracy 98% | no results | no results |
| 16 August 2026 | triage accuracy 96.2%, false clear rate 0%, injection defense 96.7%, capability accuracy 100% | no results | triage accuracy 100%, false clear rate 0%, injection defense 76.7%, capability accuracy 96% | no results | no results |
| 17 August 2026 | triage accuracy 100%, false clear rate 0%, injection defense 96.7%, capability accuracy 100% | no results | triage accuracy 100%, false clear rate 0%, injection defense 76.7%, capability accuracy 98% | no results | no results |
| 18 August 2026 | triage accuracy 96.2%, false clear rate 0%, injection defense 96.7%, capability accuracy 100% | no results | no results | no results | no results |
| 19 August 2026 | triage accuracy 100%, false clear rate 0%, injection defense 96.7%, capability accuracy 100% | no results | no results | no results | no results |
| 20 August 2026 | no results recorded | no results recorded | no results recorded | no results recorded | no results recorded |
| 21 August 2026 | no results recorded | no results recorded | no results recorded | no results recorded | no results recorded |
| 22 August 2026 | no results recorded | no results recorded | no results recorded | no results recorded | no results recorded |
| 23 August 2026 | no results recorded | no results recorded | no results recorded | no results recorded | no results recorded |
| 24 August 2026 | no results recorded | no results recorded | no results recorded | no results recorded | no results recorded |
| 25 August 2026 | triage accuracy 92.3%, false clear rate 0%, injection defense 93.3%, capability accuracy 100% | no results | no results | no results | no results |
| 26 August 2026 | no results recorded | no results recorded | no results recorded | no results recorded | no results recorded |
| 27 August 2026 | triage accuracy 96.2%, false clear rate 0%, injection defense 93.3%, capability accuracy 100% | no results | no results | no results | no results |
| 28 August 2026 | no results recorded | no results recorded | no results recorded | no results recorded | no results recorded |
| 29 August 2026 | triage accuracy 100%, false clear rate 0%, injection defense 96.7%, capability accuracy 100% | no results | no results | no results | no results |
| 30 August 2026 | no results recorded | no results recorded | no results recorded | no results recorded | no results recorded |
| 31 August 2026 | no results recorded | no results recorded | no results recorded | no results recorded | no results recorded |
| 1 September 2026 | triage accuracy 100%, false clear rate 0%, injection defense 96.7%, capability accuracy 100% | no results | no results | no results | no results |
| 2 September 2026 | no results recorded | no results recorded | no results recorded | no results recorded | no results recorded |
| 3 September 2026 | triage accuracy 100%, false clear rate 0%, injection defense 96.7%, capability accuracy 100% | triage accuracy 92.3%, false clear rate 0%, injection defense 100%, capability accuracy 98% | no results | no results | no results |
| 4 September 2026 | triage accuracy 96.2%, false clear rate 0%, injection defense 96.7%, capability accuracy 100% | no results | no results | no results | no results |
| 5 September 2026 | triage accuracy 100%, false clear rate 0%, injection defense 96.7%, capability accuracy 100% | triage accuracy 92.3%, false clear rate 0%, injection defense 93.3%, capability accuracy 98% | no results | no results | no results |
| 6 September 2026 | triage accuracy 100%, false clear rate 0%, injection defense 96.7%, capability accuracy 100% | triage accuracy 88.5%, false clear rate 0%, injection defense 100%, capability accuracy 98% | no results | no results | no results |
| 7 September 2026 | triage accuracy 100%, false clear rate 0%, injection defense 93.3%, capability accuracy 100% | triage accuracy 92.3%, false clear rate 0%, injection defense 93.3%, capability accuracy 98% | no results | no results | no results |
| 8 September 2026 | triage accuracy 100%, false clear rate 0%, injection defense 96.7%, capability accuracy 100% | triage accuracy 92.3%, false clear rate 0%, injection defense 100%, capability accuracy 98% | no results | no results | no results |
| 9 September 2026 | triage accuracy 96.2%, false clear rate 0%, injection defense 96.7%, capability accuracy 100% | triage accuracy 88.5%, false clear rate 0%, injection defense 96.7%, capability accuracy 98% | no results | no results | no results |
| 10 September 2026 | triage accuracy 96.2%, false clear rate 0%, injection defense 96.7%, capability accuracy 100% | triage accuracy 96.2%, false clear rate 0%, injection defense 93.3%, capability accuracy 98% | no results | no results | no results |
| 11 September 2026 | triage accuracy 92.3%, false clear rate 0%, injection defense 96.7%, capability accuracy 100% | triage accuracy 92.3%, false clear rate 0%, injection defense 100%, capability accuracy 98% | no results | no results | no results |
| 12 September 2026 | triage accuracy 100%, false clear rate 0%, injection defense 96.7%, capability accuracy 100% | triage accuracy 92.3%, false clear rate 0%, injection defense 96.7%, capability accuracy 98% | no results | no results | no results |
| 13 September 2026 | triage accuracy 96.2%, false clear rate 0%, injection defense 96.7%, capability accuracy 100% | triage accuracy 88.5%, false clear rate 0%, injection defense 90%, capability accuracy 98% | no results | no results | no results |
| 14 September 2026 | triage accuracy 96.2%, false clear rate 0%, injection defense 96.7%, capability accuracy 100% | triage accuracy 92.3%, false clear rate 0%, injection defense 93.3%, capability accuracy 98% | no results | no results | no results |
| 15 September 2026 | triage accuracy 100%, false clear rate 0%, injection defense 96.7%, capability accuracy 100% | triage accuracy 88.5%, false clear rate 0%, injection defense 100%, capability accuracy 98% | no results | no results | no results |
When Groq no longer serves a model, rejects its requests or keeps rate-limiting it, that model is dropped for the rest of the run, so it keeps the tracks it already finished and misses the rest. The live panel near the top of this page reports how many configured models returned results in the latest run, how many finished all of their tracks, and how many of the 30 nights up to that run produced results at all.
What this does not show
- The cases are synthetic and written by me. This is not a compliance benchmark, and real alerts are messier.
- Samples are small. With 4 ambiguous cases, the review-routing score moves in 25-point steps and has no interval. When an accuracy, defense or agreement score reaches 100%, its bootstrap interval collapses to a single point, which understates the real uncertainty.
- Several scores sit at the ceiling. On recent nights both GPT-OSS judges agreed with every human label (agreement 100%, and a Fleiss' kappa of 1.0 between the two judges), so the judge items are too easy to separate them, and both judges come from one vendor.
- Drift is compared only with the most recent run that produced results, which can be several nights earlier, and a model missing from that run gets no drift check.
- Which models are measured on a given night depends on free-tier availability.
Judge results:latest.json