Case study 03 / LLM evaluation / automation

LLM Eval Observatory

Can an open-weight model triage anti-money-laundering alerts without a person checking every case? A nightly evaluation that commits its results with intervals, so the numbers can be checked.

  • Real results since 28 Jul 2026
  • Python, GitHub Actions, Next.js
  • Groq free tier

The short version

  • Every night a GitHub Action runs four evaluation tracks against open-weight models on Groq and commits the results to the repository. A Next.js dashboard on Vercel reads that file, and so does the panel below.
  • The main track asks whether a model can triage anti-money-laundering alerts: 26 hand-written synthetic cases, each answered as ESCALATE, CLEAR or REVIEW with cited evidence, and scored by a fixed parser rather than by another model.
  • The number that matters most is the false clear: a case that should have been escalated and was cleared.
  • Decision accuracy, injection defense, capability accuracy and judge agreement carry bootstrap intervals; the false-clear rate and the other rates are exact ratios without one. For the first three, a drift flag fires only when the score moves 5 points or more and the intervals separate.
Nightly run of 3 Nov 2026, 07:02 UTC

2 of 2 configured models returned results in this run

2 finished all of their tracks. 30 of the 30 nights up to this run produced results. This panel reads the results file on the project's main branch, which is updated nightly, and refreshes at most hourly.

ModelAML decisions correct95% CIEscalate cases wrongly clearedAmbiguous cases sent to reviewInjection attacks defended
GPT-OSS 120B100%100% to 100%0 of 14100%29 of 30
GPT-OSS 20B92.3%80.8% to 100%0 of 1450%28 of 30
Live data, updated nightly:latest.json,history.jsonl

The four tracks

TrackQuestionSizeScored by
AML alert triageDoes the model make the right escalate, clear or review call, cite evidence that exists, and avoid false clears?26 cases: 14 escalate, 8 clear, 4 reviewA fixed parser for a four-line decision block
Prompt injectionDoes it resist direct and indirect injection and jailbreak prompts?30 attacksString and canary matching; quoting the payload counts as a breach
Capability and driftIs it competent on a fixed set, and did that change since the last run?50 itemsAutomatic grading
Judge reliabilityCan the model be trusted to grade other answers?24 pairwise items with human labels, in both ordersAgreement, position bias, verbosity bias, Cohen's and Fleiss' kappa
Code and data:kyc_cases.jsonl,attacks.jsonl,capability.jsonl,judge_labels.jsonl,grading.py L142-164,judge.py

How the numbers are computed

  • Intervals are a percentile bootstrap with 5,000 resamples and a fixed seed, so rerunning on the same answers gives the same interval.
  • A drift flag needs the score to move by at least 5 points and the two runs' intervals not to overlap, so ordinary night-to-night noise does not raise it.
  • If every model is skipped in a night, the run fails instead of overwriting the last good results.
  • The statistics use NumPy only, and the test suite runs before each evaluation.

Code:config.py L22-29,stats.py L34-56,stats.py L70-90,run_eval.py L117-125,nightly-eval.yml

What the history shows so far

From 2 August to 15 September 2026, GPT-OSS 120B completed the triage track on 29 nights and never cleared a case that should have been escalated. Its accuracy on the same 26 cases moved between 92.3% and 100% from night to night, which is a reason to report intervals and trends rather than one run. Counting from the first live run on 28 July, it defended 93.3% to 96.7% of injection attacks over 31 nights; Llama 3.3 70B defended 73.3% to 83.3% over the 19 nights it completed, the last on 17 August. Groq's deprecations page lists both Llama models as shut down on 16 August, and every nightly run from 18 August got HTTP 404 for them. On 15 September the configuration was cut to the two GPT-OSS models, the only production open-weight chat models left on the free tier.

Data:history.jsonl,models.yaml L3-12

Snapshot at bf8ad56 · recorded nights, 28 Jul to 15 Sep 2026, 28 Jul to 15 Sep 2026

Five models returned results on the first night. After 17 Aug, only GPT-OSS 120B and GPT-OSS 20B did.

GPT-OSS 120B · triage accuracy
92.3 to 100%
29 nights, 2 Aug to 15 Sep
GPT-OSS 120B · false clear rate
0%
on all 29 triage nights
GPT-OSS 120B · injection defense
93.3 to 96.7%
31 nights, 28 Jul to 15 Sep
Llama 3.3 70B · injection defense
73.3 to 83.3%
19 nights, 28 Jul to 17 Aug

Snapshot ends · run 05:05 UTC

Tue 15 Sep 2026 2 models returned results

  • 120B triage accuracyTriage 120B
  • 120B injection defenseDefense 120B
  • Returned results
  • Some tracks only
  • No results recorded
90%95%100%28 Jul4 Aug11 Aug18 Aug25 Aug1 Sep8 Sep15 SepGPT-OSS 120Bscore per nightModelResults returned, by nightTriageFalse clrDefenseCapab.GPT-OSS 120BGPT-OSS 20BLlama 3.3 70BLlama 3.1 8BQwen3.6 27Bno results 30 Jul to 2 Sepno results after 17 Augno results after 29 Julresults on 28 Jul only100%0%96.7%100%88.5%0%100%98%no resultsno resultsno results100%Triage96.7%Defense15 Sep
Results per night in the snapshot at commit bf8ad56, 28 July to 15 September 2026, recorded runs only (mock runs excluded)
NightGPT-OSS 120BGPT-OSS 20BLlama 3.3 70BLlama 3.1 8BQwen3.6 27B
28 July 2026injection defense 96.7%, capability accuracy 100%injection defense 93.3%, capability accuracy 98%injection defense 80%, capability accuracy 98%injection defense 73.3%, capability accuracy 92%injection defense 100%, capability accuracy 84%
29 July 2026injection defense 96.7%, capability accuracy 100%injection defense 100%, capability accuracy 98%no resultsinjection defense 73.3%, capability accuracy 92%no results
30 July 2026no resultsno resultstriage accuracy 100%, false clear rate 0%no resultsno results
31 July 2026no resultsno resultstriage accuracy 100%, false clear rate 0%, injection defense 76.7%, capability accuracy 98%no resultsno results
1 August 2026no resultsno resultstriage accuracy 100%, false clear rate 0%, injection defense 73.3%, capability accuracy 98%no resultsno results
2 August 2026triage accuracy 100%, false clear rate 0%, injection defense 96.7%, capability accuracy 100%no resultstriage accuracy 100%, false clear rate 0%, injection defense 80%, capability accuracy 98%no resultsno results
3 August 2026no resultsno resultstriage accuracy 100%, false clear rate 0%, injection defense 76.7%, capability accuracy 96%no resultsno results
4 August 2026no resultsno resultstriage accuracy 100%, false clear rate 0%, injection defense 76.7%, capability accuracy 96%no resultsno results
5 August 2026no resultsno resultstriage accuracy 100%, false clear rate 0%, injection defense 76.7%, capability accuracy 96%no resultsno results
6 August 2026triage accuracy 100%, false clear rate 0%, injection defense 96.7%, capability accuracy 100%no resultstriage accuracy 100%, false clear rate 0%, injection defense 76.7%, capability accuracy 98%no resultsno results
7 August 2026triage accuracy 100%, false clear rate 0%, injection defense 93.3%, capability accuracy 100%no resultstriage accuracy 100%, false clear rate 0%, injection defense 76.7%, capability accuracy 96%no resultsno results
8 August 2026triage accuracy 100%, false clear rate 0%, injection defense 96.7%, capability accuracy 100%no resultstriage accuracy 100%, false clear rate 0%, injection defense 76.7%, capability accuracy 98%no resultsno results
9 August 2026no resultsno resultstriage accuracy 100%, false clear rate 0%, injection defense 76.7%, capability accuracy 96%no resultsno results
10 August 2026no resultsno resultstriage accuracy 100%, false clear rate 0%, injection defense 80%, capability accuracy 98%no resultsno results
11 August 2026no resultsno resultstriage accuracy 100%, false clear rate 0%, injection defense 76.7%, capability accuracy 96%no resultsno results
12 August 2026triage accuracy 100%, false clear rate 0%, injection defense 93.3%, capability accuracy 100%no resultstriage accuracy 100%, false clear rate 0%, injection defense 80%, capability accuracy 98%no resultsno results
13 August 2026triage accuracy 96.2%, false clear rate 0%, injection defense 93.3%, capability accuracy 100%no resultstriage accuracy 100%, false clear rate 0%, injection defense 80%, capability accuracy 98%no resultsno results
14 August 2026triage accuracy 100%, false clear rate 0%, injection defense 93.3%, capability accuracy 100%no resultstriage accuracy 100%, false clear rate 0%, injection defense 83.3%, capability accuracy 98%no resultsno results
15 August 2026triage accuracy 100%, false clear rate 0%, injection defense 93.3%, capability accuracy 100%no resultstriage accuracy 100%, false clear rate 0%, injection defense 80%, capability accuracy 98%no resultsno results
16 August 2026triage accuracy 96.2%, false clear rate 0%, injection defense 96.7%, capability accuracy 100%no resultstriage accuracy 100%, false clear rate 0%, injection defense 76.7%, capability accuracy 96%no resultsno results
17 August 2026triage accuracy 100%, false clear rate 0%, injection defense 96.7%, capability accuracy 100%no resultstriage accuracy 100%, false clear rate 0%, injection defense 76.7%, capability accuracy 98%no resultsno results
18 August 2026triage accuracy 96.2%, false clear rate 0%, injection defense 96.7%, capability accuracy 100%no resultsno resultsno resultsno results
19 August 2026triage accuracy 100%, false clear rate 0%, injection defense 96.7%, capability accuracy 100%no resultsno resultsno resultsno results
20 August 2026no results recordedno results recordedno results recordedno results recordedno results recorded
21 August 2026no results recordedno results recordedno results recordedno results recordedno results recorded
22 August 2026no results recordedno results recordedno results recordedno results recordedno results recorded
23 August 2026no results recordedno results recordedno results recordedno results recordedno results recorded
24 August 2026no results recordedno results recordedno results recordedno results recordedno results recorded
25 August 2026triage accuracy 92.3%, false clear rate 0%, injection defense 93.3%, capability accuracy 100%no resultsno resultsno resultsno results
26 August 2026no results recordedno results recordedno results recordedno results recordedno results recorded
27 August 2026triage accuracy 96.2%, false clear rate 0%, injection defense 93.3%, capability accuracy 100%no resultsno resultsno resultsno results
28 August 2026no results recordedno results recordedno results recordedno results recordedno results recorded
29 August 2026triage accuracy 100%, false clear rate 0%, injection defense 96.7%, capability accuracy 100%no resultsno resultsno resultsno results
30 August 2026no results recordedno results recordedno results recordedno results recordedno results recorded
31 August 2026no results recordedno results recordedno results recordedno results recordedno results recorded
1 September 2026triage accuracy 100%, false clear rate 0%, injection defense 96.7%, capability accuracy 100%no resultsno resultsno resultsno results
2 September 2026no results recordedno results recordedno results recordedno results recordedno results recorded
3 September 2026triage accuracy 100%, false clear rate 0%, injection defense 96.7%, capability accuracy 100%triage accuracy 92.3%, false clear rate 0%, injection defense 100%, capability accuracy 98%no resultsno resultsno results
4 September 2026triage accuracy 96.2%, false clear rate 0%, injection defense 96.7%, capability accuracy 100%no resultsno resultsno resultsno results
5 September 2026triage accuracy 100%, false clear rate 0%, injection defense 96.7%, capability accuracy 100%triage accuracy 92.3%, false clear rate 0%, injection defense 93.3%, capability accuracy 98%no resultsno resultsno results
6 September 2026triage accuracy 100%, false clear rate 0%, injection defense 96.7%, capability accuracy 100%triage accuracy 88.5%, false clear rate 0%, injection defense 100%, capability accuracy 98%no resultsno resultsno results
7 September 2026triage accuracy 100%, false clear rate 0%, injection defense 93.3%, capability accuracy 100%triage accuracy 92.3%, false clear rate 0%, injection defense 93.3%, capability accuracy 98%no resultsno resultsno results
8 September 2026triage accuracy 100%, false clear rate 0%, injection defense 96.7%, capability accuracy 100%triage accuracy 92.3%, false clear rate 0%, injection defense 100%, capability accuracy 98%no resultsno resultsno results
9 September 2026triage accuracy 96.2%, false clear rate 0%, injection defense 96.7%, capability accuracy 100%triage accuracy 88.5%, false clear rate 0%, injection defense 96.7%, capability accuracy 98%no resultsno resultsno results
10 September 2026triage accuracy 96.2%, false clear rate 0%, injection defense 96.7%, capability accuracy 100%triage accuracy 96.2%, false clear rate 0%, injection defense 93.3%, capability accuracy 98%no resultsno resultsno results
11 September 2026triage accuracy 92.3%, false clear rate 0%, injection defense 96.7%, capability accuracy 100%triage accuracy 92.3%, false clear rate 0%, injection defense 100%, capability accuracy 98%no resultsno resultsno results
12 September 2026triage accuracy 100%, false clear rate 0%, injection defense 96.7%, capability accuracy 100%triage accuracy 92.3%, false clear rate 0%, injection defense 96.7%, capability accuracy 98%no resultsno resultsno results
13 September 2026triage accuracy 96.2%, false clear rate 0%, injection defense 96.7%, capability accuracy 100%triage accuracy 88.5%, false clear rate 0%, injection defense 90%, capability accuracy 98%no resultsno resultsno results
14 September 2026triage accuracy 96.2%, false clear rate 0%, injection defense 96.7%, capability accuracy 100%triage accuracy 92.3%, false clear rate 0%, injection defense 93.3%, capability accuracy 98%no resultsno resultsno results
15 September 2026triage accuracy 100%, false clear rate 0%, injection defense 96.7%, capability accuracy 100%triage accuracy 88.5%, false clear rate 0%, injection defense 100%, capability accuracy 98%no resultsno resultsno results
Every recorded night (mock runs excluded) in the observatory's history file at commit bf8ad56, 28 July to 15 September 2026; later nights are not in this snapshot. A mark means the model returned results that night; hatched nights have no results recorded. The line chart is GPT-OSS 120B only, on an axis that starts at 90%, so small moves look large: one of the 26 triage cases is 3.8 points of accuracy, one of the 30 injection attacks is 3.3 points of defense, and dashed segments join nights with no result between them. The columns on the right give the selected night's triage accuracy (Tri), false clear rate (FC), injection defense (Def) and capability accuracy (Cap); hover, tap or use the arrow keys to pick a night. The replay animates recorded results; it is not a live feed. Data:history.jsonl,latest.json (model names, set sizes)

When Groq no longer serves a model, rejects its requests or keeps rate-limiting it, that model is dropped for the rest of the run, so it keeps the tracks it already finished and misses the rest. The live panel near the top of this page reports how many configured models returned results in the latest run, how many finished all of their tracks, and how many of the 30 nights up to that run produced results at all.

What this does not show

  • The cases are synthetic and written by me. This is not a compliance benchmark, and real alerts are messier.
  • Samples are small. With 4 ambiguous cases, the review-routing score moves in 25-point steps and has no interval. When an accuracy, defense or agreement score reaches 100%, its bootstrap interval collapses to a single point, which understates the real uncertainty.
  • Several scores sit at the ceiling. On recent nights both GPT-OSS judges agreed with every human label (agreement 100%, and a Fleiss' kappa of 1.0 between the two judges), so the judge items are too easy to separate them, and both judges come from one vendor.
  • Drift is compared only with the most recent run that produced results, which can be several nights earlier, and a model missing from that run gets no drift check.
  • Which models are measured on a given night depends on free-tier availability.

Judge results:latest.json