Research dashboard
Experiment results
Compare controlled-language prompt strategies with paired effects and confidence intervals.
At a glance
Naming versus bare baseline
Each line joins a model's bare-prompt mean compliance score to its mean when ASD-STE100 is named without rules. Shaded bands show 95% confidence intervals; chevrons mark bands that extend past the 0 to 100 scale. The table below lists the exact values.
- Bare baseline
- Named without rules
- 95% confidence interval
About the experiment
The four variants are bare (prompt alone), rules (rules spelled out), named (technique named), and named_rules (name and rules together). Compliance scores run from 0 to 100; higher scores mean greater compliance. Effects are paired changes in score points.
- Named without rules
- Mean compliance score when ASD-STE100 is named without spelling out its rules.
- Bare baseline
- Mean score for the bare prompt, before naming or rules are added.
- Rule effect
- Mean paired change from spelling out the rules, averaged with and without naming.
- Naming effect
- Mean paired change from naming the technique, averaged with and without rules.
- Interaction
- The extra paired change when naming and rules appear together, beyond their separate effects.
- Paired observations
- Complete four-arm research cells used to calculate every value in the row.
A negative interaction may reflect overlap between naming the technique and spelling out its rules, but the interaction does not identify its cause.
Run time is the time the row's runs spent working, summed from when each answer was saved. Pauses while a run sat stalled before being resumed are not counted; each one is replaced by the run's typical time for one answer.
Total cost is the sum of OpenRouter's reported charges for every paid request in the row's runs, including continuations and judge calls. Runs from before cost tracking, or with a request whose charge was not reported, are estimated from their saved text at about four characters per token and current catalog prices; a row that includes any such run is marked (estimated). Estimates leave out hidden reasoning tokens, retries and continuations, so actual spending may be higher. Unavailable means a run could not be estimated, for example because its model is no longer in the catalog.
Measured outcomes
Leaderboard
| Rank | Model | Named without rules, mean ± 95% CI | Bare baseline | Rule effect | Naming effect | Interaction | Paired sessions | Run time | Total cost |
|---|---|---|---|---|---|---|---|---|---|
| 1 | deepseek/deepseek-v4.1-flash | 99.4 ± 0.4 | 97.2 ± 1.7 | +1.5 ± 1.3 | +1.0 ± 0.8 | -2.4 ± 1.3 | 6 | 1h 32m | $0.35 (estimated) |
| 2 | deepseek/deepseek-v4-flash-0731 | 98.9 ± 0.3 | 95.5 ± 3.7 | +2.8 ± 1.9 | +1.7 ± 1.8 | -3.4 ± 3.7 | 6 | 1h 11m | $0.11 (estimated) |
| 3 | deepseek/deepseek-v4.1-flash-1017 | 98.8 ± 1.2 | 93.9 ± 5.0 | +3.3 ± 3.1 | +2.3 ± 2.3 | -5.4 ± 3.6 | 6 | 1h 8m | $0.31 (estimated) |
| 4 | openai/gpt-5.6-luna | 98.0 ± 1.2 | 92.0 ± 6.0 | +5.0 ± 3.4 | +3.0 ± 2.6 | -6.0 ± 5.3 | 6 | 19m 48s | $0.20 (estimated) |
| 5 | openai/gpt-6-luna | 97.8 ± 1.2 | 90.7 ± 3.2 | +4.6 ± 1.1 | +2.9 ± 2.0 | -8.6 ± 4.2 | 6 | 21m 20s | $0.06 |
| 6 | google/gemini-3.8-flash | 97.2 ± 1.7 | 95.7 ± 1.9 | +3.6 ± 1.4 | +0.8 ± 1.1 | -1.5 ± 2.2 | 6 | 58m 25s | $0.96 (estimated) |
| 7 | anthropic/claude-haiku-5.5 | 96.6 ± 1.6 | 95.0 ± 2.8 | +3.6 ± 2.0 | +0.4 ± 1.9 | -2.6 ± 2.7 | 6 | 34m 45s | $0.15 (estimated) |
| 8 | meta-llama/llama-4-maverick | 89.3 ± 5.6 | 86.7 ± 4.5 | +9.5 ± 4.4 | +1.8 ± 3.6 | -1.8 ± 3.9 | 6 | 27m 35s | $0.08 (estimated) |
| 9 | anthropic/claude-haiku-4.5 | 89.0 ± 12.0 | 73.3 ± 9.7 | +18.2 ± 8.0 | +7.7 ± 6.8 | -16.1 ± 14.7 | 6 | 39m 23s | $2.04 (estimated) |
Understanding uncertainty
A 95% confidence interval (95% CI) uses the Student t critical value for the number of contributing sessions. Wider intervals indicate greater uncertainty. It is neither the range of individual scores nor a 95% probability that the fixed population effect lies inside this observed interval. Interpret estimates and uncertainty together, rather than reducing them to statistically significant or not significant.
In the leaderboard table, each value is written as a mean ± the half-width of its 95% CI, so the interval runs from the mean minus that amount to the mean plus it. ± n/a marks rows with fewer than 2 sessions, which cannot support an interval.
Only complete four-arm research cells matched within run, model, session, depth, protocol version, and scoring version are included; previews and incomplete runs are excluded. Effects are calculated within each matched cell, then repeated depths within each run and session are averaged before intervals are calculated. Sessions are treated as independent; this is not a hierarchical analysis across runs or other levels.