← back to the story · the arena
DadBench — technical appendix
Everything on this page is reconstructable from the research diary (48 dated entries) and the result files. Numbers here are the ones the story cites; nothing is rounded beyond three decimals.
Checkpoint names: the two shipped models are OURS:r5 = PunTune 0.5 and OURS:R6 = PunTune 0.6; other OURS:r4 / R4 / R5 labels are intermediate reward-round checkpoints, kept here for precise reference.
1 · Task and prompts
System prompt, identical for every model and every training run:
You are a master of dad jokes. When given a topic, reply with exactly one dad joke about that topic. The joke must be ORIGINAL — invented by you, not a well-known existing joke. No preamble, no explanation.
User message: Topic: {topic}. 50 prompts across four tracks: 28 plain topics
(escalators, escrow accounts, the Oxford comma…), 8 cultural references (Mahomes, Grogu…),
8 comparisons (UI vs UX, GRPO vs DPO…), 6 full news articles pasted as the topic (long-context
track). An early bug nested instruction text inside the topic slot for two tracks
(Topic: Tell me a dad joke about X); fixed before the final run, verified in
rendered traces.
2 · Training configuration
| item | value |
|---|---|
| model | thinkingmachines/Inkling (975B MoE, 41B active), via Tinker |
| method | GRPO (policy-gradient, REINFORCE family), tinker-cookbook loop |
| adapter | LoRA rank 32 |
| schedule | 100 optimizer steps/run · 8 groups × 8 siblings = 64 episodes/step |
| lr / max_tokens | 4e-5 · 1,100 (training); 3,000 (final eval generation) |
| thinking effort | 0.9 during all training (the renderer default; the sweep came later) |
3 · The seven reward designs
| run | reward | outcome |
|---|---|---|
| R1 "craft rubric" | 20-item weighted rubric scored absolutely by an LLM judge | rubric +20pts, human rating flat; rubric score correlated r = −0.45 with the human |
| R2 "rank" | judge ranks the 8 siblings; reward = rank percentile | first real human gain (1.33→2.00); template collapse appears |
| R3 "taste anchors" | rank vs canon-classic anchors in the lineup | 2.33 human; canon defends its crown |
| R4 "gemma judge" | rank under exam-selected judge (gemma-4-31b, ρ=0.636 vs human at the tryouts that picked it; later re-exams on a wider field, then under reasoning parity, put it at 0.516 then 0.489 — see §5) | 2.56 human — the round the author liked most at the time |
| R5 "fame gate" | R4 + originality gate: trigram containment vs 10,561-joke corpus; fame 0→+10/×1.0 · 1→+2/×0.8 · ≥2→−20/×0.3 | flagship; famous jokes gutted, parallel invention rides free |
| R6 "reader's objection" | rebuilt 20-item rubric, items measured by ranking, stacking priced negative, fame-gated (the author's pre-registered counter-design) | 3.22 blind; converges to R5's behavior — see saturation law below |
| R7 (spec'd) | span-level rewards (per-token advantages) — feasibility verified, not yet run | — |
The saturation law: in group-relative RL, any criterion every sibling can satisfy contributes zero within-group variance and therefore zero gradient. Additive rubrics self-erase down to their comparative residue — which is why R6 and R5 converged.
4 · Human-rating protocol
One rater (the author), blind to model identity, 1–5 scale, exact-text ledger: 143 ratings across 6 sittings, 101 unique jokes (ours, frontier-style imitations, canon classics). The fixed canon anchor set was re-rated in five of those sittings: 3.33 / 3.50 / 3.50 / 3.33 / 3.17. Known limitation: n=1 rater — the arena exists to fix this.
5 · Judge selection (two exams)
Exam 1 (P6 era, uncontrolled reasoning defaults)
Ten candidates ranked the ledger corpus; Spearman ρ vs the human, with a known-bad model planted as a negative control (it flunked at 0.141 — the exam discriminates). muse-spark-1.1 0.570 · kimi-k2.6 0.551 (n=36, fails frontier floor) · gemma-4-31b 0.516 (0.597 ours / 0.337 frontier — home-style bias, conflicted family) · gpt-5.4-mini 0.461 · luna 0.444 · mimo-v2.5 0.419 · nemotron-3-ultra 0.418 · sonnet-5 0.406 · haiku-4.5 0.266. Seated: muse + mimo + nemotron (zero family overlap with contestants).
Exam 2 (reasoning-parity re-exam)
Billing data showed judge calls carried no reasoning parameter — muse was thinking ~1,170 tokens/call vs nemotron's ~120 under provider defaults. Re-examined under a uniform 2,048 reasoning cap, same method for all rows: muse-default 0.506 · mimo-capped 0.501 at half the price · gemma 0.489 · qwen3.7-plus 0.472 (best frontier split 0.57, $0.0017/call) · nemotron-capped 0.458 · deepseek-pro 0.449 · minimax 0.371 · deepseek-flash-high 0.313. Muse's crown was method noise + default-compute asymmetry. Final panel: mimo-v2.5-pro + qwen3.7-plus + nemotron-3-ultra, all reasoning-capped 2,048. Judge wording (validated by ablation): "which would a real dad be proudest to inflict on his family?"
6 · The anchor-ladder instrument
Construction: per prompt, 6–7 reference jokes with frozen values — frontier material from 1,348 panel rankings (3 passes × 3 judges over the deep corpus; our models ineligible as rungs) plus corpus classics that survived a stability screen (45 earned rungs), pruned to ≥0.08 value spacing. Scoring: a candidate is inserted into its ladder, the panel ranks the lineup blind (order-randomized), and each placement yields one win/loss against every rung; the score is the value where the win-rate curve crosses 0.5 (monotone-smoothed interpolation). Field rows: 9 placements (3 judges × 3 passes); sweep arms: 3.
Certification (panel v2, two independent runs): split-half → full-6 reliability 0.691 / 0.661 (Spearman–Brown projects ~0.745 at 9 placements — the deployed setting); blind rung-integrity 0.585 / 0.625; sibling triangulation reproduced the known ordering (R6 ≈ r5 > base) in both runs. Replication: R6's corpus scored 0.461 during certification and its fresh generation scored 0.456 the next day (Δ=0.005).
Known censoring: the estimator clamps at bottom-rung −0.03 / top-rung +0.03. About a third of bottom-row jokes sit at the floor clamp (base 50/149, Terra 49/150, Opus 43/150), so bottom-row means are floor-supported and true gaps are larger; Kimi (17) and Gemini (12) touch the ceiling. Extending the ladders downward is queued for v2.1.
7 · Compute parity (contestants and judges)
Uniform reasoning.max_tokens = 2048 on all frontier generation (one measured
exception: the parameter suppresses Kimi's thinking instead of capping it; Kimi runs native,
in-band). Realized uptake still varies ~100× under the equal ceiling: Gemini 1,274 median
thinking tokens/joke; Opus and Terra ~0. Verified in triplicate (OpenRouter at every effort
setting; OpenAI direct — effort-high buys Terra 9 thinking tokens on a joke; Anthropic direct —
Opus 4.8's thinking is adaptive-only, no force-think exists). Judge calls carry the
same 2,048 cap as of panel v2.
8 · The effort sweep (E1, definitive)
Protocol: base Inkling, R5, and R6 at thinking-effort 0.2 / 0.5 / 0.7 / 0.9 / 0.99; every cell 150 fresh jokes × 9 blind placements on the frozen instrument (one protocol, 15 cells ≈ 20,250 placements, ~14,850 of them newly run — four full-protocol cells were reused). An earlier exploratory sweep used 50 jokes × 3 placements per point; it produced curve shapes (a flat base, a monotone fine-tune rise to 0.516 at max) that did NOT survive the full protocol and is superseded by this table. Estimator lesson kept for the record: a 50×3 cell can mislead by ±0.09.
| model | 0.2 | 0.5 | 0.7 | 0.9 | 0.99 | slope |
|---|---|---|---|---|---|---|
| inkling-base | 0.332 | 0.362 | 0.375 | 0.374 | 0.373 | +0.041 |
| OURS:r5 | 0.370 | 0.370 | 0.405 | 0.421 | 0.427 | +0.058 |
| OURS:R6 | 0.370 | 0.417 | 0.419 | 0.456 | 0.421 | +0.051 |
Manipulation check passed everywhere (median completion tokens scale monotonically: base 156→763, r5 97→304, R6 79→218). Findings: modest similar slopes for all three; the fine-tunes' advantage is level and token-economy, not dial-response; low effort genuinely costs R6 (0.370 vs 0.456, disjoint CIs); R6's peak at its training effort (0.9) is registered as a hypothesis, not a claim. R6@0.99 replicated exactly across two independent 150×9 measurements (0.421 both). Trace review: base's max-effort thinking is structured multi-candidate brainstorming that fails at the commit (merges candidates); R6 selects one and stops.
9 · Cost accounting
| item | spend |
|---|---|
| Training: seven GRPO runs + benchmark passes (verified) | ~$200 |
| Judge cleanup: judge exams, panel rebuilds, field tournaments + audit | ~$50 |
| project total | ~$250 — still less than one conference ticket |
Judging economics at panel-v2 prices: ~$0.0023/placement → a new model's full row costs ≈ $3; a training checkpoint ≈ $5. Ops lessons: never meter two concurrent runs on one credit counter; per-joke depth-first placement with incremental caching (breadth-first judge-walking made hours of progress invisible and got two healthy runs killed).
10 · Limitations, stated plainly
One human rater anchors all validity numbers. Judge validity has a shape: strong on stacked multi-pun jokes (ρ=0.73), inverted within minimal two-beat jokes (ρ=−0.47) — the basement rows write two-beat, so their internal ordering is low-confidence. The culture track partly measures judge knowledge. Floors are censored (§6). Realized reasoning is unequal under equal permission (§7). Slot-diversity collapse in minimal-style models makes their CIs slightly tight. Effort-sweep arms are n=150/point with 9-placement noise.
11 · Archive: the P6 field-relative tournament
The last field-relative boards before the ladder instrument (Bradley–Terry strengths, 3 exam-seated judges × 2 shuffled passes, compute parity, canon anchor in every heat). Preserved because the ladder era supersedes the numbers but not the two findings the story cites (Gemini-under-rationing; the two-beat basement style).
| # | model | BT strength | 95% CI |
|---|---|---|---|
| 1 | gemini-3.1-pro | 2.202 | 2.027–2.361 |
| 2 | gemini-3.5-flash | 1.744 | 1.624–1.878 |
| 3 | kimi-k3 | 1.409 | 1.324–1.510 |
| 4 | claude-fable-5 | 1.153 | 1.064–1.247 |
| 5 | glm-5.2 | 1.106 | 1.023–1.181 |
| 6 | grok-4.5 | 0.837 | 0.770–0.903 |
| 7 | OURS:r5 | 0.810 | 0.744–0.887 |
| 8 | gpt-5.6-sol | 0.722 | 0.668–0.782 |
| 9 | OURS:R6 | 0.712 | 0.606–0.831 |
| 10 | OURS:r4 | 0.677 | 0.624–0.721 |
| 11 | claude-opus-4.8 | 0.575 | 0.532–0.614 |
| 12 | gpt-5.6-terra | 0.539 | 0.500–0.573 |
| 13 | inkling-base | 0.514 | 0.464–0.575 |
Style receipts under parity
| model | think tok | words | Q&A% | canon-adjacent% |
|---|---|---|---|---|
| gemini-3.1-pro | 1452 | 19.8 | 95% | 0.7% |
| gemini-3.5-flash | 974 | 21.1 | 94% | 2.7% |
| kimi-k3 | 1525 | 19.5 | 65% | 0.0% |
| claude-fable-5 | 98 | 21.7 | 46% | 0.0% |
| glm-5.2 | 1017 | 21.8 | 69% | 0.0% |
| grok-4.5 | 897 | 17.6 | 90% | 2.7% |
| OURS:R5 | 1100* | 14.7 | 14% | 3.3% |
| gpt-5.6-sol | 77 | 15.2 | 36% | 0.0% |
| OURS:R4 | 1100* | 11.6 | 19% | 2.0% |
| claude-opus-4.8 | 29 | 21.9 | 66% | 2.0% |
| gpt-5.6-terra | 20 | 15.6 | 38% | 0.7% |
| inkling (base) | 1100* | 184.9 | 35% | 0.7% |
*ours: fixed 1,100 total token budget in that era. canon-adjacent = fame-index match rate vs 10,561 real jokes.
12 · Tinker field notes: the sharp edges in full
chz vs dataclass. Subclassing ProblemGroupBuilder mixes chz-configured
classes with frozen dataclasses; the error surfaces late and cryptically. A cookbook example of
a custom GroupBuilder would save every RL newcomer an evening.
Sampling-only deployments. Serving a trained LoRA from a container needs only prompt
rendering plus the sampling API, but the cookbook renderer path pulls torch. The standalone
tml_renderers package renders byte-identical prompts with zero heavy deps — it
deserves a documented "deploy your checkpoint" recipe. Slim containers also need system CA
certs for the SDK's rust transport, and use_pyqwest_transport: False is the escape
hatch (the rust transport cannot connect from some container networks).