Skip to content
Dark Cosmos — Dev Docs

Cheap-model subagent evaluation — full matrix design

Which cheap model is good at what — a full-matrix evaluation

Section titled “Which cheap model is good at what — a full-matrix evaluation”

Written 2026-09-30 from Chris’s brief: “try out the lowest token use models droid offers as sub agents and see which models perform the best at what tasks.” Chris chose the full matrix in one go (~12 cheap models × 6 tasks × 2-3 runs). Companion plan: 2026-09-30-factory-missions-catalog.md, which consumes the results.

0. The question this answers, and the one it doesn’t

Section titled “0. The question this answers, and the one it doesn’t”

Not “which model is best.” The routing decisions Droid actually exposes are per-seat:

Seat Setting Today
Subagent light/medium/heavy tiers subagentModelSettings.{lightModel,mediumModel,heavyModel} unset (inherits parent)
A specific custom droid model: in .factory/droids/<name>.md not ported yet
Mission workers missionModelSettings.workerModel unset
Mission validators missionModelSettings.validationWorkerModel unset
Session default sessionDefaultSettings.model glm-5.3

So the output is a task-kind → model table, not a ranking, and it lands as concrete config in four places.

The headline metric is not pass rate. It is the false-green rate: a run where the agent reports success and the gate disagrees. This project has been burned by exactly that, twice and expensively — four Haiku worktree implementers on 2026-09-27 all reported green and all needed rework (a test file that never parsed while the report claimed “17 tests pass”; a pitch formula that was constant; a codex preview that rendered every enemy as a Swarmer; one agent pushed to origin/main against instructions). A model that fails honestly is cheaper than a model that succeeds dishonestly, because the second one costs a human review cycle and trust.

1. Arms — 12 cheap models + 2 controls

Section titled “1. Arms — 12 cheap models + 2 controls”

Multipliers from the live Factory catalogue (2026-09-30). “Cheap” = ≤ 0.7×.

# Model ID Mult. Notes
1 gpt-6-luna 0.04× Cheapest thing on the platform, 25× under Opus 5.5
2 glm-5.3-flash 0.06×
3 gpt-5.6-luna 0.08×
4 deepseek-v4.1-flash 0.12× In the Factory Router pool
5 minimax-m3 0.12× In the router pool; high only
6 gemini-3-flash-preview 0.2×
7 nemotron-3-ultra 0.24×
8 gpt-5.4-mini 0.3×
9 gemini-3.8-flash 0.3× promo until 2027-01-01, then 0.6×
10 claude-haiku-4-5-20251001 0.4× The model that produced the 2026-09-27 failures — include it to quantify that, not to assume it
11 inkling 0.4× Factory’s own open model
12 gpt-5.3-codex 0.7× Code-specialised; the most expensive “cheap” arm
C1 glm-5.3 0.56× Incumbent control — Chris’s current session default
C2 claude-opus-5-5 1.6× Ceiling control — what we are trying to avoid paying for

Two capability caveats to record per arm rather than discover mid-run:

  • glm-5.3 has no image inputs (verified 2026-09-30). Any arm may lack vision; the matrix has no image task for that reason, but the capability is worth a column in the final table since Chris’s workflow is screenshot-driven ~200×/day.
  • Reasoning effort is not uniform. minimax-m3 accepts only high; mistral-medium-3.5 only off/high. Pin each arm at its default effort so the comparison is “the model as you’d actually use it”, and record the effort used.

2. Tasks — 6 kinds, each with ground truth

Section titled “2. Tasks — 6 kinds, each with ground truth”

The trick that makes this objective: every fixture is a real merged commit, reverted in a worktree. The reference diff is the ground truth, and scripts/gate.sh is the judge. No rubric invented for the occasion.

ID Kind Fixture (pick during setup; shape shown) Judge
T1 Read-only exploration 10 questions with known answers from the architecture docs: “Where does damage get subtracted, and what removes a dead enemy?” (elemental_system.damage_enemy / end-of-tick Sim._sweep_dead), “Which RNG stream rolls upgrade choices?” (upgrade_rng), “Why is biomass_v a trap in EnemyPool.add()?” Answer key, scored 0/1 per question; no file writes permitted
T2 Mechanical refactor The open cooldown_frac() dedup: extract WeaponCooldown.frac(timer, cooldown) and apply across the ~13 sim/weapon_*.gd files that implement it identically, leaving the bespoke weapons (orbit, chronowell/turret, scar_tissue) alone gate.sh green and determinism baselines unmoved and the bespoke weapons untouched
T3 Test authoring Revert a known bug-fix commit (e.g. the Story/Tutorial zero-XP fix) and ask for a regression test that drives the real spawn→kill→_sweep_dead path Mutation-proved: the test must fail on the reverted code and pass on the fixed code. A test that passes on both scores zero, however well-written
T4 Content wiring Revert a merged “add one enemy” commit; ask for it back, following bh-add-content. Choose a fixture that required no determinism re-pin gate.sh + the skill’s wiring checklist + explicit XP-award assertion (the biomass_v trap)
T5 Script/tooling (non-GDScript) A Node task against scripts/test/*.test.mjs or telemetry/test/ — e.g. the unwritten BIOMASS_TRIGGER <= BIOMASS_TARGET cross-key validation in publish-config.mjs node --test scripts/test/*.test.mjs (directory form fails MODULE_NOT_FOUND on Node 22 — use the glob)
T6 Doc synthesis Produce a weapon/enemy roster table sourced from data/bible.json + live sim consts, the exact job the public site got wrong (7→16 weapons, 17→25 enemies, boss HP off 2-4×) Automated diff against the bible: every number either matches or is a scored error

Coverage rationale: T1 maps to the explorer seat, T2/T4 to mission workers, T3 to the seat where false-greens hurt most, T5 checks whether “good at GDScript” generalises off-GDScript (typed GDScript / Godot 4.7 is probably thin in open-model training data — the original offload hypothesis), T6 to the cheap prose seat.

3. Harness

Section titled “3. Harness”

New tools/model_eval/:

tools/model_eval/
  run_matrix.sh          # the driver: arms × tasks × repeats
  tasks/
    t1_exploration/{PROMPT.md, answers.json}
    t2_refactor/{PROMPT.md, fixture.sh, reference.diff}
    t3_test_authoring/{PROMPT.md, fixture.sh, mutation.sh}
    t4_content/{PROMPT.md, fixture.sh, reference.diff}
    t5_tooling/{PROMPT.md, fixture.sh}
    t6_docs/{PROMPT.md, expected.json}
  score.mjs              # per-run scoring → one JSON line per run
  report.mjs             # aggregate → markdown table + per-task winners

Each run is one headless invocation:

Terminal window
droid exec \
  --model "$ARM" \
  --cwd "$WORKTREE" \
  --auto medium \
  --output-format json \
  --tag "model-eval" --tag "{\"name\":\"arm\",\"value\":\"$ARM\"}" \
  --tag "{\"name\":\"task\",\"value\":\"$TASK\"}" \
  --log-group-id "model-eval-$DATE" \
  -f "tasks/$TASK/PROMPT.md"

Decisions baked into the harness:

  • --auto medium, never high. Medium allows edits, builds, tests and local commits but blocks git push — which directly prevents a repeat of the 2026-09-27 rogue push. T1 runs with no --auto flag at all (read-only default).
  • One git worktree per run, created fresh from the fixture commit and destroyed after scoring, so a run can neither see nor corrupt another’s tree (droid exec -w <name> does this natively; clean worktrees auto-remove, dirty ones are preserved for inspection — which is what we want on failure).
  • Never on main. Never touching platform/tvos/.
  • --tag + --log-group-id on every run, so the Factory Analytics API can attribute real credit spend per arm afterwards rather than relying on an estimate.
  • Concurrency split. The agent phase can run ~4-wide. The scoring phase must serialise: each gate.sh spawns a headless Godot and the full GUT suite; four at once will distort wall-time measurements and may thrash the machine. Queue scoring behind a lock.
  • Prompts are identical across arms. No per-model prompt tuning — that would measure prompt effort, not the model. Each PROMPT.md states the task, the repo conventions that apply, the acceptance command, and the explicit instruction to report honestly whether it passed.

4. Scoring

Section titled “4. Scoring”

Per run, one JSON record:

Field Source Why
passed task judge (gate/mutation/answer key) correctness
claimed_success parsed from the agent’s final message —
false_green claimed_success && !passed the headline metric
duration_ms, num_turns droid exec --output-format json latency and thrash
tokens_in/out session settings file / Analytics API real cost basis
cost_proxy tokens × multiplier cross-arm comparability
diff_size, files_touched git blast radius; a “passing” 900-line diff on a 40-line task is a fail in practice
scope_violations grep the diff for forbidden paths touched platform/tvos/, hand-edited BUILD, re-pinned a determinism baseline, pushed
rework_notes human, sampled the qualitative residue

Aggregate per (arm × task): pass rate, false-green rate, median cost, median wall time. Then a cost-per-passed-run column, which is the number that actually decides seating — a 0.04× model that passes 1 run in 4 is more expensive than a 0.56× model that passes 3 in 4, once a human review cycle is priced in.

Tie-break rule. Two repeats per cell (14 arms × 6 tasks × 2 = 168 runs). Where the two repeats disagree, run a third. Expect roughly 15-30 tie-breaks, landing the total at ~185-200 runs, matching the chosen scale.

Variance is a finding, not noise. A model that passes T2 once and fails it once is unseatable for unattended mission work regardless of its average.

5. Budget and schedule

Section titled “5. Budget and schedule”

Rough order of magnitude: a T2-style task is perhaps 50-150k tokens of context and output per run. At ~185 runs, the cheap arms cost very little; the two controls dominate. claude-opus-5-5 at 1.6× across 12 cells × 2-3 runs is the single largest line item — it is also the only way to know what the cheap arms are being compared against, so keep it and accept the cost.

Sequencing: setup (fixtures + harness + scorer) is a day’s work and is the part that determines whether the results mean anything. Then the matrix runs unattended in batches, ideally overnight, with the scoring queue serialised. Spot-check the first 10 runs by hand before letting the rest go — a broken fixture discovered at run 180 wastes the whole batch.

6. Deliverables

Section titled “6. Deliverables”
  1. tools/model_eval/ — the harness, committed, re-runnable when the catalogue changes (it moves monthly: promo pricing on gemini-3.7/3.8-flash ends 2027-01-01; five Droid Core models are already deprecated).
  2. reports/model-eval/<date>/runs.jsonl + report.md — raw records and the aggregate table.
  3. docs/research/2026-XX-XX-cheap-model-subagent-eval.md — the findings write-up, per the research-persistence convention (substantial multi-source research goes in docs/research/, not only in memory).
  4. The config changes, which are the actual point:
    • subagentModelSettings.{lightModel,mediumModel,heavyModel} in ~/.factory/settings.json.
    • model: pins in each ported .factory/droids/*.md (bh-implementer, bh-reviewer, bh-visual-qa, bh-feedback-triage) — replacing today’s opus/sonnet guesses with measured picks.
    • missionModelSettings.workerModel for the mission catalogue.
    • A recommendation on sessionDefaultSettings.model (currently glm-5.3).
  5. A memory update to bullet-heaven-offload-experiments, which already holds the open OpenRouter/DeepSeek trial — this supersedes the ad-hoc version of that experiment with a real matrix, and its 📍 RESUME HERE should point here.

7. Known confounds — state them before the numbers arrive

Section titled “7. Known confounds — state them before the numbers arrive”
  • Typed GDScript / Godot 4.7 is likely thin in open-model training data. A cheap model may look bad here and be fine elsewhere. T5 exists precisely to separate “bad at code” from “bad at this code”.
  • Reasoning effort is not held constant (see §1). Defaults are used deliberately; record them.
  • Prompt caching favours long single-model sessions, which is what Factory Router optimises for. A one-shot droid exec per run does not benefit, so absolute costs here are an upper bound.
  • The fixtures are already-solved problems present in git history. A model with the repo in its training data is not plausible here (private repo), but the reference solution’s style leaks into the scorer’s expectations — which is why the judge is gate.sh and a mutation proof, not diff similarity.
  • This measures one-shot task completion, not collaboration. A model that asks a good clarifying question scores as a failure in a headless run and might be excellent interactively. Subagents cannot use AskUser at all, so for the seats being decided this is the right measurement — but do not generalise the table to session-default choice without that caveat attached.
  • One machine, one point in time. Re-run the harness rather than trusting a stale table; that is why it is committed code and not a one-off script.