Agent evaluation tooling: review and ideas (2026-10-22)

Chris asked: what do we think of the tooling built to help agents develop Dark Cosmos, and what else would help an AI agent evaluate the game? Chris’s verdict on the list below: “they are good”. None of it is built yet.

Review of what exists

Strong on headless numeric proxies: tools/balance_probe, tools/feel_probe, tools/difficulty_probe, tools/economy_sim, tools/soak, tools/story_auto, tools/determinism_trace, tools/replay_compare, the offline fight-mix checks (tools/sound_map/fight_mix.py, fight_checks.py), tools/device_sim scorecard, and the live console MCP (tools/console_mcp). Agents can get real numbers back without Chris.

Two structural weaknesses:

Every probe measures the autopilot, not a human. Bot vs human gap is large (docs/research/2026-10-13-autopilot-vs-human-gap.md, docs/research/2026-10-14-toby-replay-vs-autopilot.md): humans average 392 s vs bot 104 s, keep ~1.3 enemies near vs ~3.5, turn ~100 deg/s vs 420-1800. A balance or feel verdict is only as valid as the bot’s resemblance to a person, and nothing reports that resemblance.
Many tools, no single verdict per build. Matches the 2026-10-15 audit headline (docs/research/2026-10-15-aaa-tooling-gap-audit.md): closure, not tools, is the weak point.

Ideas, in priority order

Bot fidelity score. Fly the autopilot on the same seeds as the human replay corpus and score its distance from humans (survival, crowd distance, turn rate, warp rate, shot density). Print it alongside every probe result so a balance call carries a confidence level. Depends on replay auto-upload (docs/superpowers/specs/2026-10-14-replay-auto-upload-design.md).
Per-build report card. Run the headless suite on each build, diff against the previous build, post one summary to the GitHub tracker, and flag anything outside its band. Build on device_sim’s scorecard.
Gameplay-frame visual regression. Determinism means seed + tick gives an exact frame. Capture ~20 fixed moments (boss intro, peak bullet density, supernova, death) and perceptually diff (SSIM or FLIP) against blessed baselines in scripts/gate.sh. Today only menus are baselined (tools/ui_sweep/baseline/).
Readability metric. At fixed ticks, render ID masks for ship, telegraphs and hostile bullets. Measure ship occlusion by FX and telegraph contrast against the background as bullet density rises. Run the same frames through tools/cvd_sim. Turns “can you see what’s killing you” into a number.
Death-fairness report. For each death: was the killing threat telegraphed, on screen longer than human reaction time, from a visible direction? Report unfair deaths per run. Cheap, since the sim knows all of it.
Build viability matrix. Damage share, pick rate and survival per weapon/upgrade combination, to catch dead content and dominant builds automatically (example: Orbit dealt 79 damage across Toby’s 883 s run 881).
Pacing curve. Intensity over time (enemies on screen, damage taken, XP rate) against a target curve; flag dull stretches and spikes.
First-timer playthrough test. A vision-model agent plays the tutorial from screenshots only, no game-state access, and logs where it gets stuck: a rough proxy for onboarding clarity. Separately, a vision model scores gameplay filmstrips against a short rubric, calibrated first against Chris’s 0-5 ratings in tools/vfx_library.
Real audio output checks. Record the Mac build’s actual output via a loopback device (e.g. BlackHole) and run fight_checks.py on it. Current checks use recorded timelines, which go stale after any routing change.
Stress tests. Random input monkey, background/resume, controller disconnect, co-op relay network jitter, and 30-60 min soaks for memory growth and perf drift (humans already reach 1,581 s runs).
Input latency counters. In-engine timestamps for input event, sim tick and presented frame. True end-to-end latency still needs Chris filming the screen at 240 fps.

Recommended start: 1 and 2. Without them, the rest produces more numbers someone has to interpret by hand.