Skilled AutoPlayer + Balance-Signal Reporting — Design

Skilled AutoPlayer + Balance-Signal Reporting — Design

Date: 2026-07-16 Status: Approved (Chris, 2026-07-16, live in chat — see “Approval note” below for how the normal written-spec review gate was handled) Origin: docs/ai-collaboration-capability-map.md — “live gameplay feel/balance verification” scored 3/10 for Claude (structurally can’t watch the game and get a feel for fun). Chris proposed a skilled AI player, run hundreds of times, as a numeric proxy: getting stuck early or only surviving with one build signals a real problem; crushing a boss with no effort right after a hard stretch signals anticlimax. He then added a second signal while we were setting this up: per-upgrade “did this feel like anything” scoring — an upgrade with near-zero measurable impact is as much a problem as one that instantly trivializes the run.

Problem

ai/auto_player.gd (AutoPlayer) is the only AI player in the project. It is deliberately dumb: pure vector-steering movement (flee/seek/wall-avoid, no positioning or threat priority), upgrade choice is “take a weapon if offered, else the first option” (no synergy evaluation), dimension choice is hardcoded to always pick the first offered (“no real strategy needed”, per its own comment). It exists for attract-mode’s visual demo and for soak coverage runs, which are usually --invuln=1 specifically so the AI can reach bosses/dimensions despite being unable to really fight. Its failures measure nothing about the game — a dumb AI dying at wave 2 doesn’t mean wave 2 is too hard.

There is also no per-event “did this upgrade matter” signal anywhere — upgrades_applied is recorded as a flat list per run with no outcome attached.

Non-goals

• The “boss-kill payoff” idea (WoW-style chest / bigger reward chance) is a separate design idea, not part of this spec. It’s gameplay content, not QA tooling, and belongs in docs/ideas/ when picked up — noted in docs/ai-collaboration-capability-map.md, not lost.
• Not a general rebalance. This spec builds and runs a measurement tool. Only narrowly mechanical findings get auto-applied as tiny constant tweaks (see Guardrails). Everything else becomes a DECISION (Chris): ... entry in the report, per this project’s existing bh-overnight-swarm convention — never auto-applied.
• Not ML/parameter-search self-tuning. “Improvement cycles” means the swarm iterating on hand-written heuristics through the normal build → test → measure loop, same as any other feature work here, not a training/optimization system.

Key facts that shape the design

• ai/auto_player.gd must not be touched — it’s load-bearing for attract-mode and coverage soaks, and its simplicity there is intentional, not a bug.
• tools/soak/soak_run.gd (SoakRun.execute()) and tools/soak/soak_runner.gd (CLI sweep + _find_anomalies cross-run pass) already exist and are the natural extension point — this is additive to working infrastructure, not a rewrite.
• Sim.spawn_phase (sim/sim.gd:21, enum PHASE_WAVES/PHASE_WIND_DOWN/PHASE_BOSS_PREP/ PHASE_BOSS/PHASE_REST) already exists, so segmenting “pre-boss” vs. “boss fight” is reading a field that’s already there per tick.
• Sim.stats_sink (sim/sim.gd:356) is a pure observer interface — on_damage(context, amount) fires from elemental_system.gd:173-174 on the player’s OUTGOING damage (damage dealt TO enemies). There is currently no equivalent hook for damage the player TAKES — _hurt_player (sim/sim.gd:2680) applies taken to pilot.hp but never reports it. A new stats_sink.on_player_damage(amount, phase) call is needed there, guarded by if stats_sink: exactly like the existing on_kill call at sim.gd:2408-2409 — purely additive, no read-back into sim state, so it cannot affect determinism.
• sim/upgrade_system.gd upgrade defs carry a kind field (weapon, evolve, transformative, etc. — see upgrade_system.gd:230,486). Useful for interpreting impact-scoring results later, but the scoring mechanism itself (see Component 2) doesn’t need to branch on it.
• This project’s bh-overnight-swarm skill explicitly treats balance-value changes as Chris-only decisions, escalated as "DECISION (Chris): ..." entries, never auto-fixed overnight. Tonight’s Guardrails section is a deliberate, narrow, live-approved exception to that rule for this one run — not a new standing precedent.
• Under the current v0.1 feature freeze (docs/RELEASE-0.1.md), this whole effort qualifies as QA tooling, which is explicitly in-scope even under freeze.

Architecture — 4 components

1. ai/skilled_auto_player.gd (new class, parallel to AutoPlayer)

Same public shape as AutoPlayer (compute_move_for, pick_upgrade, pick_dimension, wants_dash, wants_decoy) so SoakRun can take either as an injected strategy — no shared inheritance needed, duplication of the tiny interface is fine and keeps AutoPlayer untouched. Improvements over the dumb baseline:

• Movement: kite relative to the pilot’s actual effective weapon range (read from pilot.arsenal) instead of pure fixed-radius flee; prioritize the nearest dangerous threat (ranged/elite/boss) over nearest-any when choosing what to path away from.
• Upgrade choice: score offered choices instead of “first weapon, else first choice” — a simple heuristic scorer (e.g. prefer filling an empty weapon slot > evolving an active weapon with enough investment > a generic stat bump), tunable consts, not a full solver.
• Dash/decoy: same proximity-reactive triggers as AutoPlayer to start (no evidence yet that smarter timing here is a bottleneck — revisit only if soak data shows dash/decoy misuse).

2. SoakStats/SoakRun extensions — phase-aware + per-upgrade impact tracking

• Add sim.stats_sink.on_player_damage(amount, phase) call in _hurt_player (see Key Facts).
• SoakStats gains damage_taken_by_phase: Dictionary (phase enum int -> total taken), fed by the new hook — gives pre-boss vs. boss-fight damage-taken for free (the anticlimax signal: ratio of PHASE_WAVES+PHASE_WIND_DOWN+PHASE_BOSS_PREP damage-taken to PHASE_BOSS damage-taken; a run that took a beating pre-boss but sailed through the boss untouched has a high ratio).
• Upgrade impact scoring: when SoakRun._consume_levelups() applies a pick, snapshot two rolling-window metrics — damage-dealt rate and damage-taken rate — for a short window before the pick and again after (~3s/180 ticks each side, tunable). Record the pick’s id, kind, and the before/after deltas in a new SoakStats.upgrade_impacts: Array[Dictionary]. Scoring is banded, not monotonic: near-zero delta flags "dud", an extreme swing (tunable threshold, starting guess ~150%+ single-pick change) flags "overpowered", the middle is unflagged/healthy. Overlapping windows from rapid multi-level-up chains are an accepted approximation, not a bug — note it in the report if it looks noisy in practice.

3. Cross-run fun-signal analysis (extends soak_runner.gd)

Extends the existing _find_anomalies pass (which already does zero-damage-context and early-end detection) with, aggregated across the whole sweep:

• Death-tick/wave distribution — where mortal runs actually end.
• Build diversity/dominance — which upgrades_applied combinations correlate with longer survival vs. which never appear in a surviving run.
• Anticlimax signal — the pre-boss/boss damage-taken ratio described above, aggregated.
• Upgrade dud-rate / overpower-rate — per upgrade id, the fraction of picks across all runs flagged "dud" or "overpowered". Output: extend the existing reports/soak-<unix>.json shape with a fun_signals section: a human summary printed to stdout (matching the existing [soak] ANOMALY: ... convention) plus the raw aggregates in the JSON for anyone who wants to dig in later.

4. Guarded auto-tune (a swarm procedure, not new GDScript)

Phase 3 of the execution flow below. Not a code component — a documented procedure the overnight runbook spells out for whichever worker/iteration handles it.

Execution flow (overnight)

• Phase 1 — Build. SkilledAutoPlayer + the SoakStats/SoakRun/soak_runner.gd extensions, full bh-dev-chunk/bh-sdd-concurrent TDD ritual. This is the bulk of the night.
• Phase 2 — Measure. Once Phase 1 is solid and green, run soak sweeps at scale (multiple seeds x ships, mortal, SkilledAutoPlayer) and produce a findings report.
• Phase 3 — Guarded auto-tune. See Guardrails — narrow, mechanical, branch-only.
• Morning handoff. One docs/overnight-swarm-report-2026-07-16.md (or -night.md if it runs past midnight, matching the existing dated-file convention), decision queue up top, branch name + commit SHAs for any auto-tweaks, matching the existing runbook report format.

Guardrails (safety-critical, non-negotiable)

Auto-tweaks are allowed ONLY when ALL of the following hold:

1. Single named numeric constant, never a structural/mechanic change.
2. Magnitude-capped to roughly a 15% nudge from its current value.
3. Triggered only by unambiguous, mechanical, cross-run-consistent evidence — the same family as the existing zero-damage-context anomaly, extended by this spec to three trigger types:
   • a damage context deals ~zero damage across (nearly) every run → small buff candidate.
   • an enemy type spawns but is essentially never killed across (nearly) every run → small HP/armor nerf candidate.
   • an upgrade’s dud-rate or overpower-rate is extreme (near-100%) across (nearly) every run it appears in → small magnitude buff/nerf candidate.
4. One tweak per finding, then re-measure (re-run the relevant soak slice) before considering a second tweak to the SAME constant — no unsupervised chained adjustment.
5. Full ritual — bh-dev-chunk gates, and bh-determinism-repin if the tweak shifts the pinned baseline window (routine per that skill, not risky, as long as followed).
6. Branch only, never main. Each tweak is its own commit with the triggering evidence (numbers, not vibes) quoted in the commit message.
7. Everything that doesn’t cleanly meet 1-3 stays report-only — difficulty-curve/early-death clustering, general build dominance, and the anticlimax signal are almost never a single clean lever, so they become "DECISION (Chris): ..." entries with supporting numbers, exactly like the existing swarm convention for balance calls. When in doubt, escalate — don’t guess.

Testing

• Every new/changed .gd file is TDD’d via GUT, matching project convention — no exceptions.
• Determinism: SkilledAutoPlayer and the soak/report extensions are dev-only tooling, never reachable from main.gd or any production code path, so they cannot affect the pinned baseline by construction. The new stats_sink.on_player_damage hook is purely additive and observer-only (same pattern as the existing on_damage/on_kill hooks) — also baseline-safe. Only Phase 3 auto-tweaks (actual sim/content constant changes) can legitimately shift the baseline, and those go through bh-determinism-repin properly, not a blind number swap.

Approval note

Chris approved this design live in chat on 2026-07-16 while wrapping up for the night. The normal “user reviews the written spec before planning” gate is compressed here given his unavailability overnight — the spec, the resulting plan, and everything the swarm actually does are all there together in the morning handoff report for review as a whole, rather than gated step by step.