Soak Death-Cause + Level Tracking Implementation Plan

Soak Death-Cause + Level Tracking Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Surface what killed the pilot and aggregate level-reached across a soak sweep, so Chris’s “chargers beat Ace’s late-game maneuvering” hypothesis can be tested with real data before any AI behavior change is designed.

Architecture: Sim.death_cause already exists and is already correct (latched at the first fatal blow) — this plan only plumbs it through SoakRun’s report and adds three new aggregates to soak_runner.gd’s _analyze_fun_signals(). No /sim changes.

Tech Stack: Godot 4.6.3, typed GDScript, GUT 9.6.0.

Global Constraints

• No changes to /sim — Sim.death_cause (sim/sim.gd:344) already does exactly what’s needed.
• No AI behavior change in this plan — Chris explicitly chose “instrument first, then design.”
• soak_runner.gd’s aggregation functions are NOT GUT-tested by established project convention (its own header comment: the CLI runner “isn’t GUT-tested,” only SoakRun is) — the 3 new aggregates in Task 2 follow that same convention; do not add a new test file for them.
• death_cause_breakdown counts only runs that ended in game_over == true; a run that hit max_ticks without dying is excluded entirely (not bucketed under "").

---

Task 1: SoakRun.execute() exports death_cause

Files:

• Modify: tools/soak/soak_run.gd:78-94 (the report dict returned by execute())
• Test: tests/test_soak_run.gd

Interfaces:

• Produces: report dict gains a "death_cause": String key — empty string if the run never reached game_over, otherwise the source string Sim._hurt_player latched at the fatal blow (an enemy type name from EnemyPool.type_name(), or "ranged").

• Step 1: Write the failing tests

Read the current content of tests/test_soak_run.gd first — it already has a _cfg() helper and an established pattern for constructing runs (see the existing test_run_completes_and_reports and test_invuln_run_keeps_pilot_hp_full tests for the exact helper signature and SimContentFixture usage). Append these two tests, using that same _cfg() helper:

func test_death_cause_is_empty_when_run_never_dies() -> void:
  var report := SoakRun.new(_cfg({"max_ticks": 10})).execute()
  assert_eq(report.death_cause, "", "a short run that never reaches game_over has no death cause")

func test_death_cause_is_populated_on_a_mortal_death() -> void:
  # seed 1234 with the default (dumb) AutoPlayer and no invuln reliably dies well before
  # 3600 ticks (60s) at this content/seed -- long enough to guarantee game_over without
  # an excessively long test run. If this seed/tick count doesn't produce a death, increase
  # max_ticks rather than switching seeds (seed 1234 is this project's canonical baseline seed).
  var report := SoakRun.new(_cfg({"max_ticks": 3600})).execute()
  assert_true(report.game_over, "sanity: this run must actually end in death for the assertion below to mean anything")
  assert_true(String(report.death_cause) != "", "a run that ended in game_over must report what killed the pilot")

• Step 2: Run tests to verify they fail

Run: godot --headless --path . -s res://addons/gut/gut_cmdln.gd -gselect=test_soak_run -gexit Expected: FAIL on both new tests — report.death_cause doesn’t exist yet (GDScript Dictionary access on a missing key returns null, so "" comparison and String(...) conversion will both fail the assertions, not throw — confirm the failure message names the missing/null value).

• Step 3: Add the field

In tools/soak/soak_run.gd, find the report dictionary returned by execute() (currently ends around line 93-94 with "upgrade_impacts": stats.upgrade_impacts,). Add one line:

    "level_reached": sim.player.level,
    "death_cause": sim.death_cause,

(Insert "death_cause": sim.death_cause, immediately after the existing "level_reached": sim.player.level, line — read the current file first to get the exact surrounding lines right, since this is a single-line insertion into an existing dict literal, not a rewrite.)

• Step 4: Run tests to verify they pass

Run: godot --headless --path . -s res://addons/gut/gut_cmdln.gd -gselect=test_soak_run -gexit Expected: PASS — all tests in the file green, including the 2 new ones.

• Step 5: Commit

git add tools/soak/soak_run.gd tests/test_soak_run.gd
git commit -m "feat(soak): SoakRun reports death_cause (already-computed Sim.death_cause)"

---

Task 2: soak_runner.gd aggregates level + death-cause across a sweep

Files:

• Modify: tools/soak/soak_runner.gd:71-130 (_analyze_fun_signals())

Interfaces:

• Consumes: rep.death_cause: String, rep.level_reached: int, rep.game_over: bool (Task 1’s new field, plus two fields that already existed).

• Produces: the fun_signals dict gains 3 new keys: level_distribution: Array[int], death_cause_breakdown: Dictionary[String, int], mean_level_at_death_by_cause: Dictionary[String, float].

• Step 1: Read the current _analyze_fun_signals() implementation

Read tools/soak/soak_runner.gd in full (it’s ~139 lines) to confirm the exact current shape of _analyze_fun_signals(reports: Array) -> Dictionary before editing — in particular the existing skilled filter loop (if String(rep.get("player", "dumb")) == "skilled": skilled.append(rep)) and the existing death_ticks accumulation loop, since the new code follows the identical for rep in skilled: iteration pattern already established there.

• Step 2: Add the three new aggregates

Inside the existing for rep in skilled: loop in _analyze_fun_signals() (the one that already builds death_ticks, build_survival, pre_boss_taken/boss_taken, upgrade_flags), add:

  var levels: Array = []
  var death_cause_counts: Dictionary = {}     # cause -> count
  var death_cause_level_sums: Dictionary = {} # cause -> {"sum": float, "n": int}

  for rep in skilled:
    # ... existing loop body stays exactly as-is, these lines are ADDED inside it ...
    levels.append(int(rep.level_reached))
    if bool(rep.game_over):
      var cause := String(rep.get("death_cause", ""))
      if cause != "":
        death_cause_counts[cause] = int(death_cause_counts.get(cause, 0)) + 1
        if not death_cause_level_sums.has(cause):
          death_cause_level_sums[cause] = {"sum": 0.0, "n": 0}
        death_cause_level_sums[cause].sum += float(rep.level_reached)
        death_cause_level_sums[cause].n += 1

(This is illustrative of the values to add — levels, death_cause_counts, and death_cause_level_sums must be declared before the existing for rep in skilled: loop starts, alongside the file’s existing death_ticks/build_survival/etc. declarations, and their .append/increment lines go INSIDE that same existing loop body, not a second separate loop — follow the file’s own established pattern of one pass over skilled building multiple accumulators at once.)

After the loop, before the function’s final return statement, compute the mean-level-by-cause dict from death_cause_level_sums:

  var mean_level_at_death_by_cause: Dictionary = {}
  for cause in death_cause_level_sums:
    var entry: Dictionary = death_cause_level_sums[cause]
    mean_level_at_death_by_cause[cause] = float(entry.sum) / float(entry.n)

• Step 3: Add the 3 keys to the returned dict

Find the function’s existing return { ... } dict literal (currently returns skilled_runs/death_tick_distribution/build_diversity/builds_never_seen_surviving_full_run/ anticlimax_ratio_pre_boss_to_boss_damage_taken/upgrade_dud_rate_ge_90pct/ upgrade_overpowered_rate_ge_90pct). Add three more keys to that same literal:

    "level_distribution": levels,
    "death_cause_breakdown": death_cause_counts,
    "mean_level_at_death_by_cause": mean_level_at_death_by_cause,

• Step 4: Verify with a real sweep, not a unit test

Per this plan’s Global Constraints, _analyze_fun_signals() is not GUT-tested by established convention. Verify instead with a real, short sweep:

godot --headless --path . -s res://addons/gut/gut_cmdln.gd --import
bash scripts/soak.sh --runs=8 --ticks=36000 --player=skilled --seed0=9500

Read the printed [soak] fun signals: {...} JSON line (or the written reports/soak-*.json file’s fun_signals block). Confirm:

• level_distribution is present with exactly 8 integer entries, one per run.
• death_cause_breakdown is present, its values sum to the number of runs that had game_over: true in the per-run [soak] ... log lines printed above it (count them by eye).
• mean_level_at_death_by_cause has the same keys as death_cause_breakdown, and each value is a plausible level number (roughly in the same range as level_distribution’s values, not wildly off — e.g. not 0 or 1000).
• No SCRIPT ERROR lines anywhere in the run’s output.

Paste the actual fun_signals JSON output into your task report so the reviewer can check the real numbers, not just that the code compiles.

• Step 5: Run the full suite to confirm no regressions elsewhere

Run: godot --headless --path . -s res://addons/gut/gut_cmdln.gd -gdir=res://tests -ginclude_subdirs -gexit Expected: same pass count as Task 1 left it (this task adds no new GUT tests, only changes an untested dev-tool script), 0 failing, same 4 pre-existing risky/pending.

• Step 6: Commit

git add tools/soak/soak_runner.gd
git commit -m "feat(soak): aggregate level + death-cause across a sweep in fun_signals"

---

Task 3: Verification + push

Files: none (verification only).

• Step 1: Full suite

Run: godot --headless --path . -s res://addons/gut/gut_cmdln.gd -gdir=res://tests -ginclude_subdirs -gexit Expected: 0 failing, same 4 pre-existing risky/pending, script count unchanged from before this plan (no new test files were added beyond Task 1’s 2 tests in the existing test_soak_run.gd).

• Step 2: Count guard

Run: bash scripts/check-test-count.sh Expected: test-count guard OK.

• Step 3: Determinism baseline unaffected

Run: godot --headless --path . -s res://addons/gut/gut_cmdln.gd -gselect=test_determinism_survival -gexit Expected: PASS, unchanged (this plan touches only tools/soak/, never /sim).

• Step 4: A real diagnostic sweep answering Chris’s actual question

This is the payoff step — run a larger sweep and report the real death_cause_breakdown / mean_level_at_death_by_cause numbers to Chris, specifically checking whether "charger" (TYPE_ELITE’s display name) is both a leading cause of death AND has a meaningfully higher mean level-at-death than other causes (the concrete test of his stated hypothesis):

bash scripts/soak.sh --runs=30 --ticks=36000 --ships=cobalt,obsidian --coop=solo --player=skilled --seed0=10000

Report the death_cause_breakdown and mean_level_at_death_by_cause blocks directly — this data is what determines whether the next step is “design charger-specific dodge/orbit AI” or “look elsewhere, chargers aren’t actually the bottleneck.”

• Step 5: Push

git push origin main