Balance evidence: Obsidian vs frigates, player crit, early-run deaths (2026-10-20)

Balance evidence: Obsidian vs frigates (#66), player crit (#69), early-run deaths

Research only. No game code or remote config changed. Orbit/Nova are out of scope (patch b335-001 just shipped for them).

Verdicts

Question	Verdict	Proposed change
#66 Obsidian a strict upgrade over the frigates?	No. The July claim is stale (armor went flat on 10-07, speeds were cut on 09-27). The whole edge is the +10 flat armor. +70% HP is cancelled by the speed and size penalty, and the 5 weapon slots never get used. Against Aurum, the best frigate, the edge is not significant.	Make the hull table tunable first. Then optionally Obsidian armor 10 -> 8. Keep everything else.
#69 Crit (x2.0, 15% per Focus) balanced?	Close to fair for one pick, falls behind Overclock as picks stack. It is not a dead pick.	Nothing for 1.1. For 1.2, Sim.PLAYER_CRIT_MULT 2.0 -> 2.5, plus a copy fix (the card’s sentence goes blank when the multiplier isn’t 2.0).
Early-run deaths: does Normal fix them?	In the sim, yes. Deaths under 30 s drop from 11-16/40 to 0/40 for the weak bot. Not shown on humans yet: there is one human Normal run since 1.1 builds. The Hard human runs on 331-333 repeat the old wall (9 of 14 died under 25 s, at level 1, to swarmers).	No knob change. Re-check once 1.1 store data exists (criteria below).

Method

• Harness: a scratch copy of tools/balance_probe/balance_probe.gd, at reports/scratch/hull_probe.gd (gitignored, not committed). It differs from the original in four ways:
  • It takes --ship.
  • It always passes a shop save, so the run starts in the same order as main.gd: MetaRunSetup, then Difficulty.apply_run_start.
  • It takes --ovr= stat overrides, applied in SoakRun’s on_start hook (used for the hull ablation).
  • It records crit/fire-rate picks, damage at 90 s, total damage and hard hits.
  • Everything else is the real SoakRun path.
• Runs: real headless SoakRun, SkilledAutoPlayer under ai/skill_profile.gd profiles, 6-minute cap (--ticks=21600). Fresh save unless stated. 40 seeds per arm (block A = --seed0=5000; block B = 5040 where shown). The DPS arms are 30 seeds with a 3-minute cap.
• Statistics: every arm uses the same seeds, so comparisons are paired. The tables give the paired per-seed median difference, a 95% bootstrap CI (2000 resamples), wins/losses and a two-sided sign-test p. A difference under about ±10 s is noise at this n.
• Sanity check: two arms built different ways but ending up with the same stats came out identical on every seed (Obsidian overridden to frigate body = Aurum overridden to armor 10; and Obsidian = Obsidian with 3 slots). So the override path is the real game path.
• Telemetry: read-only SELECTs on D1 (runs, events), builds >= 331.
• Caveat on every sim number: the bot is a proxy. It is weaker than humans early on (skill bh-economy-sim). It rarely takes weapon grants (median 2-3 weapons at death). On Normal, many arms hit the 360 s cap, which squeezes differences between them.

Harness bug found (affects the 10-16 Normal calibration slightly)

[Certain] SoakRun with no shop save (the path balance_probe and feel_probe use) runs Difficulty.apply_run_start before ShipBonuses.apply_base. apply_base then overwrites max_hp, so Difficulty.NORMAL_MAX_HP_MULT (1.25) is silently dropped. Checked directly: on Normal, start max_hp is 100 without a save and 125 with one. main.gd applies the multiplier after the hull, so players do get it.

Impact is small. Median bot, Normal, block A: 289.0 s with the multiplier against 282.6 s without (--tun=Difficulty.NORMAL_MAX_HP_MULT=1.0), and 0/40 deaths under 30 s either way. The calibration doc’s conclusions stand. Its Normal numbers are about 6 s pessimistic, and its “MAX_HP_MULT is a confound” caveat never applied in the probe. Fix (an issue for the tooling, not done here): in SoakRun.execute, move Difficulty.apply_run_start after the shop == null hull block.

Update 2026-10-21: fixed (#107, merged) and re-measured, see “Re-measure after #107” at the end.

1. Hulls (#66)

Static numbers (live code)

sim/ship_bonuses.gd TABLE is a const, so none of these are registered tunables today.

Hull	HP	Speed	Radius	Weapon / drone slots	Headline
Manta, Cobalt, Aurum, Prism, Amethyst	100	230	16	3 / 1	+20% dmg, +15% speed, +5 armor, +5% lifesteal, +15% fire rate
Obsidian (800 g)	170	185 (-20%)	24 (+50%, +125% area)	5 / 2	+10 armor

Armor has been flat since 10-07: max(hit - armor, 0.25 * hit). A swarmer contact hit is 12 x LETHALITY_MULT 1.6 x CONTACT_HIT_MULT 1.2 = 23.0. [Certain]

	Hit after armor	Hits to die (Hard)
Plain frigate	23.0	5
Aurum	18.0	6
Obsidian	13.0	14

Against the early killer, Obsidian has about 3x the effective HP of a plain frigate. Against a 54-point brute hit, armor only saves 19%. Armor is therefore strongest exactly where new-player deaths happen.

Seeded runs (median bot, 40 seeds)

Arm	Hard median	Dead <30 s	Normal median	Boss seen (N)
Manta	54.7 s	7	292.9	37
Cobalt (default)	57.5	7	289.0	37
Amethyst	62.9	4	303.3	38
Prism	71.0	5	314.1	39
Aurum	85.7	3	328.8	39
Obsidian	103.5	0	345.8	40

Paired differences:

Comparison	Mode	Difference (95% CI)	W/L	p
Obsidian vs Cobalt	Hard	+49 s [+34, +70]	33/6	1e-5
Obsidian vs Cobalt	Normal	+55 s [+31, +61]	33/4	1e-6
Obsidian vs Prism	Hard	+27 s [+9, +59]	29/10	0.003
Obsidian vs Aurum	Hard	+10 s [-8, +48]	22/16	0.42
Obsidian vs Aurum	Normal	+3 s [-2, +26]	20/15	0.50

With a modest shop save (vitality 2, bulwark 2, haste 2, swiftness 1) on Hard:

Hull	Median
Obsidian	128.3 s
Prism	124.0
Aurum	113.4
Cobalt	97.8
Manta	94.7
Amethyst	93.7

Obsidian vs Prism on this save: +6 s [-14, +11], p 0.43.

Ablation: which Obsidian stat does the work (Hard / Normal medians, paired vs Cobalt)

Obsidian variant	Hard	vs Cobalt	Normal	vs Cobalt
As shipped (170/10/185/24, 5 slots)	103.5	+49 (p 1e-5)	345.8	+55
Armor 0	56.1	-0.2 (20/20, p 1.0)	304.3	+4 (p 0.52)
HP 100 (keeps armor)	80.3	+8 (p 0.11)	320.7	+25
Frigate speed 230 + radius 16	143.6	+64 (39/1)	354.7	+44
3 weapon / 1 drone slots	103.5	identical on every seed	345.8	identical
Cobalt + 10 armor (control)	105.9	+48 (36/2)	352.3	+36

What this shows:

• [Certain, for this bot and a 6-minute window] The +10 armor is the whole advantage. Take it away and Obsidian is exactly Cobalt. Give Cobalt 10 armor and it matches Obsidian.
• [Certain] The +70% HP is paid for by the speed and size penalty. Obsidian at frigate speed and radius gains another 40-64 s, so the penalty is real. The July “offset by one level-up card” reasoning no longer holds: speeds were cut on 09-27, and the bigger radius is a real hit-rate cost (23 hard hits a minute against 16 for Cobalt).
• [Certain for the bot, Likely for humans] The 5 weapon slots never get used. The results are identical seed for seed. A 4th weapon is only offered from level 9 (count x WEAPON_LEVEL_GAP 3), and the bot dies at a median level of 8-16 with 2-3 weapons. Humans look the same: no human run since 331 has more than 3 weapons, including a 38-minute, level-76 run that ended with 2.
• Not a strict upgrade. Obsidian is slower, bigger and gets hit more. It is the best hull early because flat armor beats small contact hits. Aurum takes the same route at half the armor and lands within noise of it. The real imbalance is inside the frigates: the armor and lifesteal hulls (Aurum, Prism) beat the damage, speed and fire-rate hulls by 14-28 s on Hard. Manta’s +20% damage is worth -2 s (p 0.42), and Amethyst’s +15% fire rate +7 s (p 0.43).

Proposals (#66)

1. [Certain that it is a prerequisite] Register the hull stats as tunables (ShipBonuses base HP, speed, radius and magnitude per hull). Today they are a const and need a rebuild. That breaks the “console-tunable by default” rule, and these are exactly the numbers to tune by eye.
2. [Guessing] obsidian.magnitude (armor) 10 -> 8. The 800 g cruiser then stays the best early hull (the purchase is meant to be power across runs) without tripling effective HP against swarmers. Swarmer hit becomes 15.0, 12 hits to die. Run it as one arm of the probe before shipping. The armor arms bracket the effect, paired vs Cobalt on Hard: +5 armor (Aurum) gave +22 s, +10 armor (Cobalt control) gave +48 s.
3. [Likely] Leave Obsidian’s HP, speed, radius and slots alone. They already cancel out. The slots are a late-game promise this game rarely reaches, which is a weapon-pacing question, not a hull question.
4. [Guessing] For the frigate spread, look at Manta and Amethyst, not Obsidian. A damage or fire-rate bonus does little when early deaths come from contact damage. Candidates for a later pass: Manta +20% damage -> +30%, or give Manta and Amethyst a small survival rider. Unmeasured; for Chris to decide.

2. Player crit (#69)

What it covers (live code)

[Certain]

• Focus: +0.15 crit_chance. A crit does x Sim.PLAYER_CRIT_MULT (2.0), rolled from the isolated player_rng.
• Rolled by Rift Bolt, Blaster, Scatter, Pulse, Nova, Beam, Melee/Blade, Scythe, Turret, Railgun, Wildfire, Thunder Wake, Momentum Core and Scar Tissue.
• Not rolled by Orbit, Chalice or Chronowell, or by DoTs and zones.
• Rift Bolt is 52-62% of all bot damage, and it crits.
• The crit is applied before enemy flat armor, so it is worth more than 2x against armored targets (Warden 6, Sentinel 5, Orbiter 6, Brute 8).
• Its competitor, Overclock, is x1.2 compounding fire rate, capped at x3.0 (Sim.FIRE_RATE_CAP, about 6 picks). Crit is linear: x(1 + 0.15n), capped at x2.0 after 7 picks.

Expected damage multiplier:

Picks	Crit x2.0	Crit x2.5	Overclock
1	1.15	1.225	1.20
3	1.45	1.675	1.728
5	1.75	2.125	2.49
Cap	2.0	2.5	3.0

ui/upgrade_copy.gd returns no sentence for the Focus card when PLAYER_CRIT_MULT != 2.0. A live or remote change to the multiplier therefore blanks the card’s explanation ([Likely] it falls back to the bare stat line). That needs a copy change before the knob moves.

Seeded runs (Cobalt, median bot, picks granted at tick 0, 40 seeds)

Survival, paired against no pick:

Arm	Hard median	vs base (p)	Normal median	vs base (p)
No pick	57.5		289.0
Focus x1	58.2	+3.6 (0.75)	298.1	+3 (0.43)
Overclock x1	68.8	+6.5 (0.64)	284.1	+2 (0.64)
Reinforced Hull x1 (+25 HP)	67.8	+7.3 (1e-8)	291.9	+2 (1e-8)
Focus x3	77.1	+7.5 (0.11)	309.8	+20 (0.006)
Overclock x3	81.7	+23.4 (7e-4)	352.0	+32 (7e-4)
Focus x3, mult 2.5	64.0	+9.7 (0.27)	303.8	+4 (0.88)
Focus x3, mult 3.0	84.0	+21.4 (0.017)	301.5	+5 (0.52)

Head to head, Focus x3 against Overclock x3, paired: Hard -8.7 s [-18.5, -0.5], W11/L26, p 0.02. Normal -12 s, p 0.51.

Damage with an unkillable pilot (pilot HP 100,000, 3-minute cap, 30 seeds; isolates damage output from survival):

Arm	Total damage (median)	Damage by 90 s	Kills
No pick	7504	4087	560
Focus x1	8530	4274	615
Overclock x1	8266	4315	607
Focus x3	9326 (-4.8% vs Overclock x3)	4734	654
Overclock x3	9796	4966	676
Focus x3, mult 2.5	10190 (+4.0% vs Overclock x3)	5125	683
Focus x3, mult 3.0	10633 (+8.5%)	5238	708

What this shows:

• [Likely] At one pick, Focus is level with Overclock: total damage +3%, survival within noise. Armored targets make up for the 15% vs 20% on paper.
• [Likely] At three picks it falls behind: -5% damage and -9 s on Hard (p 0.02). The arithmetic says the gap keeps growing (x1.75 vs x2.49 at 5 picks). Crit is a fair early pick and a weaker build to stack.
• [Certain, a sim artifact to keep in mind] Survival isn’t monotone in damage. On Normal, higher crit multipliers dealt more damage but did not survive longer, and more orbiter deaths appeared. Likely cause (as in the 10-16 calibration doc): Sim._danger_mult scales enemy HP with player level, so faster levelling toughens the field. Judge crit on the unkillable-pilot damage table, not survival alone.
• Humans pick Focus about half as often as Overclock: 77 vs 144 picks across 84 human Survival runs, builds 201-335; offer counts aren’t recorded. So they read it as the weaker card. [Guessing] It may also be because the card’s wording undersells it.

Proposals (#69)

1. [Likely] No change for 1.1. Crit is not a dead pick and not overpowered.
2. [Guessing] For 1.2: Sim.PLAYER_CRIT_MULT 2.0 -> 2.5.
   • It brings the 3-pick damage to parity with Overclock (+4%), and its cap stays below Overclock’s (x2.5 vs x3.0), so stacking crit never beats stacking fire rate late.
   • Do the upgrade_copy.gd fix first (read the multiplier, don’t hard-code “double”).
   • Chris judges the crit feel on device. A crit number has to read as a big hit, and that is a look call.
3. [Guessing] Don’t use 3.0: +8.5% damage over Overclock at 3 picks, and a x3.0 cap equal to Overclock’s.

3. Early-run deaths and Normal

Sim (Cobalt, fresh save, shop applied in the real order, two 40-seed blocks)

Profile	Mode	Block	Median	p25-p75	Dead <30 s	Alive @60 s	Boss seen
weak	Hard	A	39.2	25-52	16/40	4	0
weak	Hard	B	41.9	28-49	11/40	3	0
weak	Normal	A	182.7	144-275	0/40	40	15
weak	Normal	B	179.3	142-254	0/40	40	14
median	Hard	A	57.5	47-78	7/40	16	0
median	Hard	B	54.4	46-74	4/40	15	1
median	Normal	A	289.0	270-322	0/40	40	37
median	Normal	B	289.8	272-307	0/40	40	38

• On Hard the swarmer is still the top killer (20-25 of 40 deaths), with swarmer shots next.
• On Normal the weak bot’s killers spread out (charger shot, shooter shot, swarmer), and no death comes before 60 s in 160 Normal runs.
• [Certain for the bot] Normal removes the early wall, consistently across both blocks and both profiles.

Telemetry (D1, read-only, builds >= 331)

• Normal: 1 human Survival run. Store channel 335 (TestFlight; 1.1 is not yet released): ATV, 353.8 s, level 26, died to a wyrm, on a fresh install after the tutorial. n = 1 tells us nothing.
• Hard: 14 human Survival runs on dev builds 331-333. All Cobalt, meta_spent 0.
  • 9 of 14 died in 14-22 s at level 1 to a swarmer, Blaster only.
  • 7 of those 9 came in one 10-minute ATV burst on 10-17 (who played isn’t recorded; per memory bullet-heaven-shared-atv-player-attribution, don’t guess).
  • The other runs: 46-95 s (3), 301 s, 444 s, and a 2314 s level-76 run.
  • This matches the 1.0.1 pattern (median 68 s, 5/14 under 30 s, all swarmer or charger), and those humans die faster than the weak bot on Hard.
• [Certain] New saves default to Normal (MetaState.survival_difficulty = "normal", and the same on load), so those Hard runs were picked deliberately.

Proposals (early run)

1. [Likely] Change no Normal knob for 1.1. The sim fix is large and robust. The only open question is human behaviour, and that needs store data, not more bot runs.
2. [Certain that the tool needs it] Fix the SoakRun ordering bug above before the next Normal calibration.
3. [Guessing] Pass criteria to check two weeks after 1.1 ships, query runs WHERE difficulty='normal' AND is_auto=0 AND channel='appstore': median >= 120 s, under 30 s <= 10%, first-boss reach >= 25%. If it fails, the first lever per the calibration doc is still density (Difficulty.NORMAL_BIOMASS_START_FRAC 0.15 -> 0.10). Swarmer contact on Normal is not reduced today (only shots and chargers are); a Difficulty-scoped swarmer multiplier would be a new knob.
4. Hard stays as it is (balance philosophy: Hard is the identity, no rubber-banding).

Reproduce

Each arm is one command:

godot --headless --path . -s res://reports/scratch/hull_probe.gd -- --ship=<hull> --difficulty=<hard|normal> \
  --profile=<weak|median> --seeds=40 --seed0=5000 --ticks=21600 [--ovr=armor=0,...] [--start=crit,crit,crit] \
  [--tun=Sim.PLAYER_CRIT_MULT=2.5] [--levels=vitality:2,...] [--pilot_hp=100000] --json=res://reports/scratch/out/<arm>.json

The harness is a gitignored scratch file. If this is going to be repeated, upstream --ship, --ovr and the always-pass-a-save fix into tools/balance_probe/balance_probe.gd.

Re-measure after #107 (2026-10-21)

Measurement only; no balance value changed. tools/balance_probe (the no-shop-save path that #107 fixed), Normal, Cobalt, fresh save, 40 seeds per block, 6-minute cap, current main (130356d3 + the #107 fix). Blocks A/B = --seed0=5000/5040.

Profile	Block	Median	p25-p75	Dead <30 s	Alive @60/120/180	Boss seen
median	A	297.5	276.3-328.3	0/40	40/40/40	40/40
median	B	296.4	274.8-321.0	0/40	40/40/39	39/40
weak	A	182.5	146.2-264.0	0/40	40/35/20	15/40
weak	B	177.8	151.4-256.7	0/40	40/34/19	15/40

Control (median, block A, --tun=Difficulty.NORMAL_MAX_HP_MULT=1.0, same code): 290.9 s, boss seen 37/40.

• [Certain] The 1.25 multiplier is worth about +6.6 s (290.9 -> 297.5) on the median bot, matching the “about 6 s” estimated above.
• [Certain] The Normal median is now about 297 s, not 282 s, and the boss is seen in 79/80 median runs (33/40 per block in the 10-16 calibration). Of the +15 s against 10-16, about 7 s is the restored multiplier; the remaining ~8 s (290.9 vs 282.6 with the multiplier off in both) is other changes to main since the 282.6 s run, which used 10-20 code (reviewer, 2026-10-21), so this is same-day drift (not bisected here).
• [Likely] The weak bot is unchanged within noise (182.5 / 177.8 against 182.7 / 179.3 in the shop-save table above): same path, same multiplier.
• The calibration’s acceptance bar (median >= 150 s, boss in > 50% of runs) still passes with more headroom. Hard is untouched: apply_run_start is a no-op there, so the Hard figures (50-59 s) stand without a re-run.

DECISION (Chris): none required. Normal is slightly easier than the calibration doc reports (boss seen in nearly every median run). If that is more generous than intended for 1.2, the first knob to try is Difficulty.NORMAL_MAX_HP_MULT 1.25 -> 1.15 (about -3 s), not a re-tune of the opening.