2026-10-07 — Performance + controller-responsiveness workflow audit

Performance + controller-responsiveness workflow audit (2026-10-07)

Trigger: Chris, after BUILD 274 on the Apple TV: “there are still some slow [moments], also there seem to be times when the controller stops responding, not sure if it’s Bluetooth”.

Method: a 70-agent workflow. Six lens-specific finders (live D1 telemetry, shader prewarm / PSO coverage, sim CPU with tools/sim_bench A/B, render / FX frame cost, controller input path incl. the tvOS fork, backlog triage) produced 51 findings; the top 16 by severity went through 3 adversarial skeptics each (reproduce, root-cause, fix-safety), 10 survived, 7 were implemented in isolated worktrees and reviewed by bh-reviewer. A second round ran the measured sim-CPU items the severity cap dropped. Every number below was re-derived by a skeptic from production D1 or a headless measurement unless marked [not verified].

1. What the telemetry actually says (ATV = tvOS/AppleTV11,1, 60 Hz cap)

Builds	Avg fps	Min fps	Hitches / session	Note
269–271	59.8–59.9	52–57	1–6	Human-ish play, 0 sub-50 windows in every enemy bucket
272	53.4	17	387	ONE 79-min autopilot session; auto inflates sim_ms via the planner
273	58.0	19	79	Includes a weapon-showcase auto session cycling every evolved look
274	—	—	—	No ATV/iPhone samples yet (organic nebula + parallax unmeasured)

• The “272 regression” is not a controlled drop: single autopilot session, live nebula tunables changed mid-session, and sim_ms is ONE Performance.TIME_PHYSICS_PROCESS sample per 5 s window (the last physics frame, whole _physics_process), not a per-second max as the docs said. Round 2 keeps sim_ms as-is and adds tick_ms (windowed mean) + tick_max_ms + ticks from timed sim.tick_single() calls so the next comparison is attributable.
• Webs (Scar Tissue v2) do cost fps at equal enemy counts: auto session 9874f3 webs 0 → 59.6, 1–49 → 56.7, ≥50 (avg 149) → 48.2 fps; human 142ad2 webs ≥50 (avg 66) → 53.8. Dropped by skeptics only because the causal isolation (webs vs the rest of the look set) needs tools/perf_probe/look_bench.gd on device. Still open.
• Frozen-clock screens ran at ~42 fps on 272/273 (59.9 elsewhere) and one was labelled play. Fixed in telemetry first (frozen flag + reason on samples/hitches) so the next session names the screen.
• Game-over screen is back to 59.9 fps (264, 273). Positive.

2. Stalls (the thing that feels like a dead controller)

1. Forced-tutorial boot never ran PrewarmPass. Every launch with the tutorial unfinished (all new players, and every fresh Dev install) skipped the start menu and linked every pipeline cold on the first run frame: 3288 ms (268), 2084 ms ATV / 1977 ms iPhone (271), 12–13 pipelines, proc_ms ≈ frame_ms (script side; no joypad events are pumped during it). The new:node:BladeSlashLook... tags were red herrings: those shaders WERE in the pass; the pass just didn’t run on that route. Fixed: the boot route now runs the pass behind an opaque cover (NeonBackdrop + GlassPanel + Label so the UI pipelines also link), then starts the tutorial. Verify on device: fresh install, tutorial unfinished, expect zero hitches WHERE pipelines > 0 AND run_time < 1.
2. 18 shaders were outside the pass even on the menu path; two had recorded ~1 s first-use stalls (aim_lance, arena_aurora). Fixed: six more families (enemy archetypes on enemy_smear, aim lance, dissolve, grid, full-screen passes, glass) + tests/test_prewarm_coverage.gd walks the built pass (13 → 0 uncovered, 7 deliberately excluded with reasons). Advisory: ScreenFeedback’s 65k set_pixel vignette and the full ArchetypeRenderer now build as the menu appears; possible menu stutter on ATV, unmeasured.
3. Fresh-install stalls are unchanged by the BUILD 262 metallib bake: ~8 stalls / 7.9 s in the first 30–90 s on the ATV, near-identical on 260 / 262 / 264. The bake is not reaching the device path. [not verified why] — next step is skill bh-fresh-install-stall-check after this merge.
4. Mid-run 0.9–1.3 s stalls are NOT script time: proc_ms 6–90 ms against frame_ms ~1000 ms, tight 940–1050 ms cluster at ~200 enemies on 262/264 (looks like a Metal nextDrawable timeout, not a shader compile). Rate: ~1 per hour of ATV play since 268. Needs drawable-acquire / GPU time in the fork’s hitch payload to split GPU-bound from present-blocked. Open, needs-device.
5. GRAVITON / BOSS DOWN ~1–1.3 s stalls (4 rows since 255): skeptics refuted the “unwarmed screen pass” attribution (BOSS DOWN drives the already-warmed ring; GRAVITON link unproven). Probe before adding families.

3. Controller: what we found, ranked (nothing proves Bluetooth yet)

1. App background/foreground flips mid-fight freeze play 2–25 s then drop the player back in. D1 background/foreground pairs in play/boss with run_time unchanged: 262 2.1 s, 265 2.3 s, 266 2.3 s, 267 2.5 s, 273 24 s, 272 1308 s then crash; 16 of 27 ATV play lifecycle events are backgrounds. A resign-active (controller reconnect UI, Home/Siri press, system dialog) does exactly this. Fixed behaviour: a human in live play now gets the normal pause menu on APPLICATION_PAUSED (and FOCUS_OUT on iOS/tvOS only). Cause of the resign-active is still unknown: nothing records what the pad was doing.
2. No controller connect/disconnect handling existed anywhere in-run (only the start menu’s glyph refresh). New PadLiveness sends pad_conn (connect/disconnect, rough pad type, never the Bluetooth name), input_gap (human in a live fight, enemies > 0, pad silent after being active) and input_resume. Reading the next report: pad_conn disconnect at the moment = Bluetooth / Nimbus sleep; input_gap with a big max_frame_ms and no disconnect = stall; neither = the OS overlay case above.
3. Survival runs start in attract-mode autopilot and ignore RB/LB until the stick or aim moves (main.gd _new_run(human=false), hand-off only on move_dir > 0.01 or aim_dir > 0.35; while _auto the whole InputState incl. dash/decoy is replaced by the AI’s). Buttons appear dead at run start. Round 2.
4. Dev-only R2 voice-feedback overlay freezes the run and, with the tvOS keyboard up, ignores pad events; cancels emitted no event so accidental opens were uncountable. Round 2 adds feedback_open/close + banner.
5. Co-op pilot↔pad mapping is connection order, so a reconnect swaps or orphans pilots. Local co-op only. Open, medium.
6. Nothing flips INTO autopilot mid-run; no deadzone/latch bug in InputRouter.poll; warp dash_dir reuse can’t disagree with move for the human controller.
7. Fork notes: SDL_HINT_TV_REMOTE_AS_JOYSTICK=0 (Siri Remote is keyboard input, our “remote” name filter is a no-op on tvOS); joypad_sdl.cpp drops a pad silently if SDL_OpenGamepad fails; the fork pumps SDL_PollEvent once per iterate, so any main-thread stall is an input stall. GCSupportsControllerUserInteraction=true in the tvOS plist lets the system route controller UI to the app — candidate trigger for the overlay, [not verified].

Repro plan for Chris: two 30-min Nimbus sessions on the ATV on the next Dev build, noting wall-clock times of each “stopped responding” moment, then join against events types pad_conn / input_gap / background and hitches in D1 (skill bh-telemetry).

4. Sim CPU (headless, seed 1234, ticks 1800–3599, 166 enemies, blaster only, M5)

Per-phase: _move_enemies 390 µs/tick (37%), apply_status_and_decay 198 (19%), SpatialHash rebuilds 2×/tick ~36 µs each (7%, up to 6× with nova/orbit/scar/thunder), chip_fire 47, resolve_collisions 42. The gfx-loop-1007 merge did not regress sim CPU (A/B within ±10% noise, identical checksums). Hoisting apply_status_and_decay measured 198 → ~60 µs/tick with an unchanged 3600-tick checksum. All three are round-2 items; the bench is blaster-only so weapon costs are still unmeasured (round 2 adds a 6-weapon --loadout flag).

5. Render CPU

EnemyAuraLook.sync_from_sim rebuilt up to 400 × 3 rings × 16 segs × 2 appends per frame in GDScript. Rewritten look-neutrally (cached unit ring transformed in one native call, colour runs cached, skipped on frames without a sim tick): 1 stack n=400 1072–1171 → 648–694 µs; 3 stacks n=400 2629–2691 → 1136–1344 µs. Geometry pinned to the old formula by test. WeaponTurret.render_states rescans all enemies per turret per frame (round 2 reuses the fire target). Adaptive QualityManager is OFF by default so none of the new looks are shed under load; skeptics demoted this to a design call: fix the looks’ own cost first, then re-enable deliberately with 60 Hz DOWN_FPS.

6. Shipped from this audit (merge train on main)

• EnemyAuraLook rewrite; frozen-state telemetry (needs 5 D1 ALTERs then Worker deploy, Chris’s OK); PadLiveness controller events; PrewarmPass +6 families + coverage test; forced-tutorial boot prewarm behind a cover; auto-pause on background; PadLiveness.reset_run clock fix.
• Dropped as duplicates (not merged): joy_telemetry.gd, pad_telemetry.gd, joypad_diagnostics.gd branches.
• Round 2 (merged 2026-10-07, all sim checksums unchanged): apply_status_and_decay hoist + zero-timer guards; _move_enemies dispatch hoist + inline plain-walker step (734 → 306 µs/tick); redundant SpatialHash rebuilds dropped with a weapon-phase hash verifier test; tools/sim_bench --loadout + --phases per-weapon timers (all six weapons ≈ 9% of a tick); samples.tick_ms / tick_max_ms / ticks (the sim’s own cost; sim_ms keeps its whole-physics-step meaning, documented in net/telemetry.gd); RB/LB press hands the run over from attract autopilot; turret barrels aim along the real fire target (no per-frame rescan); dev R2 feedback overlay emits feedback_open / feedback_close; dev candidate looks prewarmed (family 25).
• Still open from this audit: webs render cost (implementer in flight), mid-run ~1 s non-script stalls (needs drawable/GPU time in the fork’s hitch payload), fresh-install stalls vs the metallib bake, co-op pad mapping by connection order, QualityManager re-enable (design call).

7. Must land before the next STORE upload

The controller events are a new KIND of data. marketing/appstore/compliance.md now lists “Other Diagnostic Data” for all platforms; the public privacy page (~/Claude/bullet-heaven-site/privacy/) and the ASC label for tvOS/macOS are NOT yet updated. Tracked in docs/RELEASE-1.0.1.md.