Learning autopilot (design)

Learning autopilot: persona genome, MAP-Elites archive, watchable overlay

Date: 2026-10-11 · Status: Implemented (branch learning-autopilot, 2026-10-11). §8 records where the shipped code differs from §2-§7; current architecture: docs/architecture/learning-autopilot.md.

Scope: sub-projects A (persona genome), B (learning loop), C (watch UI) of the wider “AI players” direction. D (personas as in-game AI players / co-op pilots) and E (class roles) are later specs that consume this archive.

1. Problem

The autopilot (SkilledAutoPlayer) is a fixed heuristic: it does not store builds, experiment, or learn. ExperimentalUpgradePlayer / UpgradeStrategies only swap between four hard-coded upgrade strategies in one-off soaks, read by a human. SkillProfile knobs were hand-calibrated.

Chris wants to watch the autopilot try different things and improve while he tests the game, and wants the result to become a set of distinct “player types” that later feed AI players the real player can meet, and the planned class roles.

2. Decisions

• Quality-diversity, not single-objective evolution. Optimising survival alone converges on one meta build, i.e. one player. A MAP-Elites archive keeps the best persona per niche.
• Niches: role × skill = 12 cells. Role ∈ {tank, support, damage, utility}, measured from the career archetype tags (docs/superpowers/specs/2026-10-04-career-archetype-tags-design.md) of the cards the run actually took; skill ∈ {weak, median, strong}, the existing SkillProfile limits, assigned by the target cell, never evolved.
• Fitness: mean survival time over samples; run gold breaks ties.
• The brain lives in the game; the Worker only stores. One GDScript implementation (pure RefCounted, ai/persona/) is used by soak and device alike; the telemetry Worker stores the archive and does an atomic replace-if-better.
• Shared archive, two learners: headless soak does bulk evolution; Dev-build devices fly experiments in autopilot runs and post results. Both write the same archive.
• Dev builds only. Store builds keep today’s fixed attract AI; no privacy page or ASC label change.
• Fixed learning conditions: learning runs use survival mode, the default area, and a fixed reference shop loadout (fresh save) on device and soak, so scores are comparable. A device run on Chris’s upgraded save would otherwise outscore soak on shop power, not genome.

3. Persona genome (A)

PersonaGenome — RefCounted, plain serialisable Dictionary, carries schema_version and AUTOPILOT_VERSION.

Gene group	Genes	Use
Build	weight per archetype tag (tank, support, damage, utility, drone); per-weapon preference; new-weapon-vs-owned-upgrade bias	Level-up pick: score each offered card = Σ(card archetype tags × weights) + weapon preference + bias; highest wins
Playstyle	~8 of today’s SkilledAutoPlayer consts: ENGAGE_SEEK_WEIGHT, ORBIT_WEIGHT, HZ_KITE_WEIGHT, DASH_WINDOW_MIN_MULT, DASH_WINDOW_MAX_MULT, HZ_OFFENSIVE_WARP_GAP_S, ELITE_DANGER_WEIGHT, BOSS_FLEE_WEIGHT	Steering/warp behaviour, each clamped to a declared range

• Not evolvable (stay const): survival-critical dodge logic (ESCAPE_RINGS, REACTION_S, LATE_REACT_S, STEP_LOOK_S, warp landing search). A mutant must not learn to suicide.
• PersonaAutoPlayer extends ProfiledAutoPlayer reads knobs from a genome plus the cell’s skill profile. The default genome reproduces today’s SkilledAutoPlayer exactly (pinned by test), so attract mode and the determinism baselines are unchanged when learning is off.
• The genome only affects the controller, never /sim; a run is reproducible from seed + genome.
• Role measurement at run end: archetype-tag share across cards taken; the largest non-drone share sets the role (ties → the target cell’s role). Superseded by §8: taken share minus offered share, with the tie rule as implemented.

4. Learning loop (B)

Archive: per epoch, 12 cells. Each elite: genome, persona id, generated name (e.g. “Tank-Strong #14”), parent id, generation, mean survival, sample count, finder (soak / device kind, see §8), build number.

Per run, one job:

Job	Default share	What
Mutate	70 %	Pick a target cell (prefer empty, under-sampled; “stale” deferred, §8); take its elite (or a neighbour’s if empty); Gaussian-nudge 2–3 genes within clamps; occasional crossover with another cell’s elite
Re-check candidate	20 %	Fly a promising candidate again to add samples
Re-check elite	10 %	Fly an elite unchanged; corrects lucky scores

• The run scores into the cell of the role it actually played, not the target; this is how empty niches fill.
• Promotion: a child is a candidate until it has ≥ 3 samples and its mean beats the cell elite’s mean. Soak children get 5 seeds at once; device children accrue over runs.
• Epochs: elites carry the build they were scored on. personas new-epoch starts a fresh archive seeded from the previous elites, which must be re-scored before holding a cell.

Storage (telemetry Worker + D1; superseded by §8: control Worker):

• persona_elites — one row per (epoch, cell).
• persona_samples — one row per run: genome hash, genome (for candidates), target cell, scored cell, survival, gold, seed, device, build, job.
• (Superseded by §8: control Worker, no RETENTION_TABLES.)
• Routes: GET /personas (current epoch archive + top candidates), POST /personas/result (records sample; promotion is a single atomic UPDATE … WHERE so concurrent posts can’t clobber). Writes require the existing dev/console auth (to verify in the plan; add a token if none fits). Schema/version mismatches are rejected.

Soak learner: scripts/personas-evolve.sh --hours N --jobs K — pull archive, breed a batch, evaluate children headless in parallel (5 seeds each), post results via the same route. Measure headless per-run cost before quoting throughput.

Device learner: Dev “Learning autopilot” Debug Settings row + console twin (on by default in Dev, absent in store). Every autopilot run (attract + dev autopilot) fetches the archive at run start, picks a job, flies it, posts at run end through EventQueue (offline-queued). Shipped (§8): every menu Survival / Redeploy is a learning run while the flag is on; posts go through PersonaClient’s own user:// queue to the control Worker, not EventQueue. No archive reachable → cached copy → default genome.

Console tunables: mutation σ, job split, promotion sample count, soak seeds per child.

5. Watch UI (C)

On-device overlay (Dev, autopilot run, learning on) — compact glass corner panel:

• Persona card: name, cell, job (MUTANT of Tank-Strong #14 / RE-CHECK candidate 2/3 / ELITE re-check), the mutated genes old → new.
• Live: role meter drifting as cards are taken; survival vs. cell elite mean as a ghost bar; per-pick note (“took Armor (tank 0.8)”).
• Run end: 3–4 s verdict banner (“New candidate: Utility-Median 71 s vs elite 64 s (1/3)”, “PROMOTED: new Tank-Strong elite”) with the 4×3 grid flashing the changed cell. Grid dropped (plan): the verdict names the cell; the full grid is on the console tab.
• Corner, opacity, verdict duration are console tunables.

Console /personas page: 4×3 grid (score, samples, generation; finder moved to the detail box, §8); click → genome + lineage; per-cell survival-over-time chart; latest sample feed (device + soak). Commands: new-epoch, pin-persona <id> (next device run flies it), learning on|off.

6. Failure handling

• Fetch fails → cached archive → default genome. Post fails → EventQueue retry.
• All genes clamped on load and after mutation; a corrupt row cannot produce absurd steering.
• Learning off = today’s behaviour, byte-identical.

7. Testing

• GUT: default genome == SkilledAutoPlayer inputs over a seeded run (determinism baselines untouched); mutation stays in clamps and is deterministic per RNG; role measurement on a known card sequence; promotion rule; job-split weighting; epoch seeding; result posting via an injectable push spy (EventQueue.push is a no-op headless).
• Worker: atomic promotion under two concurrent posts; auth required; tables in RETENTION_TABLES (superseded by §8: not in it; the single-statement upsert conditional is tested directly because the local D1 stand-in serialises statements).
• Overlay fit test at the iPad / iPhone / TV ShopLayoutFixture profiles.
• End-to-end: a 20-child soak smoke fills ≥ 3 cells; one Dev device run shows the overlay and its sample lands in D1.

8. Plan-time amendments (2026-10-11, from verified code facts)

These override §4/§5 where they conflict.

• Storage moves to the control Worker (control/), not the telemetry Worker. The telemetry Worker has no write auth (all POST ingest routes are open; only STATS_KEY for viewing), while the control Worker already gates every route on CONTROL_KEY (embedded only in dev/playtest builds, net/control_client.gd), and the console panel’s CSP is connect-src 'self', so the /personas page can only read data served by the control Worker anyway.
• No RETENTION_TABLES entry. That list is the telemetry Worker’s 12-month privacy purge. Persona rows carry no personal data: finder is a device kind, never a device name. Samples from epochs older than the previous one are pruned when a new epoch starts.
• Promotion runs in the Worker (one atomic upsert); the game does breeding and job selection. PROMOTE_MIN_SAMPLES = 3 is a Worker constant, not a game console tunable. Mutation σ, the job split and soak seeds-per-child remain game-side tunables.
• Personas always fly with humanize = true and throttle_planner = true, headless too. The kite and warp-gap genes only act when humanize is on, and attract mode throttles the planner; forcing both in soak makes soak and device scores comparable.
• Upgrade scoring is normalised. upgrade_effects() scores reach about 100, so a raw sum would drown the ±2 tag weights. The pick score is w_base × (score ÷ the best score in the offer) plus the tag, weapon and new-weapon terms. With the default genome the result is identical to SkilledAutoPlayer.pick_upgrade_for.
• A human take-over cancels the learning run: no result is posted, so mixed human/AI runs never score.

8.1 Amendments from implementation (2026-10-11, as shipped)

These override everything above where they conflict.

• Finder = OS kind. The game sends OS.get_name().to_lower() (or soak), so iPhone and iPad both report ios. The Worker’s allow-list is soak / tvos / ios / macos / android / linux / windows; anything else is a 400, and a platform with no kind (web, the BSDs) does not post.
• Role = preference, not content. The role played is, per role, the share among cards TAKEN minus the share among cards OFFERED; the largest difference wins. Raw taken share could almost never read support (every support card is also tank-tagged) and read damage for nearly every genome (15 of 16 weapons are damage-tagged). Mutation is directed: every mutant also steps w_<target role> up by PersonaTuning.TARGET_NUDGE (0.15) of its range, on top of the 2-3 random genes. Open call for Chris: difference vs ratio. Difference is kept; a ratio (taken ÷ offered) would favour rare roles harder. With the difference metric a genome like w_support +2, w_damage -2 can read as tank.
• Tie rule (as PersonaJobs.role_of implements it): among the roles tied for the top preference, the target cell’s role wins if it is one of them; otherwise the first tied role in tank, support, damage, utility order. Nothing taken at all reads as the target role. An empty offered tally falls back to raw taken shares.
• Which device runs learn: with the Dev learning flag on, every menu-picked Survival and every Redeploy is a learning run until a human takes over (Chris, 2026-10-11); flag off is normal play on the real save. A take-over cancels the job (nothing posts) but the rest of that run keeps the fresh save + Cobalt (plan F16), and a “LEARNING RUN · fresh save · ” tag says so on screen for the rest of the live run (final review D2, Chris: learning stays ON by default in Dev). An untouched learning death starts the next learning run after PersonaTuning.ATTRACT_CHAIN_S (12 s) on the death screen; any press stops it. Learning is skipped while a spawn / hard-tuning A/B, perf bench or showcase is on; Bay save/deploy, a P2 join, a dev build reset and any non-NET_SAFE dev command cancel the job; a live console tunable edit skips the post. Each sample carries the active remote-config patch_id ("" = shipped defaults).
• Console grid cell shows the best elite, or the best candidate when the cell has no elite; the finder is shown in the detail box only (review LOW-7 ruling).
• GET /personas gains cell_candidates: the best candidate per cell (highest mean, then more samples, own elite excluded), via a ROW_NUMBER() OVER (PARTITION BY cell …) window query; a previous-epoch seed fills only a cell with no current-epoch candidate. candidates (the global top 20, read by the game’s job picker) is unchanged.
• Console Personas tab is reachable before pairing: with a console key and no paired device the tab bar shows Personas only. Pin, clear pin and New epoch use an in-page two-click confirm with a 400 ms gap before the second click counts (so a double-click cannot arm and fire); the button is disabled while the request is in flight. The tab re-polls every 15 s while visible.
• Overlay verdict: shown only if the Worker answer matches the sample this run posted (the queue can flush older samples; a slow answer can land after the next run started). The panel has its own CanvasLayer, raised above the death fade and results screen when the verdict arrives, and collapses to a compact title + verdict so it never covers the results buttons.
• Telemetry: a learning run (also a taken-over one) reports the save and hull it actually flew and carries a persona_run: true marker (run_start, repro block, stored timeline_json); tools/economy_sim/human_targets.py excludes marked runs from the human calibration targets.
• Deferred: “stale cell first” mutate targeting (§4) is not implemented; targeting is empty cell first, else least-sampled. Epochs cover staleness for v1.
• A human level-up or Dimension pick also cancels the job (final review D1): the frozen panel’s taps never reach the take-over check, so the panel signal voids it.

9. Out of scope

Personas as in-game AI players / co-op pilots (D); class definitions (E); store builds; learning in story mode or non-default areas; evolving survival-critical dodge logic.