Wave Summary
| Field | Value |
|---|---|
| purpose | turn the asynchronous gate into an operationally safe fleet; make every refinery, repository, and agent lane explicit; remove hidden fallback and liveness assumptions |
| development window | July 20, 13:55 to July 26, 13:37 PDT—5 days 23 hours 43 minutes |
| baseline | 3ca2edff |
| final commit | 31938769 |
| Git commits landed | 94 total: 83 non-merge commits and 11 merge commits |
| contributors | 4 Git author identities |
| change surface | 96 files, 13,670 insertions, 735 deletions |
| exit evidence | 93 focused fleet tests passed; 12+ complete eight-shard fleets launched and reaped cleanly; no ThrottlingException in roughly 870 subsequent proxy-log lines |
| exit state | Yesod’s gate defaults to MicroVM with explicit local rollback; refinery recovery is lease- and checkpoint-based; per-tool routing and foreign-repository credentials are explicit; the project and agent factory can provision more than one kind of lane |
Wave 2 built the asynchronous gate protocol and ended with a failed image cutover. Wave 3 dealt with the consequences of making that protocol real. A remote gate cannot be treated as one more transport implementation: it has leases, fleets, credentials, flavors, repository identity, and failure modes that must agree across the worker, proxy, guest, supervisor, and dashboard.
The central change was a shift from fallback-oriented operation to explicit ownership. Yesod no longer quietly runs a local gate when the remote path is unavailable. A profile says which repository it gates, which proxy and image it uses, and which credentials it may receive. If that path cannot run, the item remains infrastructure-failed and queued for recovery. The operator may still select local gating, but that is now a deliberate rollback decision rather than an accidental safety net.
Readiness Became a Control-Plane Fact
The wave opened by fixing a quieter source of starvation: readiness was being
reconstructed from several imperfect views. The collector now mirrors Dolt’s
ready_issues into PostgreSQL; the runner claim path reads that mirror and
performs a single candidate verification; and a slower reconcile pass compares
the mirror with Dolt and alerts on drift (0547a214 through 95663048).
That work exposed the default 100-issue limit in bd ready. With more than
100 ready Beads, a valid armed note could exist in PostgreSQL and still be
invisible to every runner. The runner and collector now request the complete
set with --limit 0, and a regression test covers a note beyond position 100
(88c7df18, 94e24499, d539227a). Readiness is no longer “whatever the
last subprocess happened to print.”
The same boundary became louder when external commands misbehaved. Yesod now
detects and logs trailing stdout pollution instead of silently accepting a
partial or contaminated JSON document. Profile-loading exceptions are
surfaced, and repeated gate_infra_unavailable outcomes escalate through the
janitor. These are small changes with a common purpose: a control-plane claim
must carry enough evidence to be trusted, and an infrastructure failure must
remain distinguishable from a test failure.
The Refinery Became Crash-Only
The largest cluster was the crash-only refinery work. A killed worker should not strand the merge queue, a dead process should not retain the main lock, and a gate whose client connection disappears should not lose its verdict.
The new lifecycle is built from several cooperating mechanisms:
| Failure boundary | Mechanism | Evidence |
|---|---|---|
| gate result | write the verdict and log checkpoint before reporting completion, so a severed client can recover the result | cff9186f, migration 0035 |
| operator cancellation | refinery abort-gate marks the queue item aborted without pretending that it was a test red; gate-attempt accounting remains explicit |
124a3d33, 31d39168, migrations 0036 |
| merge lock | renew a committed lease with heartbeat, record owner PID and host, and reap a dead or stale owner | eccb266e, migration 0037 |
| worker death | a lease-heartbeat thread and abort flag let the worker terminate itself when its ownership is lost | b85f9fdc |
| supervisor hang | bound database connection, statement, and lock waits; abort a tick that exceeds its wall-clock budget | bebedcd6, f7d3af33, 605f109e |
| fleet churn | tear down terminal fleets immediately, exclude terminated ghosts from capacity, and debounce relaunches | 1d0f4367, 3c086616, c0b8323c |
This is stronger than adding retries. Retries are useful only when the system knows who owns the work and which result is authoritative. The lease and checkpoint model makes recovery a state transition: a new worker can see that the old owner is gone, reclaim the lock, inspect the last durable gate state, and continue without guessing whether the previous attempt merged, failed, or was merely disconnected.
The practical trigger was a real AME hang. A supervisor sat in an unbounded Postgres poll for roughly thirty minutes while work waited and the merge lock was free. The fix made database operations bounded and added a crash-only tick watchdog. A separate dashboard heartbeat fix writes at tick start and periodically, and conditions stale/dead display on a live worker row. The dashboard can now distinguish “the supervisor is busy” from “the supervisor is gone.”
The Fleet Survived Its First Real Storm
The first implementation of shard fan-out could create a new failure loop. The
gate proxy and the dashboard poller both called AWS ListMicrovms; under
throttling, a missing count was interpreted too optimistically, shards cycled
from running to terminated, and relaunches consumed the fleet cap without
reaching pytest.
The fix combined adaptive SDK retries, a TTL cache for fleet listings,
terminated-ghost exclusion, partial-fleet adoption and relaunch, two-strike
shard-death debounce, and a corrected poller port/interval (1d0f4367, with
the deployed patch also recorded as 43d18702). The fleet now tears down
terminal instances promptly and retains per-shard failure tails in the gate
log (861d2f39, a291d9e6). A stale fleet snapshot cannot override live
merge_jobs or gate_runs telemetry (bd283ef1, 02c276ed, 3e7b065f).
The evidence is unusually concrete. The focused proxy, poller, and plist tests passed 93 tests. The subsequent operational note reports more than twelve complete eight-shard fleets launched and reaped cleanly, with zero throttling errors across approximately 870 proxy-log lines; before the fix, the same class of log accumulated roughly 290 throttling errors per hour. This does not prove that every possible repository flavor is green, but it proves that the fleet lifecycle itself stopped self-amplifying under normal eight-shard load.
A Generic Gate Acquired a Repository Contract
Wave 2 showed that a generic image could be asynchronous and still be wrong for its repository. Wave 3 made repository identity a first-class gate input.
The gate_registry is now the source of truth for per-tool proxy ports and
MicroVM flavors. The live convention reserves 8099 for Yesod, 8100 for
TTRAC, and 8101 for SJBGTD; new lanes allocate upward. Python/uv/Postgres
and Node flavors are named explicitly, and the refinery gate-plan command
shows the intended cutover state before an operator changes it
(22bcbea9, 82c97a35).
The worker no longer assumes that every remote batch belongs to the warm Yesod
repository. Non-Yesod lanes carry their own repository URL, and the worker
refuses a foreign lane whose manifest information is missing rather than
silently gating the wrong checkout (3ccefebc, 6c37b383). The remote proxy
fetches the selected tool’s deploy-key secret and injects only the material
needed for that job; the guest consumes it during the per-job clone
(f2a803a1, 9c8d9c94, 8d2bf045). The old single baked deploy key was the
wrong abstraction for a multi-repository factory.
The yesod profile now defaults to the MicroVM gate, and the automatic local
fallback is retired (cc6374b9). A remote failure is a typed infrastructure
failure that leaves work queued. YESOD_GATE_BACKEND=local remains available
as an explicit operator rollback. Other languages or bespoke shell stacks
without a registered flavor can remain local until their gate image exists;
“MicroVM everywhere” became a registry and validation problem, not a blanket
assumption.
The Factory Learned to Create Projects and Agents
The wave widened the boundary of Yesod itself. yesod project new now
scaffolds a new repository from a template, with Rust, Python/uv, and Apple
flavors in the initial surface. It initializes the project, creates and shares
the GitHub repository, accepts the bot invitation, and registers the project
with the factory’s workspace, refinery, and Beads conventions
(76402192, 056f0f18). The factory is beginning to produce new production
lines, not merely changes to its own machinery.
Agent execution received the same treatment. The six-layer spawn repair adds
the correct launchd PATH, session inspection through the opencode HTTP API
with CLI fallback, spawn preflight with PATH/YESOD_BIN injection, prompt
delivery checks, a post-spawn registration/liveness gate, and a turn watchdog
that re-prompts sessions left with a dangling tool (32a8f1fc through
db914780). This turns “the agent died silently” from an operator intuition
into a sequence of checkable lifecycle states.
Dispatch policy also became more mechanical. Arm paths refuse notes that are
not open or are already held in merge work unless --force is explicit;
propulsion excludes done and awaiting-merge notes from false readiness findings;
and repeated nudges are rate-limited to roughly two hours instead of flooding
the mayor inbox (e9435a54). The dead opencode-claude lane was removed, an
opencode-minimax lane was added, and the infra-monitor seat default was
provider-prefixed (25cee8cb, 31938769).
The Dashboard Became a Deployable Surface
The dashboard is now aware that it may live behind a reverse proxy rather than
at /. A request-scoped prefix helper rewrites HTML links, forms, fetches,
EventSource URLs, and redirects. Live Viz derives its API base from the
pathname and uses secure WebSockets behind HTTPS. The canonical operator URL
is now advertised as the proxied Yesod path, while direct local access remains
unchanged (0248d393, 05da3ae5, ba9af467).
The operator experience also gained state-transition audio controls and a shared LCARS-style audio engine. That work is cosmetic compared with leases and credentials, but it belongs to the same pattern: the factory’s state is becoming something an operator can perceive directly, not a collection of database rows and raw logs.
What Shipped, and What Stayed Out
| Area | Shipped result | Boundary at wave exit |
|---|---|---|
| readiness | PostgreSQL mirror, complete ready-set retrieval, single-candidate verification, and drift alerts | Dolt remains the external readiness source; the mirror must continue reconciling |
| refinery safety | write-ahead checkpoints, abort-gate, leases, heartbeats, dead-owner reaping, bounded DB calls, and tick watchdog | crash diagnostics and broader supervisor self-healing remain follow-on work |
| fleet execution | adaptive throttling defense, ghost suppression, clean terminal teardown, per-shard failure retention, and remote-default Yesod gating | non-Python or bespoke flavors still require explicit local treatment |
| repository isolation | gate registry, per-tool ports/flavors, manifest guards, and per-job deploy-key injection | each new lane still needs operator-approved provisioning and one real end-to-end gate |
| factory expansion | project templates, GitHub bot acceptance, six-layer agent spawn health, and safer dispatch nudges | AWS-hosted Yesod remains an exploratory roadmap, not a shipped deployment |
| operator surface | reverse-proxy-safe dashboard, secure WebSockets, and clearer live state | dashboard polish does not replace the durable database and lease model |
The New Exit Criterion
Wave 2 asked whether an asynchronous gate could be made real. Wave 3 asked whether it could be trusted after the process, host, repository, credential, or network around it failed. The answer is now much stronger: the fleet has a recoverable lifecycle, the Yesod lane has an explicit remote default, and a foreign repository cannot silently inherit Yesod’s identity or key.
The next wave should measure the factory by lane onboarding and recovery time: how quickly a new repository can be assigned a valid flavor, credential, proxy, refinery profile, and green canary—and how quickly a killed worker or stale lease returns that lane to useful work. The architectural lesson of this wave is that autonomy is not the absence of intervention. It is the presence of enough typed state, bounded waiting, and explicit ownership that intervention becomes rare, safe, and explainable.