Wave Summary

Field Value
purpose turn the remote gate into an asynchronous, shard-capable job protocol; convert incident lessons into safe operator commands; make the control plane honest about local and remote execution
development window July 15, 11:13 to July 20, 20:18 PDT—5 days 9 hours 5 minutes
baseline b838cd56
final commit 3ca2edff
Git commits landed 77 total: 65 non-merge commits and 12 merge commits
contributors 4 Git author identities
change surface 81 files, 10,280 insertions, 613 deletions
completed feature notes 12 feature requests created after the Wave 1 cutoff, plus incident-driven bug fixes
exit state async and shard code merged; operator CLI gaps closed; 22 refinery profiles normally gate locally; MicroVM cutover rolled back after a broken image; observability work remains active

Wave 1 ended with a production path that was safe enough to deliver external work. Wave 2 began when that path met reality. Between July 15 and July 17, operators repeatedly reached for direct database queries during recovery. At the same time, the remote gate still held a long-running request open, and the first attempt to make it asynchronous exposed a deeper image and deployment problem.

The wave therefore had two connected objectives. The first was architectural: replace a long, blocking gate request with a job protocol that could support parallel shards. The second was operational: make the state, permissions, and repair actions visible through Yesod itself. The resulting wave is not a story of a clean remote cutover. It is a story of a better protocol, better control surfaces, and a sharper boundary around what had actually been proven.

The Gate Became a Job Protocol

The first three steps of the remote-gate chain changed the unit of work:

  1. POST /run validates the request, writes a queued job record, starts the gate in a background thread, and returns a job identifier quickly.
  2. GET /status/<job_id> reads the job record and progress files without waiting for pytest. It reports phases such as queued, fetching, running, done, and error.
  3. The final verdict is written before it is reported. A client that loses its connection can recover the result by polling the durable job record.

The lock discipline matters as much as the endpoint shape. The worker holds the setup lock for database and batch preparation, but pytest runs without holding that lock, so status polling does not compete with the whole test run. This turns a request timeout from an ambiguous failure into a recoverable observation problem.

The protocol then grew a second dimension. A run can carry shard_index and shard_count; pytest-split partitions the suite; and the pool proxy launches and tracks a fleet rather than one MicroVM. The proxy polls each job, reports per-shard progress, retries bounded infrastructure failures, and computes the overall verdict as the conjunction of the shard verdicts. A dead or setup-failed shard is infrastructure failure, not a red test result.

This is a meaningful change in the factory’s state model. The gate is no longer “one HTTP request that eventually returns a verdict.” It is a collection of durable jobs whose progress, failure class, and terminal evidence can be reconciled independently.

The supporting work followed the same logic. The image build gained optional duration data for shard balancing and a flavor-specific provisioning path. Launch and reap loops gained jitter so several workers do not synchronize expensive AWS polls. Agent spawning gained asynchronous prompt delivery and session pruning after an incident left more than 160 stale opencode sessions. The intended performance target was a shard fleet completing in roughly four to eight minutes instead of a roughly fifty-minute local gate. That target motivated the design; it was not established as a production guarantee by the end of this wave.

The Cutover Failed at the Image Boundary

The most important evidence in this wave is the failed remote cutover.

The rebuilt yesod-gate-async-shard image, active as version 26.0, carried a venv without the yesod package. Every shard reached setup, attempted to load the pytest plugin, and failed with ModuleNotFoundError: No module named yesod; zero tests ran. A gate attempt burned sixteen MicroVMs before the storm guard stopped the launch pattern. The operator rolled the supervisor back to YESOD_GATE_BACKEND=local, exactly as the rollback plan required.

The root cause was more specific than “the image was bad.” The general flavor still contained a TTRAC repository URL and gate database name inherited from an earlier lineage. Because build.sh was the input for the Yesod async-shard image, the image could be structurally healthy while cloning and configuring the wrong project. The follow-up added a build-time image-environment self-check and replaced the stale TTRAC values in the Yesod path. Those fixes improve the next build, but they do not retroactively make the failed image a green end-to-end proof.

The operating state at the close of the wave was therefore explicit: all 22 refinery profiles were gating locally. Local execution was the normal path, not a degraded remote fallback. The MicroVM implementation remained present and was rendered accurately in the UI for a later, operator-gated cutover. This distinction prevented a fast protocol from being mistaken for a proven deployment.

The Operator Stopped Needing psql

The incident marathon also produced the most practical feature cluster in the wave. A durable operator surface replaced roughly 25 direct database pokes, including 10 writes:

Area New capability Evidence
note administration inspect status, disposition, attempts, branch, and dispatch together; set a validated branch; reset gate attempts with an audit reason ys-yes-00fd, fb293fc0
refinery administration inspect/edit profiles, inspect the merge lock, list workers, and reap only verified-dead rows ys-yes-00fd, 4a4c7184
truthful views compute branch_exists from the profile’s repository rather than the caller’s current directory; expose queue state and age ys-yes-00fd, 4a4c7184
diagnostics answer “why is this armed note not running?” and check test-schema health from sanctioned commands ys-yes-00fd, 1a986359
mail read complete message bodies from scripts; purge off-roster agent mail while preserving human channels ys-yes-00fd, ys-yes-gchl
recovery rearm a repaired refinery item atomically instead of manually resetting monotonic counters ys-yes-lgsz, fadf1483

The important result is not command count. It is that repair actions now have names, validation, and an audit trail. A branch repair can verify that the remote ref exists. A gate-attempt reset can carry a reason. A worker reap can require evidence that the process is really dead. The control plane is moving from “the operator knows which tables to edit” to “the system exposes the allowed state transitions.”

That is also why the wave added explicit model-and-effort lanes, tool-scoped gating marks, and a visible planning status. Model selection, work ownership, and planning are policy decisions; they should not be hidden in free-form prompts or inferred from whichever worker happens to claim a row. The guidance change that one note represents one independently mergeable feature, while its Bead tree represents internal sequencing, made this boundary explicit (ys-yes-yu4c, 712d5c52).

Observability Learned to Name States

The Refineries page had been shaped around the MicroVM path even while local gating was the live production state. The wave made it backend-aware: local and MicroVM gates now have distinct rendering and progress helpers, and the page retains remote detail without implying that remote execution is active. The final readability pass also removed a stale panel timestamp, labeled gate speed as relative to supervisor start after a restart, and replaced a bare zero-percent collection bar with an explicit collecting state.

Those changes reflect a broader lesson. A percentage is not a state machine. During pytest collection, zero percent can be healthy. After a supervisor restart, “no completed gate” can mean “no completed gate in this generation,” not “no gate has completed.” A dashboard that collapses these distinctions creates operational work even when the underlying worker is correct.

The last review of this wave also found two gaps that the final clarity pass did not yet close. Local gates did not export the same progress signal as the fleet path, so a healthy long-running local gate could still appear stalled. And the worker could stop heartbeating while blocked in pytest, allowing a live worker to look dead after the ghost threshold. Those findings belong in the next wave, not in the shipped-results column.

What Shipped, and What Stayed Out

Area Shipped result Boundary at wave exit
async execution job identifiers, background execution, status polling, write-ahead verdicts full remote production cutover was not proven
shard execution pytest-split integration, pool-proxy fan-out, per-shard aggregation and bounded infra retry the first rebuilt image failed before tests ran
image safety Yesod-specific environment correction and build-time image self-checks one green end-to-end rebuild and cutover approval still required
operator control sanctioned note, refinery, mail, health, and recovery commands direct SQL knowledge remains useful for diagnosis, but is no longer the normal repair interface
dispatch policy explicit Codex model/effort lanes, planning lifecycle adoption, tool-scoped gating model economics still need observation across external work
observability backend-aware rendering and clearer collection/timestamp semantics local progress export and heartbeat semantics remain open

The evidence is therefore layered. Seventy-seven Git commits landed across 81 files, and feature-specific tests accompanied the async, sharding, CLI, and observability changes. The remote image incident supplied a stronger kind of evidence than a green unit suite: it showed that build inputs, package contents, runtime environment, and rollback behavior all belong to the gate contract. A focused validation run covering the async gate server, pool proxy, refinery CLI, CLI black-box, and backend-aware Refineries tests passed 304 tests. The local rollback succeeded; the remote cutover did not.

The New Exit Criterion

Wave 1 asked whether Yesod could safely leave self-hosting mode. Wave 2 asks a more demanding question: can a distributed execution design be proved at the same boundary where it will operate?

The next cutover should require a build-time environment check, one single instance /run diagnostic, one genuinely green shard, and then one complete multi-shard gate whose verdict matches a local run. Only after that evidence should the supervisor be pointed at the fleet. Separately, the local path needs progress and heartbeat semantics that cannot declare a healthy gate stalled merely because pytest is quiet.

The wave’s durable lesson is simple: asynchronous execution increases the need for explicit state; parallel execution increases the need for typed failure; and operational autonomy increases the need for sanctioned, audited control surfaces. Speed is valuable, but the factory earns the right to use that speed only when the image, worker, gate, and dashboard tell the same story.