We tried to ship to AWS. The factory shipped us bugs instead.

The goal was simple to say out loud: push two notes—the AWS runtime contract and the AWS deployment design—through the factory and stand up AWS-hosted Yesod v2.

What happened was more useful. The AWS branch moved only a little. Each small, offline, docs-and-contracts change put a new kind of pressure on the factory around it, and that pressure surfaced defects that had been waiting for a sufficiently disciplined workload: a fingerprint that changed depending on who computed it, a publisher that orphaned its own children, an immutability proof that a stray .pyc file could break, and a hold that arrived one tick too late.

The infrastructure initiative stalled. The infrastructure beneath it became more honest.

The fingerprint changed depending on who asked

The first root sailed through. The second—a note with review comments—would not plan. Attempts superseded themselves with stale_note_revision, as if the note kept changing underneath us. It was not.

The planning system hashes a normalized note into a fingerprint; admission and finalization must agree on it or the attempt is discarded as stale. The operator CLI installed globally was running an older services checkout whose planning input excluded note comments. The deployed runner release included them. The release-skew record captured the exact divergence. The moment a note carried a comment, the CLI hashed it one way and the runner another. Uncommented notes planned normally; commented notes repeatedly superseded, and the error did not explain why.

The immediate response was to align the operator and runner releases and prove parity across the surfaces involved. The durable response is broader: explicitly version the fingerprint schema and make the server authoritative, so admission and finalization cannot silently disagree. That separate schema-skew bug remains part of the public record. The lesson travels well: when two releases jointly decide whether an input changed, the normalization contract is part of the protocol—not an implementation detail.

The publisher orphaned its own children

When a note is replanned with fewer child Beads, the removed children have to go somewhere. The BeadPlanPublisher was supposed to retire them. On a rev6 → rev7 replan it instead left dropped siblings open with stale metadata, beyond the validator meant to catch them. The orphan-child report records the failure and the repair.

The repair took two adversarial rounds. First came a descendant fence: never retire a dropped sibling that has descendants, even if those descendants look terminal, because the path still reaches a destructive delete. Then came a canonical-negative proof: a dropped child can be retired only when the lifecycle log proves it never ran. Every field of the runner’s negative-launch signature must agree. Missing or contradictory evidence vetoes retirement.

That sounds fussy until the operation is deletion. Then “prove the negative” is exactly the right default.

The mutation happened before the fence

Hardening the publisher exposed a more dangerous cousin. The job-completion path recomputed a plan and mutated Beads before its final freshness check. A recompute during finalization could rewrite child Beads beneath an artifact already being published.

The publish-intent race was marked critical and serialized ahead of further AWS work. The repair places a durable intent and a revision/fingerprint fence around the entire mutation window. A stale or lease-lost job fails before external Beads effects; a crash after partial deterministic publication must resume the same artifact rather than switch plans halfway through.

The bytecode broke the proof

Deploying the fix became its own gauntlet. A Yesod release is staged as an immutable tree, made read-only, and proven read-only before it goes live. That proof is load-bearing—and surprisingly easy to invalidate. A root-run Python process that imports from the staged tree can create a __pycache__ directory. The supposedly immutable tree now has a writable child, so promotion correctly stops in needs_operator.

The catalog records both the audit-transport write and the broader missing bytecode guard. The remediation was deliberately redundant: set PYTHONDONTWRITEBYTECODE for every relevant unit, recreate the vulnerable cache directory empty and read-only, then make the sentinel immutable so an audit cannot repopulate it.

The promotion needed three attempts. A projector-quiescence timeout rolled back the first; the bytecode class stopped the second. On the third, the fleet landed uniformly. An immutable release is not a folder with a reassuring mode bit. It is a property every consumer must preserve.

The helpful command poisoned every start

An adjacent runner-start failure added a second warning. A service unit invoked uv run, which can synchronize against the release directory it runs from. That behavior is useful in a development checkout and toxic in an immutable release. One command in the wrong directory can leave every subsequent start consuming a tree that no longer matches its proof.

The pattern is larger than Python: immutable releases and tools that helpfully mutate their working tree are natural enemies. Release consumers need execution-only entrypoints, with environment mutation moved back into the build phase.

The hold arrived one tick too late

To inspect a change before it gates, we initially tried setting the note disposition to HOLD at awaiting_merge. That does not create the guarantee its name suggests. Refinery v2 can admit the attempt at the edge before the hold monitor observes it. The race is documented, along with a second edge: an “unchanged” disposition can make a reason-only update silently no-op, as captured in the disposition-write report.

The mechanism that actually fences admission is the durable global admissions pause: a compare-and-swap revision, a capability token, and a transaction boundary shared with every admission. It gates enrollment, not completion. An attempt enrolled before the pause is allowed to settle. To merge exactly one reviewed attempt, resume, observe one enrollment, re-pause, and let that in-flight attempt finish with the gate closed behind it.

That distinction—enrollment versus completion—is what turned “hold this” from a wish into a protocol.

The status query was secretly a sweep

The smallest finding may be the most reusable. A bare yesod estimate-complexity looks like a question. It is a live mutating LLM sweep. The CLI trap record documents that the accidental invocation updated six notes, one model call and one database update at a time. The audit found no correction worth applying, but the command was not read-only.

There is a safe preview flag. The operational lesson is cheaper: run --help before an unfamiliar verb, especially when the verb sounds like a report. A command named like a question can still be an action.

What the initiative was actually for

Together, the defects fall into four families: determinism, destructive safety, immutability, and ordering. They were quiet until a workload demanded the same input produce the same decision on multiple releases; that nothing be deleted without proof; that read-only remain read-only under every consumer; and that a closed gate close before the thing it must stop.

There is a deployment lesson too. Much of the friction came from trying to preserve immutable releases on persistent hosts while operational tools naturally wanted to write nearby. The durable direction is to provision fresh at a known SHA, rather than make mutation of old hosts increasingly ceremonial. That is the direction of the create-and-destroy runner work.

We set out to push two notes toward AWS. We are not there yet. But the factory now has a durable publication fence, stronger proof before child retirement, stricter immutable-release consumers, and a real admissions boundary for reviewed work. The remaining fingerprint contract is named and public instead of hiding behind phantom staleness.

That is not the initiative we planned. It may be the preparation the initiative required.

Machine-authored from the factory record. The links above open the live Yesod notes—including their current status and Bead trees—so the article’s claims can age in public with the system that produced them.