Relay Relay / Docs Open the board
Menu

ARCHITECTURE

The runner: how work physically gets done

Today's system. ADR 0006 landed a server-side flow engine + thin runner for every stage — Spec (RLY-136), Plan (RLY-138), and Code (RLY-139, this doc's most recent cutover). The legacy board-runner (relay watch, relay_config.json, .claude/workflows/execute-plan.js) is deleted; there is no fallback dispatcher to describe. The runner lives on a developer machine — it needs the checkout, git worktrees, and the claude CLI — and talks to the deployed app only through the board-key REST API.

./relay (Python, single file) is two things:

  1. A CLI for every card operation an agent needs (card, move, comment, needs-input, approve, …) — the surface documented in ../../relay.md.
  2. relay start — the only runner mode: a poll loop that claims node-jobs from the server and runs them (see "Runner mode" below).

Dispatch is server-side

A sketch of a Code flow in this model (edges labeled with the outcome that routes them):

flowchart LR
    start([start]) --> impl["agent: implement task<br/>(runs repo skills)"]
    impl -- succeeded --> review["agent: spec + quality review"]
    review -- failed --> fixf["agent: fix findings"]
    fixf -- succeeded --> review
    review -- succeeded --> pre{"gates: mix precommit<br/>+ mix test.browser"}
    pre -- failed --> fix["agent: final fix"]
    fix -- succeeded --> pre
    pre -- succeeded --> smoke["agent: final review → smoke"]
    smoke -- failed --> fix
    smoke -- needs_input --> human{{"human answers<br/>(implicit pause — card blocked)"}}
    human --> smoke
    smoke -- succeeded --> acceptance["agent: acceptance"]
    acceptance -- succeeded --> resync["shell: rebase onto origin/main"]
    resync -- succeeded --> reverify{"gates: mix precommit<br/>+ mix test.browser"}
    resync -- failed --> resync_fix["agent: rebaser (parks on semantic conflict)"]
    resync_fix -- succeeded --> reverify
    reverify -- succeeded --> merge["shell: push · PR · auto-merge"]
    reverify -- failed --> resync_fix
    merge -- succeeded --> deploy["shell: await merge + deploy"]
    merge -- failed --> resync
    deploy -- failed --> ghfix["agent: ci-fixer"]
    ghfix -- succeeded --> resync
    deploy -- succeeded --> post["agent: post checklist"]
    post -- succeeded --> done([done])

A run rebases onto origin/main twice — once before the expensive review/smoke/acceptance tail and once immediately before merge, each followed by mix precommit and mix test.browser — so a busy board moving main under a long run no longer strands the work at merge (RLY-192); real conflicts route to the rebaser agent, which parks for a human on a semantic conflict. After merge queues the auto-merge, deploy (bin/await_deploy.sh) waits for the PR to merge and main's CI to deploy it, so a card reaches Review only once its change is live; post runs last so its summary describes what shipped. The authoritative graph is docs/designs/flows/code.json.

The RE board: PR-less Code, then a Deploy stage (RE408)

The graph above is the default library, which other boards still run. The RE board (this repo's own board) overrides two flows, checked in at .relay/flows/code.json and .relay/flows/deploy.json and pushed with ./relay flow-push <key> <file>. Neither file has a version, so a push is last-write-wins. test/relay/flows/re_board_flow_documents_test.exs runs both through Relay.Flows.upsert_from_document/3, the same path PUT /api/flows/:key takes.

flowchart LR
    rebrowser["… rebrowser"] -- succeeded --> merge["shell: bin/ship_to_main.sh<br/>(squash, fast-forward push)"]
    merge -- failed --> resync["resync"]
    merge -- succeeded --> await_ci["shell: bin/await_main_ci.sh<br/>(main's CI on that commit)"]
    await_ci -- succeeded --> post["agent: post checklist"]
    await_ci -- failed --> github_fix["agent: ci-fixer"]
    github_fix -- succeeded --> resync
    post -- succeeded --> codedone[["Code:Done"]]
    codedone --> checkout["shell: checkout -B {branch} origin/main"]
    checkout -- succeeded --> fly["shell: deploy_fly.sh"]
    fly -- succeeded --> ios["shell: deploy_ios.sh"]
    ios -- succeeded --> android["shell: deploy_android.sh"]
    android -- succeeded --> review[["Review"]]
    checkout & fly & ios & android -- failed --> human{{"needs_input"}}
  • Code (Plan:Done → Code → Code:Done) opens no PR. merge is bin/ship_to_main.sh {ref}: it squashes origin/main..HEAD into one <REF> <card title> commit and pushes it to main as a plain fast-forward, never with force. A rejected push fails into resync (max 2 loops). await_ci (bin/await_main_ci.sh) then waits for ci.yml's push run on that exact commit, re-running failed jobs once for flakes. Only a green run moves on to post and Code:Done, so the Deploy flow never picks up a commit CI hasn't passed. A red one goes to github_fix, the ci-fixer agent (max 2 loops). It commits a fix on top of origin/main, and the flow loops back through resync and the gates to ship that fix as a new commit. The PR-era deploy node is gone.

  • Flutter gate (RE424). The Code flow's flutter gate (bin/flutter_gate.sh) runs between precommit and browser. It diffs flutter/ against git merge-base origin/main HEAD and exits 0 immediately when the branch didn't touch it; otherwise it runs bin/flutter_validate.sh (flutter pub get, flutter analyze, a non-mutating dart format check, flutter test), and a failure goes to final_fix (max 2 loops), which re-enters at precommit. A detection error validates rather than skips. The gate is not repeated in the pre-merge tail (reverify / rebrowser): a rebase breaks Flutter only when main also changed flutter/, and the Deploy flow's ios / android still validate before any store upload. The board runs the pushed flow, so a change to the file takes effect only after ./relay flow-push code .relay/flows/code.json (then /relay-doctor, interactively).

  • Deploy (deploy flow, Code:Done → Deploy → Review, exclusive, stage WIP 1) only deploys what is already on main. Its four shell nodes (max_retries: 1, each parking on needs_input when it fails, with no AI fixer) run on our own runners:

    • checkout: {relay} git-fetch && git checkout -B {branch} origin/main.
    • fly: bin/deploy_fly.sh. This deploy always runs, and it bakes GIT_SHA / BUILT_AT into the image.
    • ios: bin/deploy_ios.sh {ref}, which goes to TestFlight.
    • android: bin/deploy_android.sh {ref}, which goes to Google Play.

    ios and android skip with exit 0 when the card's commit didn't touch flutter/ (bin/card_touched_flutter.sh), and Android also skips until its credentials exist (RLY-103).

  • Secrets come from the runner's own environment, which shell nodes inherit. On the RE board that is .envrc.local, loaded by direnv in the shell that ran relay start. Each deploy script fails fast naming any secret it's missing.

  • Why checkout -B and not --detach. On an exclusive flow the runner re-attaches the card's branch before every node, and it refuses a detached HEAD unless the command contains checkout -B. Pointing the card's branch at origin/main keeps that guard happy and runs every deploy on exactly what is on main. This is harmless, since the branch's work already shipped as the squash.

CI still runs the tests on pushes to main and on PRs. Its deploy jobs (ci.yml's deploy and flutter-deploy.yml's push: trigger) are commented out, not deleted, and flutter-deploy.yml's workflow_dispatch still works for a manual store build.

A card in any AI-enabled stage is dispatched by Relay.Runs.Scheduler (folding over every enabled Flow on the board, rightmost flow stage position first; each flow pulls from the stage before its own in board order, worked out by Relay.Flows.neighbours/2) straight to the node-job engine (Relay.Runs) — no per-stage config file, no board-runner poll loop. relay start claims the resulting NodeJob rows over the node-job REST API (below) and runs whatever node it is handed; it knows nothing about stages, columns, or which flow a job belongs to. Board-specific facts (stages, prompts, per-node budgets) live entirely in Flow/Flow.Node/Flow.Edge rows, seeded from docs/designs/flows/ and editable in Settings › Stages (each stage row's FLOW band).

A broken board shape pauses a flow (RE430). Relay.Flows.Shape.problems/2 is the one shape rule: the column before a flow's stage must be a :queue or :done column, and there must be a column after it. Scheduler.Server.build_snapshot/2 reads every flow of the board (enabled and disabled, Relay.Flows.list_flow_snapshots/1 — still one query), keeps the enabled ones as the snapshot's flows, and computes the snapshot's problems in memory from the same ordered stages and the full flow list (a disabled flow is still named as an upstream). An enabled flow with a problem is paused: plan/1 starts no fresh run for it but still resumes its parked in-flight runs, and a run already going finishes and lands from board order exactly as before — nothing on the landing or resume path checks the shape, and Relay.Runs.start_run/2 is not gated. The server subscribes to {:stages_changed, board_id} on the board topic and treats it like a card event (a full, debounced reconcile, never the quiescent skip), so fixing the board un-pauses the flow on the very next reconcile.

Shared-budget arbitration: rightmost flow wins ties. Relay.Runs.Capacity keys free slots runner_id => %{shared_clean: n, exclusive: n} per isolation class, not per flow (capacity.ex:5-7), and Relay.Runs.Scheduler.plan/1 threads one shared capacity accumulator through its fold, sorted rightmost flow stage (flows.stage_id) position first (scheduler.ex:38-45, rule documented at scheduler.ex:9-13). So when two flows share an isolation class and both have eligible cards under scarce capacity, the flow closer to Done draws first — intended WIP discipline, not starvation, even though under real scarcity it looks like the leftward flow is being starved. Pinned by test/relay/runs/scheduler_test.exs and exercised live over the REST API by test/relay_web/api/plan_flow_e2e_test.exs / test/relay/runs/code_flow_e2e_test.exs.

The capacity store (Relay.Runs.Capacity, ETS) is keyed by runner id; each entry carries the runner's board id and is evicted by the reaper's reclaim sweep once the runner goes stale (RE402). The scheduler snapshot is assembled from five narrow reads (stages, the Cards.list_scheduler_cards/2 projection, active runs, runners, every flow — RE402, RE430) and is board-scoped: Scheduler.Server.build_snapshot/2 keeps a capacity entry only for a runner of this board that Relay.Runs.counting_runner?/1 accepts (not :gone) — an allow-list, so another board's runner or an orphaned ETS entry contributes nothing to dispatch or to relay why's capacity evidence (RE338). Every snapshot capacity key is therefore a runner Relay.Runs.list_runner_status/2 names. What remains unscoped is one physical machine registered on two boards: that is two runner rows advertising the same slots, so two boards' schedulers can both count the same free slot. The runner's own live capacity is the final backstop, so an over-assigned job waits there rather than double-booking (YAGNI: no multi-board reservation yet).

Side channels

  • Log mirror: every feed line is queued to a background LogForwarder thread that batches POST /api/board/logs (best-effort: drops on full queue, swallows all errors) — landing in Activity.LogSink → the card timeline, and AgentLog → the live log sheet.
  • Runner heartbeat: RunnerHeartbeat posts {runner, capacity, running: [job-ids], held: [{ref, state}], harnesses, sessions} to POST /api/node-jobs/heartbeat every heartbeat_intervals (RLY-164) and reads back {revoked: [job-ids], release_held: [{ref, status}], want_capabilities, runner_outdated, required_version, latest_runner_version, harnesses_digest} (the digest since RE433 — see "Harness routing" below). It terminates each revoked job's live subprocess via its JobControl (see "Node-job transport" and "Runner mode" below). The advertised capacity is the runner's configured per-class total and this route is its single writer (RE311) — the claim's capacity is a live FREE count, passed to Relay.Runs.claim_next_job/4 as an argument and never persisted, because one column carrying both meanings made the roster flip-flop between the total and the free count while the board sat deadlocked. running is the jobs the runner believes it holds, so the server can name the ones it no longer considers live. held (RE311) is every per-card worktree the runner holds, each with a state from Schemas.Runner.holding_states/0 (bound | retained | running | talk); release_held is its ref-keyed analogue of revoked — the subset whose card's runs have all ended server-side, named with the status (done/failed/cancelled, Relay.Runs.releasable_held/2) that chooses remove vs retain. It replaces RLY-218's retired run-id-keyed release channel, which structurally could not see a worktree adopted by recover() after a restart — its run_id is unknown — so a run cancelled while the runner was down leaked its exclusive slot permanently. Since RE337 release_held also carries human-requested releases: the runners page's Release worktree action appends the card ref to the runner row's runners.release_requests (Relay.Runs.request_worktree_release/4, accepted only while the runner's latest held reports the ref bound), and the reply (Relay.Runs.release_held/3) adds {ref, status: "cancelled"} — a remove disposition (Schemas.Runner.release_remove_status/0), never failed — for each pending request this beat still reports bound. The run stays parked; on resume the runner's assign() rebuilds the worktree from the branch. Clearing rule: every beat that reports held prunes the column in one atomic UPDATE (upsert_runner/2) — a request survives only while its ref is reported bound or talk (teardown deferred under a talk turn); absent (torn down) or running (the run resumed first) clears it, so a stale request can never tear down a resumed run's tree. A runner that goes silent keeps its requests until it beats again. No runner change was needed: RunnerPool.release_held already refuses a live or talk-attached tree and stashes dirty edits as auto-salvage <slot> (never restored). The same running list also refreshes card liveness (RLY-226, Runs.refresh_running_card_liveness/2): the server stamps agent_heartbeat_at on the cards whose reported job is still active, the positive complement of the revoke query, so a live-but-quiet agent never falsely reads :stale in Cards.health/1. latest_runner_version (RE185/RE304) is the RUNNER_VERSION of the ./relay the app itself serves — Relay.Runs.latest_runner_version/0, delegating to Relay.Scaffold.runner_version/0, read from priv/scaffold/relay at runtime. Truthful by construction. It is a target, distinct from required_version's floor; a runner with auto_update on updates itself against it. Each harnesses[] inventory entry carries that harness's usage (RE436) — {source, read_at, windows, paused}, its latest readings and its pause if any — read fresh on every beat (see Usage limits below). RE320's top-level rate_limit is no longer sent. sessions (RE358) is {"<job-id>": "<session-id>"} — the session each in-flight agent job's stream has captured so far (JobControl.note_session, fed by _stream_agent_job's on_session), sent only when at least one job has one. Relay.Runs.record_job_sessions/2 writes each onto that job's NodeExecution.session_id — board-scoped and id-parsed like revoked_among/2, and only for an active flow job whose execution has no outcome yet (a beat can race the outcome POST). Anything malformed is ignored; the beat still answers 200. See "Steer now" below for why it exists.
  • Run ids: each runner worker tags its log lines with the claimed job's run_id (RLY-112) so a card's timeline can group lines by run.

Observability surface (RLY-177)

Read-only endpoints answering "why isn't this card moving?" without an fly ssh console Ecto query, board-scoped like every other /api route (a ref/id on another board 404s, never 403s):

  • GET /api/cards/:ref/diagnosis (RelayWeb.Api.DiagnosisController.show/2) — one verdict plus the evidence behind it, produced by Relay.Runs.diagnose/3, the boundary-safe facade the web layer calls (Relay.Runs' exports list is unchanged by this surface, so docs/architecture/domain.md needed no edit). It in turn calls Relay.Runs.Scheduler.explain/2, which replays Scheduler.plan/1's real dispatch decision — sharing its predicate functions — rather than reimplementing it, so the verdict cannot drift from what actually dispatches. A card the pulling flow would take while that flow is paused by a broken board shape diagnoses flow_paused (RE430): detail is Relay.Flows.Shape.paused_detail/1 and evidence.problem is Relay.Flows.Shape.wire/1 of the problem (flow_key, kind, what, why, fixes, columns); every other verdict carries evidence.problem: null. It is checked after no_enabled_flow and before blocked_by_dependencies, so earlier reasons (human-owned, needs input, a run in flight) still win.
  • GET /api/cards/:ref/runs (RelayWeb.Api.RunController.index/2) — the card's runs newest-first with node_executions preloaded, composing Relay.Runs.list_runs_for_card/1. detail and failure_detail are serialized in full, never truncated — the exact text a failing review's findings need to be readable for.
  • GET /api/runners (RelayWeb.Api.RunnerController.index/2) — composes Relay.Runs.list_runner_status/2 (no second runner read): advertised capacity per isolation class, last heartbeat, the tri-state freshness (Relay.Runs.runner_freshness/2; stale? is the freshness != :fresh convenience flag), version/outdated (Relay.Runs.runner_outdated?/1 — orthogonal to freshness, since a refused runner can still be beating normally), display_state (:gone > :stale > :outdated > :rate_limited > :fresh), rate_limit (Relay.Runs.active_rate_limit/2 — set only while every installed harness on the runner is paused, as the pause that lifts first: {harness, window, used_pct, pause_at, resets_at, reason}, nil once it resets; RE320, per harness since RE436), harnesses (the stored per-harness inventory and usage readings, [] for a runner that never reported), and the jobs each runner currently holds.
  • Web: the Runners view (/board/:slug/runners, RelayWeb.BoardRunnersLive) — the same Relay.Runs.list_runner_status/2 roster rendered one panel per machine, plus (RE307) the board-wide active queue from Relay.Runs.list_queue/2: every queued or claimed node job on the board, both kinds (:node and :talk), claimed or not, ordered as Relay.Runs.claim_next_job/1 will hand them out, with Relay.Runs.stopped_work/2's verdict above the rows. It renders whether or not a runner is connected — an empty roster is exactly when a stacked queue matters. Read-only and deliberately no new endpoint and no new CLI verb: POST /api/node-jobs/claim stays the only path a job is handed out on, and list_queue/2 is the function a future relay queue would render. Each runner panel's HARNESSES section (RE436) joins list_runner_status/2's harnesses with the board's harness definitions and agents (Relay.Runs.runner_harness_rows/4, read once per refresh): one row per board harness the runner reported, with its version, signed-in state, one usage bar per window the harness's USAGE setting defines and its own paused line. A harness removed from the board stops showing, and a harness the runner has not reported yet is not shown.
  • Rate-limited roster (RE320, per harness since RE436). A runner pauses per harness, and the server derives "rate limited" from the readings stored in runners.harnesses. Relay.Runs.harness_pause/2 reads one entry's live pause: the entry is installed, its usage.paused is set, and that pause's resets_at is still ahead. A stored pause past its reset reads as not paused before the next beat clears it. Relay.Runs.runner_rate_limited?/2 is the one predicate: the runner reported at least one installed harness and every installed one is paused (a nil or [] inventory is not rate limited). A partly paused runner is not rate limited: it still counts as connected and claims its other harnesses' work, and the claim's harnesses keeps the paused one's jobs away from it (see Harness routing). Relay.Runs.active_rate_limit/2 returns the pause that lifts first, as %{harness, window, used_pct, pause_at, resets_at, reason}, since that is when the runner can claim again; list_board_runners/1 selects harnesses for it. RE436 retired the runner-wide runners.rate_limit column, Schemas.RunnerRateLimit and the heartbeat's top-level rate_limit. Scheduler.capacity_diagnosis/1 returns :runner_rate_limited when the roster has live runners, not all of them are outdated, and every live, current runner is paused. Its evidence carries resumes_at (the earliest reset) and rate_limited_runners. The roster-blaming reasons live once in Scheduler.roster_blocking_reasons/0. Relay.Runs.stopped_work/2 returns the new reason for the board and Runners-page banner (warning tint). Relay.Runs.diagnose/3 (relay why, GET /api/cards/:ref/diagnosis) returns verdict :runner_rate_limited for a live run whose job is queued behind that roster; its sentence is Relay.Runs.rate_limited_run_detail/1 ("… every connected runner is paused at its usage limits"). Rate-limited runners are also excluded from :job_awaiting_slot's "connected runners". The card face reads Relay.Runs.roster_rate_limit/2 (one runners query through the same diagnosis) intersected with Relay.Runs.queued_run_ids/1 on BoardLive's health tick, and shows Rate limited · resumes … without the 5-minute stall threshold. The card drawer does not call diagnose/3 (a full snapshot per render); it reads the same per-card :run_face_meta entry, rendering rate_limited_run_detail/1 as a Run-tab banner and swapping the Activity tab's "gone quiet" chip for the rate-limited label. Copy comes from Relay.Runs.rate_limit_phrase/1 (codex 5h 93% / 90%, or claude-code refused (5h) for a refusal) and resume_time_label/1. No new PubSub topic.
  • GET /api/version (RelayWeb.Api.VersionController.show/2) — the git SHA the running app was built from, baked in at image build time (Dockerfile's final stage, fed by bin/deploy_fly.sh's flyctl deploy --build-arg in the RE board's Deploy flow; for a manual or other-board deploy, the commented-out deploy job in .github/workflows/ci.yml passes the same build args when it is restored). Unauthenticated, on the plain :api pipeline — it leaks nothing a deploy does not.
  • GET /api/flows/:key/metrics (RelayWeb.Api.FlowMetricsController.metrics/2) — the per-node rollup for a flow over a ?window=7d|30d|all window (default 30d): a scope ("flow" | "card"), the echoed card ref (null in flow scope), a summary stat band (total_runs, completed, completed_pct, total_spend, median_end_to_end, total_end_to_end) and a nodes array (runs, duration_p50/p95, cost_p50/p95 — null until runners report spend — duration_total, cost_total, attempts_mean, verdict_split, loop_laps, and the RE345 idle-gap keys wait_p50/p95, wait_total, wait_count, held_p50/p95, held_total, held_count, and the RE348 span keys work_total, rework_total, work_count, rework_count, rewind_total, rewind_count). The nodes come from Relay.Runs.node_metrics_for_flow/2, which merges in Relay.Runs.node_waits_for_flow/2: a node's gap is the time from the previous execution in the same run finishing to this one starting, and it is held when that previous execution ended needs_input/blocked (Schemas.NodeExecution.holding_outcomes/0) or the card logged a needs_input / input_answered activity inside the gap (an escalation park, whose predecessor stays failed — RE348), otherwise hand-off wait; a run's first node has none, and the window applies to the execution the gap precedes. The span keys come from Relay.Runs.execution_spans_for_flow/2: an execution is work when it is in the first visit of its (node, sub_task) binding (so foreach tasks 2..N are work) before that visit's first failed attempt, and rework otherwise — later visits, retries after a failure, and every execution of a fix-role node. Rewind (rewind_total / rewind_count, fix nodes only) is the non-fix rework charged to the fix that opened the charge, closed by the next work or fix execution. work_total + rework_total == duration_total. An optional ?card=<ref> (RE235) scopes every figure to one card's node executions across ALL of its runs of that flow: ?window= is then ignored, the eight percentile keys (duration_*, cost_*, wait_*, held_* p50/p95) and median_end_to_end come back null (a percentile over one card is noise), and an unknown, unparseable or other-board ref is a 404. Read-only, board-scoped.
  • GET /api/flows/:key/audit (RelayWeb.Api.AuditController.audit/2) — run-history health findings for a flow over a ?window=7d|30d|all window (default 30d), composing Relay.Runs.audit/2 (Relay.Runs.Audit.findings/2 over Relay.Runs.recent_runs_for_flow/2). Returns flow_key, the echoed window, the runs count examined, and a findings array of severity / check / flow_key / node_key / run_id / summary / evidence / fix. Four checks today: findings_dropped (a foreach cursor advanced past a failed review), verdict_flipped (a retry turned failed into succeeded at the same git_sha), planner_not_migrated (RE368 — one WARNING per flow when a run's tasks were seeded by the legacy plan-parse fallback, read from runs.tasks_from_plan; names the affected-card count and most recent refs, fix /relay-doctor), and outcomeless_attempts (RE410 — one WARNING per {run, node, visit} with at least Relay.Runs.max_outcomeless_reentries/0 executions that never reported an outcome: something keeps killing the node's job). The check ids are pinned as vocabulary.audit_checks in the runner contract fixture. The other half of relay audit — CI parity — is computed in ./relay, because the server has no checkout of any board's repo. Read-only, board-scoped, advisory.
  • GET /api/flows (RelayWeb.Api.FlowController.index/2) — every flow on the board, fully serialized in stable key order. One round trip is all relay doctor (RLY-240) needs.
  • GET /api/flows/:key (RelayWeb.Api.FlowController.show/2) — one flow as the canonical Relay.Flows.Document (RLY-241): key, version, enabled, isolation, the trigger as the one stage name the flow works in, {"stage": "<name>"} (portable across boards), and the ordered nodes/edges arrays. Sparse — nil fields and schema defaults are omitted. Beside the document sits a read-only "derived": {"pulls_from", "lands_on"} block (RE429, Relay.Flows.Document.derived/1): the stage names worked out from current board order, null at either end. ./relay flow prints it as pulls_from → stage → lands_on. A read-only "problem" (RE430) sits beside it: null, or Relay.Flows.Shape.wire/1 of the flow's broken board shape — flow_key, kind, what, why, fixes (action + label) and columns (stage_id, name, type, mark). ./relay flow marks an enabled flow with a problem PAUSED and relay flow KEY prints its What / Why / Fix verbatim. Every key set, plus Shape.kinds/0 / Shape.fix_actions/0, is pinned in test/fixtures/runner_contract.json ("flows", vocabulary.shape_problem_kinds / shape_fix_actions). 404s an unknown key.
  • PUT /api/flows/:key (RelayWeb.Api.FlowController.update/2) — upsert a flow from a document via Relay.Flows.upsert_from_document/3, in one transaction: Schemas.Flow.changeset/2's graph validation, save_definition/2's version-bump semantics (an unchanged push bumps nothing — pull → push is a genuine no-op), and enable_flow/1/disable_flow/1's arming rules. The response renders through the same view as GET, derived included. derived on input is accepted and ignored (a pulled file pushes back unchanged); a legacy three-key trigger is accepted, reading only works_in. 201 on create, 200 on update. Refusals: 422 invalid_document / key_mismatch / unknown_stages / stage_occupied (the named stage already holds another key's flow — error.stage / error.flow name it) / invalid, and 409 stale_version when the document carries a version that no longer matches (absent version = last-write-wins).
  • CLI: ./relay why REF / ./relay runs REF / ./relay runners / ./relay version / ./relay flow-stats KEY / ./relay flow [KEY] / ./relay flow-push KEY FILE / ./relay audit [KEY], documented in ../../relay.md.

Bootstrap surface (RE304, ADR 0010)

The board serves the scaffold again. Two routes on the unauthenticated /api pipeline, alongside /api/version:

  • GET /api/scaffold (RelayWeb.Api.ScaffoldController.manifest/2) — the manifest: {"version": "<12 hex>", "items": [{"path", "sha256", "bytes"}, …]}; the relay item also carries unbaked_sha256 (RE443, see Hashing below).
  • GET /api/scaffold/*path (.show/2) — the raw bytes of one manifest entry, 404 for anything else. Relay.Scaffold.fetch/2 checks a static allowlist, so this is not a general file server and traversal is impossible by construction.

Unauthenticated because the board's install script (/b/<slug>/install) downloads ./relay before anything has authenticated, and because these files were published openly regardless.

Exactly five files are Relay-owned (Relay.Scaffold.items/0): ./relay, the three .claude/skills/relay-{update,doctor,onboard}/SKILL.md, and relay.md. .claude/skills is the manifest's canonical skill dir (Relay.Scaffold.skill_dir/0, RE457, the ONE Elixir definition); Relay.Scaffold.items/1 maps the skill items to another LLM's dir (.agents/skills for Codex) without changing the served manifest. (RE443 retired the fourth, setup skill: the board's setup wizard hands out the install command instead.) They are never user-edited, which is what lets ./relay update overwrite them unconditionally — no provenance ledger, no per-file diff prompt. Everything else in a project (agents, other skills, AGENTS.md/CLAUDE.md, .relay/runner.json, flow documents) is out of scope and is never served by this surface; the flow documents and .relay/runner.json are wired by /relay-onboard, which works step 5 of the board's setup checklist (5.1–5.8). The one local exception (RE383): when .relay/runner.json is missing, ./relay update renders a starter one from the runner's own template — no endpoint serves it — and reports it under the config key (created, missing under --check, or present). It is never overwritten, never part of the scaffold version or manifest, never touched by auto-update, and not behind the in-app-repo --force refusal. --base REF sets the base of a runner.json the update creates.

The project's LLM layout (RE457). .relay/runner.json carries two optional, user-owned keys: skill_dir (default .claude/skills) and pointer_file (default CLAUDE.md) — the defaults are ./relay's DEFAULT_SKILL_DIR / DEFAULT_POINTER_FILE, pinned against the board by setup.runner_config_defaults in the runner contract fixture, as ./relay's POINTER_LINE is by setup.pointer_line (Relay.Setup.pointer_line/0). The manifest keeps its canonical .claude/skills/… paths; ./relay update maps every item under .claude/skills/ to <skill_dir>/… (local_scaffold_path) for its comparison, its install and its report — a skill is still fetched from its manifest path but written, hashed and reported at the local one, and the report carries skill_dir. ./relay update --skill-dir DIR --pointer-file FILE (either flag alone works) switches an existing project to another LLM's layout: it rewrites only those two runner.json keys in place, installs the skills under the new dir, removes Relay's three old skill files (reported as obsolete, removed on apply — never anything else in the old dir), and appends the pointer line to the new pointer file (listed in written). It commits nothing; every path and the runner.json rewrite are checked before anything is written, and the app-repo refusal applies. With --starter-only, the same flags only set the two keys of a runner.json it creates. Board Settings → Agents shows the switch command as a flash when "Make default" moves the board to an LLM with another skill dir (Relay.Setup.llm_switch_notice/2); the board never changes repo files.

./relay update --starter-only [--base REF] [--skill-dir DIR --pointer-file FILE] (RE443) writes only the user-owned starter files, each only when missing: that .relay/runner.json and a generic .relay/prepare-worktree.sh (mode 755, render_prepare_hook() — the copy-from-cache-or-main- checkout warm-up, generalized over a WARM="" list the project fills in; empty, it copies nothing and exits 0). No network, no scaffold work, not refused in the Relay app repo, never an overwrite. Report: {"config": "created"|"present", "hook": "created"|"present", "applied": true}. The install script runs it right after downloading ./relay, before ./relay update.

The served ./relay carries its board's URL (RE443). The committed file has a placeholder line BOARD_URL = "" right after RUNNER_FINGERPRINT; ScaffoldController.show/2 serves it via Relay.Scaffold.fetch("relay", board_url: RelayWeb.Endpoint.url()), which rewrites that one line to BOARD_URL = "<url>" (Relay.Scaffold.bake_board_url/2 / board_url_line/1, the ONE Elixir regex; ./relay's bake_board_url / board_url_of / BOARD_URL_RE are the ONE Python parse; the contract fixture's scaffold.board_url_line_example pins the two). The URL is the endpoint's configured one, never the request's Host, because this text gets executed.

  • Lookup: relay_url() — RELAY_URL when set and non-empty, else the baked BOARD_URL, else it dies pointing at the install command. Every URL-needing call site goes through it; a node's {url} variable uses the non-dying configured_relay_url().
  • Hashing: the manifest describes the served bytes — Relay.Scaffold.manifest(board_url: RelayWeb.Endpoint.url()) sets the relay item's sha256/bytes to the baked copy's, because every runner up to v89 verifies a download with a raw sha256 against that field, and adds unbaked_sha256, the built placeholder copy's (the build never bakes). ./relay v90+ hashes relay with its BOARD_URL reset to "" (runner_sha256) and compares it with manifest_sha(item) — unbaked_sha256, or sha256 from a board that predates the key — both for scaffold_status and for verifying a downloaded body before install; otherwise a baked local copy would read as changed forever. The scaffold version is derived from the built (unbaked) shas, so it is one value per build on every board. The contract fixture pins unbaked_sha256.
  • Fingerprint: FINGERPRINT_MASK masks RUNNER_VERSION|RUNNER_FINGERPRINT|BOARD_URL, so a baked copy fingerprints the same as the committed one.
  • Carried across rewrites: install_runner keeps the existing file's non-empty BOARD_URL over the one it is installing, so neither ./relay update nor auto-update ever drops (or repoints) a project's board.

The install handshake (RE443). GET /b/:slug/install (RelayWeb.InstallController, a pipeline-free public route) serves a POSIX sh script rendered by RelayWeb.InstallScript.render/2 from lib/relay_web/install_script.sh.eex, for any slug in Schemas.Board.slug_format/0 (whether or not that board exists; anything else is a plain 404). It bakes RelayWeb.Endpoint.url/0 and the format-checked slug — never the request's Host — plus the check ids from Relay.Setup.preflight_checks/0 and the CLIs from Relay.Setup.onboarding_llms/0. Run in a repo as curl -fsSL <host>/b/<slug>/install | sh with RELAY_API_KEY exported, it:

  1. runs the local checks (git_repo — pwd is the work tree's top level; remote — origin; default_branch — origin/HEAD, else the current branch; python3) and command -v for each onboarding CLI, using only curl git python3 uname mktemp mkdir chmod mv rm cat grep printf;
  2. POST /api/setup/preflight (board key) — the key's first authed call, which ticks setup step 3 and marks step 4 waiting or failed with the fix text. The answer's llm carries harness / label / cli / skill_dir / skill_path / pointer_file, and (RE457) its relay_files is Relay.Scaffold.items(llm.skill_dir), the five Relay-owned paths under that LLM's skill dir. It prints the board's answer, one ok/!! line per check; on any failure, the fixes and fix_command, then exits 1 having written nothing;
  3. (RE457) prints a "Writing:" preview — one line per relay_files path, then the pointer file (only when its line is absent), .relay/runner.json, .relay/prepare-worktree.sh and .gitignore (+N lines) — and downloads only relay from /api/scaffold/relay (to a temp dir, then ./relay, mode 755). It appends the pointer line (Relay.Setup.pointer_line/0, rendered into the script) to the LLM's pointer_file and the three .gitignore entries — each only if absent — then runs ./relay update --starter-only --base origin/<branch> --skill-dir <llm.skill_dir> --pointer-file <llm.pointer_file> and RELAY_URL=<board> ./relay update --json, which lays down relay.md, the three skills under skill_dir and .relay/scaffold.json, every body checked against the manifest's sha256. A failed update is recovered by re-running the install;
  4. POST /api/setup/installed (board key) with files: relay, then every other path in the update report's written (once, in report order), then the pointer file, .relay/runner.json, .relay/prepare-worktree.sh and .gitignore. That finishes step 4 and activates step 5; the script then prints Next: open <cli> here and run /relay-onboard.

The key travels only in the Authorization header and is never written to a file. Both /api/setup/* endpoints refuse a key from another board (403 wrong_board); preflight also refuses a board still on step 2 (409 step_2_pending). A re-run is safe: steps already past are left alone and nothing is duplicated. A board with no checklist still gets the install (its LLM is the default agent's), with no checklist writes. RelayWeb.ApiAuth assigns :current_api_key (from Relay.ApiKeys.authenticate_key/1) for this. Every /api/setup/* route works the key's own setup (Relay.Setup.target_for_key/1, RE446): a key minted by Connect a machine reports into that machine's checklist (step 3 reads "minted for <name> · first authed call …"), never the board's, and a board key never touches a machine's.

Onboarding reports (RE445). After the install, step 5 is worked by the /relay-onboard skill through two more board-key routes on RelayWeb.Api.SetupController, rendered by RelayWeb.Api.SetupJSON: GET /api/setup answers {"data": checklist} — the board name, connect (true for a connected machine's checklist, RE446: the skill then does 5.1 and a pull-only 5.6 check), the template, the onboarding LLM (llm: cli, harness, label, pointer_file, skill_dir, skill_path — pinned as setup.llm_keys in the runner contract fixture, beside RE457's setup.pointer_line and setup.runner_config_defaults), current_id / current_sub_id / current (Relay.Setup.reportable_id/1, the one step a report may move), first_flow_stage, first_card_stage (the column that flow pulls from, where 5.8's first card goes), the position counts, the 5.3 interview catalog, the 5.4 change_decisions and missing_file_resolutions (Relay.Setup.missing_file_resolutions/0), referenced_files (RE447: the .claude/ agent and skill files the board's flows name, with their repo paths and used_by {flow, node} pairs — computed in the controller from Relay.Flows.referenced_files/1 over Flows.list_flows/1, [] on a template board; the skill lists the missing ones at 5.1 and resolves each at the start of 5.4, reporting missing files k of N resolved), and every step (string statuses and actors, ISO 8601 datetimes, step 5's sub-steps in children). POST /api/setup/steps/:step takes {"status": "waiting"|"done"|"failed"|"skipped", "evidence"?, "error"?, "data"?} and calls Relay.Setup.advance/4, answering the same body. Refusals: 404 no_checklist ("no guided setup on this board"), 409 not_current with the current step in error.current, 422 invalid for a bad body. The board also ticks 5.5 (every flow pushed through PUT /api/flows/:key and enabled) and 5.8 (a card created in first_card_stage, or in the first flow stage) itself. The wire shape is pinned by the "setup" section of test/fixtures/runner_contract.json.

The skill reaches those routes only through two ./relay verbs (v91+). ./relay setup status calls GET /api/setup and prints the checklist (--json / --field give it raw). ./relay setup report STEP --status S [--evidence] [--error] [--data JSON] calls POST /api/setup/steps/STEP with only the fields given. --status choices are SETUP_REPORT_STATUSES, pinned equal to Relay.Setup.advance_statuses/0 by the contract fixture. --data must be a JSON object, checked before the request. Exit codes: 0 success; 3 (SETUP_NO_CHECKLIST_EXIT, used by nothing else) on 404 no_checklist from either route — the skill's signal to stop and point at /relay-doctor; 1 on 409 not_current, with a stderr line naming not_current and the server's current step, and on any other API failure.

The version is derived, never maintained. Relay.Scaffold.version/1 is the first 12 hex characters of the sha256 of the sorted "<path>:<sha256>" lines, so it changes exactly when content changes and cannot be forgotten. The client only compares it for equality; "N versions behind" is deliberately unsupported.

mix relay.build_scaffold writes priv/scaffold/ (a gitignored build artifact) — from mix setup, from the test alias, and from the Dockerfile after mix compile and before mix release, because a release ships priv/ but ships neither relay nor .claude/.

Consequence, accepted deliberately: publishing is coupled to deploying. A skill fix reaches projects only when the app ships. In exchange, the whole publish/marker/drift apparatus is gone.

./relay update is the one mechanism for getting those files onto disk. The work list is the verdict: every item is hashed against the manifest on every run, and "current" means nothing needs writing — never a version comparison. That is what makes a deleted or edited file come back, and what keeps the RE185 steady state honest, since auto-update rewrites ./relay in place without touching .relay/scaffold.json, so the marker legitimately lags the bytes. Applying with an empty work list reconciles the marker; --check reports and writes nothing, ever; --json on either. The runner rewrites itself through RE185's verified installer (verify_runner_source + install_runner's atomic os.replace and write ledger), and every body is additionally checked against the manifest's sha256 before it touches disk.

relay update also removes what an older layout left behind (RE319): a bin/relay that declares the pre-rename version constant is Relay's, so after installing ./relay the update deletes it (and bin/ when that leaves it empty) and reports it as removed (--check reports it as obsolete and writes nothing). A bin/relay without that line is someone else's and is never touched.

An old bin/relay running the upgrade installs ./relay without the executable bit: its installer gives mode 755 only to its own path and writes every other item at 644. The one-time step is chmod +x relay && ./relay update, which also removes the leftover, and the commit must carry relay as mode 755. From then on the new CLI keeps it that way: scaffold_status counts a ./relay whose bytes match but which is not executable as changed, so relay update reinstalls it through install_runner and restores 0o755.

Node-job transport (RLY-134, ADR 0006 card 04)

The first slice of ADR 0006's target shape: a pure REST transport on top of the runs engine (W5, Relay.Runs), board-key auth like the rest of /api, no scheduling/dispatch policy — that stays server-side.

  • POST /api/node-jobs/claim (RelayWeb.Api.NodeJobController.claim/2) — upserts the advertising runner (a claim doubles as a liveness touch, via Relay.Runs.upsert_runner/2) then atomically claims the oldest eligible queued NodeJob (Relay.Runs.claim_next_job/1, SELECT … FOR UPDATE SKIP LOCKED). Long-polls up to ~25s on the board:<id>:runs topic when nothing is immediately claimable (?wait=0 short-polls instead); serialises the raw run + resolved vars W5 already stored, never a worktree path. Eligibility is three-way (ADR 0006 §5, RE311): a job is offered when it is pinned to the requesting runner (runner_name = its name), OR when that runner declares it holds that card's worktree (the job is exclusive and its vars.ref appears in the claim's held with a state in Schemas.Runner.active_holding_states/0), OR when it fits the advertised free capacity the claim carries. The middle clause exists because the pin is only a proxy for the holding and the proxy can go missing: settle_retry_pin/3's :readopted branch releases a pin to a machine that is provably not answering (RE297), the job is inserted unpinned, and the machine then returns and adopts <ns>-<ref> — consuming the very slot the unpinned job needed to be offered through. The bypass is exclusive-only: a shared_clean job runs in the shared worktree and must still respect shared capacity. Pinning is persisted on the run: runs.pinned_runner_name is set when a runner claims an exclusive run's job (Relay.Runs.maybe_pin_run/2), kept through a :runner_gone park (so the resume returns to the holder), and cleared by a human-baton park (Relay.Runs.park_claimed/1, so the hand-back resume re-offers anywhere). Relay.Runs.exclusive_holder/2 reads that column to pin each successive job, and Relay.Runs.active_runs/1 resolves it to the runner row id so the scheduler resumes a parked exclusive run on its holder (RLY-199) — one column, two readers. The runner's name is therefore a durable, run-affecting key, not a label: renaming a running runner strands every run pinned to the old name (the resume targets a row nothing beats again, and retry refuses it as "not connected") because runner rows are never pruned. RE305 changed the default name from the bare hostname to <checkout-dir>@<short-host>, so a defaulted runner warns loudly at startup and names the legacy identity — see the Config bullet below.

  • Setup side effects (RE446). Both endpoints also drive the board's guided-setup checklist (Relay.Setup): heartbeat/2 calls Setup.observe_heartbeat/2 with the authed key and the post-upsert runner, so a board waiting on step 6 ticks it on the first heartbeat (machine and per-harness sign-in as the evidence) — or, for a connected machine's key, that machine's step 6 (recording its host / runner_name), never the board's; a granted claim calls Setup.observe_claim/2 from Relay.Runs.claim_next_job/4 after the FOR UPDATE SKIP LOCKED transaction commits, so a board waiting on step 7 finishes setup on its first claim — a setup write never rolls a claim back and the setup board lock never nests in the job lock. Both are no-ops returning :ok on every other board (one exists? query), so neither response changes. (A parked run whose holder advertises exclusive: 0 can still be handed its own resume — the runner keeps polling while it holds bound slots via RunnerPool.has_bound_slots/0.)

  • A process that predates the rename is refused, never served (RE319). The wire's identity key is runner, renamed as a hard cut with no alias. A claim or heartbeat whose body has no runner object but carries the pre-rename identity key gets 409 runner_outdated — the same code a merely old runner gets, pinned in test/fixtures/runner_contract.json (claim_refused.outdated) — with required: Relay.Runs.min_runner_version/0 and a message telling the operator to install ./relay (relay update) and restart with ./relay start. Nothing is written for it. The check is one private function in RelayWeb.Api.NodeJobController; a body that carries runner is current whatever else it carries.

  • Version negotiation (RLY-184). Every claim and heartbeat carries runner.version, the RUNNER_VERSION the running ./relay declares. claim/2 compares it against Relay.Runs.min_runner_version/0 and answers 409 runner_outdated (with required and running) instead of handing out work — claim is the only call that dispenses jobs, so that is the load-bearing check. nil counts as outdated: a runner sending no version predates the card. heartbeat/2 deliberately still succeeds for an outdated runner — the beat is how it stays on the roster and how revokes still reach it — and its reply carries runner_outdated / required_version. A refused runner stays alive, advertises {"shared_clean": 0, "exclusive": 0} so nothing queues behind it, finishes in-flight work, and wears an OUTDATED badge on the runners view until a human restarts it. Auto-update (RE185). The heartbeat reply also carries latest_runner_version, and a runner with auto_update on downloads that version from the board's /api/scaffold (RE304) and re-execs at a job boundary, so being refused is normally self-healing. The fail-stop below is unchanged and remains the floor: when auto-update is off, refused, or does not take, the runner still stops loudly.

  • Paused plan (RE460). When billing is on and the board's owner's plan is paused, every board-key /api/* call — claim and heartbeat included — is refused 402 plan_paused (RelayWeb.PlanPaused, sent by RelayWeb.Plugs.RequireActivePlan), and the board's Scheduler.Server dispatches nothing. ./relay mirrors the code as PLAN_PAUSED_CODE (pinned by claim_refused.plan_paused in the contract fixture) and never retries the 402. A one-shot command exits 1 with relay: This board's plan is paused. Manage billing at <board>/account/billing. relay start claims with soft_paused=True: it logs one PAUSED: … line on the transition, waits poll_timeout between claims (the 402 answers instantly, so this is what keeps it from hot-looping), and logs plan active again — claiming resumed on the first claim reply that isn't paused. The heartbeat also posts with soft_paused=True, so a refused beat counts as landed and starts no failure streak.

  • POST /api/node-jobs/:id/outcome (.outcome/2) — Relay.Runs.get_claimed_job/2 (board- scoped) returns a three-way result: a claimed job runs Relay.Runs.report_outcome/2 against the closed outcome set (422 unknown_outcome on a bad value); an already-finalized (:done) job is first-writer-wins — 200 with the run's recorded run_state, ignoring the resent payload, so a retried outcome POST after a dropped response never turns finished work into a failure (RLY-202); and only a :queued (reassigned) or :revoked (zombie) job answers 409 conflict. A run whose server can't start answers 503 run_server_unavailable: the outcome is not recorded, the job stays claimed and the run :running, and the runner's idempotent retry re-POSTs it (RE412). The runner's report_outcome is the one API call with a retry budget (RE449): after the usual API_MAX_ATTEMPTS it keeps retrying a transient failure (a connection error, a read timeout or a 5xx — _retryable) with exponential backoff capped at OUTCOME_BACKOFF_CAP_S for up to OUTCOME_RETRY_BUDGET_S (10 min), logging one line on entering that stretch and one on leaving it, then gives up as before. Throughout, the worker stays in in_flight, so the heartbeat keeps listing the job in running and the server doesn't reclaim it; a 4xx (the 409 above, a 422) still fails at once, and every other call keeps the API_MAX_ATTEMPTS bound. A job of a run that is not :running is refused like a revoked one — except a fan-out sibling's while another branch waits on a human (Runs.report_accepted?/2, RE437). The body's optional resume_at (RE267, ISO-8601 UTC) is read only with outcome blocked — an unparseable value is dropped to nil, never refused — and makes the engine requeue the node until that reset instead of parking it (failures.md A11b). The node outcomes (including the runner-only blocked, RE308) and what each does to the run and the card are tabulated in the state reference.

  • Harness routing (RE433). Relay.Runs.build_payload/4 resolves every agent node's llm (nil = the board default) to a board agent with Relay.Agents.resolve/2, and the job payload — hence the node claim — carries top-level harness ({key, command, resume_command, session_id_path}, the template the runner expands), model (the agent's model) and effort (the node's raw value; the runner drops it for a template without {effort}). All three are null on shell/gate nodes. Relay.Runs.insert_job!/3 copies harness.key onto node_jobs.harness_key, so the RE267 requeue copy keeps it. A red agent (Relay.Agents.red?/1) or an llm naming no agent on the board still gets a job, but its payload carries refusal (Relay.Agents.red_agent_detail/1 / unknown_agent_detail/2): RunServer's one dispatch seam never announces it and records the attempt :blocked with that detail and no resume_at, so the run parks, the card goes needs_input with the refusal on its timeline, and no retry is spent. refusal is never sent on a claim. The claim filter: a job with a harness_key is offered only to a runner whose reported runners.harnesses lists that key installed: true (Schemas.Runner.installed_harness_keys/1; a runner that never reported is treated as having only Relay.Agents.default_harness_key/0, claude-code), and a job whose payload has refusal is offered to nobody — both apply to pinned jobs too. Shell/gate/talk jobs have no harness_key and pass. Inventory + definitions: the heartbeat may carry top-level harnesses — [{key, installed, version, signed_in, usage}], normalized by Schemas.Runner.normalize_harnesses/1 (malformed entries dropped; a malformed usage degrades to nil, see Usage limits) and absent-means-untouched like held — and its reply always carries harnesses_digest (Relay.Agents.harnesses_digest/1, SHA-256 of the board's definitions in wire shape, free of ids and timestamps). GET /api/harnesses (RelayWeb.Api.HarnessController, board key) returns {harnesses: [Relay.Agents.harness_wire/1 …], digest}; the runner refetches it when the digest moves. The claim's harness set (RE436): a claim may carry harnesses, the keys the runner will run now (installed minus paused at a usage limit), normalized by Schemas.Runner.normalize_claim_harnesses/1 and passed to Relay.Runs.claim_next_job/4, which narrows the installed keys to that intersection. Like the claim's held it is request-scoped and never written to the row; an absent one leaves the claim unnarrowed. Each harness carries its usage rule (RE436) as usage — Schemas.HarnessUsage.wire/1: {source, match, command, interval_min, used_unit, resets_unit, windows: [{label, used_path, resets_path, pause_at}], pause_now: {when, resets_path} | null, recheck: {model, interval_min} | null}, source one of Schemas.HarnessUsage.sources/0 (output / command / none). Paths and conditions are a jq subset evaluated by Schemas.HarnessUsage.Rule and, line for line, by ./relay's usage_reading and friends; both pass the contract fixture's usage_rule_cases. Seeds: Claude Code reads its stream's rate_limit_event (5h / 7d), Codex polls its newest rollout file's token_count line (5h / wk), Gemini CLI is none; the AddUsageToHarnesses migration backfilled existing boards' seeded harnesses from a frozen copy. The AddHarnessRouting migration pointed already-queued/claimed agent jobs at a frozen Claude Code harness with model opus.

  • Talk rides the same claim, a different transport (RE268 / ADR 0009). Every POST /api/node-jobs/claim reply now carries kind ("node" or "talk"), so the runner can branch without a second endpoint. A "talk" claim carries exactly {id, kind, ref, turn_id, prompt, author, branch, seed, resume_session} — RelayWeb.Api.NodeJobController.claim_payload/1's talk-only clause — never the flow shape's run_id/node_id/vars. A talk turn's outcome does not go through POST /api/node-jobs/:id/outcome: that route finalises a job through the run lifecycle via Relay.Runs.get_claimed_job/2, which is flow-only (kind in NodeJob.flow_kinds()) and 404s a talk job on purpose. Instead RelayWeb.Api.TalkController, board-scoped by the same board-key auth, offers two routes: POST /api/talk/turns/:id/events appends a batch of transcript lines (at-least-once — a replayed client_seq is accepted and stored once, per Relay.Talk.append_events/2) and POST /api/talk/turns/:id/outcome ends the turn (done/stopped/failed, 422 unknown_status on anything else) via Relay.Talk.finish_turn/3.

  • The wire contract is pinned by a fixture. test/fixtures/runner_contract.json is generated from these routes by test/relay_web/controllers/api/runner_contract_test.exs (never hand-edited) and read by bin/test_relay.py, so both suites assert against one shape instead of each side's idea of the other (RLY-176). Renaming a claim field, or changing what the outcome/heartbeat bodies carry, breaks CI on the next run. Regenerate with RELAY_WRITE_CONTRACT_FIXTURE=1 mix test test/relay_web/controllers/api/runner_contract_test.exs, which rewrites the file and still fails so the diff gets reviewed.

  • Runner heartbeat superset. BoardController.heartbeat/2's /api/board/heartbeat route carries an independent, additive branch: a beat carrying name + capacity calls Relay.Runs.upsert_runner/2, writing/refreshing a durable Schemas.Runner row ({board_id, name}, capacity map, last_heartbeat) — the durable row is what feeds the Runners view (RLY-167). A capacity-less beat never touches the Runner table.

  • Capability inventory (RLY-182). The runner heartbeat may carry an optional capabilities payload — {"agents": [...], "skills": [...]}, the names this machine can actually resolve from its repo .claude/, the user-level ~/.claude/, and the CLI's built-in agents. ./relay's collect_capabilities() enumerates BOTH skills/<name>/SKILL.md and commands/<name>.md, because a slash command can live in either (/write-plan lives only in commands/). It rides send-on-change, not every beat: the runner hashes the inventory each beat and includes the key only when the hash differs from the last successfully-acknowledged send, so a failed POST retries on the next beat. The server persists it on runners.capabilities, where null means never reported and is deliberately distinct from {} (reported, and empty). Because a beat that omits the key must never erase a stored value, Relay.Runs.upsert_runner/2 builds its on_conflict replace list dynamically. For the case where the server genuinely has none (recreated row, or a runner predating this change), the heartbeat response carries want_capabilities: true; the runner clears its cached hash and resends on the next beat. Relay.Runs.preflight_flow/1 reads the stored inventory.

  • Runner liveness + reclaim. Relay.Runs.RunnerReaper (supervised, see runtime.md) periodically calls Relay.Runs.reclaim_stale_runners/0. A runner that goes quiet is not the same as a runner that is gone, so the sweep applies three liveness windows (RE449), each defined once on Relay.Runs and measured from the runner's last_heartbeat:

    Window Length Effect once silent past it
    stale threshold (Relay.Runs.runner_stale?/2) max(60s, 2 × interval) the runner stops counting: freshness :gone, its Relay.Runs.Capacity entry is evicted (Capacity.clear/1), it is not offered work. No job is touched.
    Relay.Runs.shared_reclaim_after_s/0 3 min its in-flight shared_clean jobs go back to queued, unpinned, so another runner can take them
    Relay.Runs.exclusive_reclaim_after_s/0 10 min its exclusive runs are parked (Relay.Runs.park_for_reclaim/1, parked_reason: :runner_gone) and their jobs revoked — never requeued, since an exclusive run is pinned to one runner's worktree

    Each reclaim window is applied as max(window, the runner's stale threshold), so a long-interval runner is never reclaimed before it is even stale; every comparison is a strict >. Shared work is reclaimed sooner because another runner can pick it up; an exclusive run can only resume on the same machine, so it waits longer. The sweep stays idempotent (its active-state query skips work an earlier sweep already reclaimed).

    Reattach for free. Inside a window the run stays :running and the job :claimed by that runner, so a runner back from a network blip simply resumes beating: its heartbeat lists the job in running and nothing happens — no new attempt, no second execution, the agent is never killed. A runner that restarted during the outage and lost the job in-process is handled by the RLY-170 orphan path below (Relay.Runs.requeue_orphaned_jobs/3): its first beat omits the job, so the job is requeued — pinned back to that runner when exclusive, unpinned when shared.

    "Is it live?" checks (working_run_ids/2, the roster's freshness, preflight, the scheduler's counting filter) all stay on the stale threshold. A :gone runner's advertised capacity is dropped from the scheduler snapshot (Scheduler.Server.build_snapshot/2, which keeps only this board's Relay.Runs.counting_runner?/1 runners — RE338) — after reserve_active_runs has debited its still-:running held runs — so the planner never resumes a pinned run onto a machine that has stopped counting; without this a parked exclusive run oscillates resume↔reap forever and relay why misreports it as "dispatchable" (RLY-199). The same reaper tick also calls Relay.Runs.close_orphaned_runs/0 — a companion sweep, not a runner-liveness check — closing any run still active while its card already sits in a terminal-type stage (RLY-233) or has been archived (RE335 — the board's Archive button is unguarded, and an archived card's parked run would otherwise hold its exclusive worktree forever). This is safe to treat as an unambiguous leak because run dispatch (Relay.Runs.start_run/3) now moves the card into the flow's work lane and inserts the run row in one transaction: no committed state ever has an active run sitting on a terminal-type stage except a genuine leak. The tick's third sweep, Relay.Runs.abandon_unresumable_runs/0, applies the clock to the refusals the scheduler recorded: a parked run whose resume has been refused continuously for Relay.Runs.unresumable_after_s/0 (30 minutes) is failed, so a run the planner keeps allowing but can never place becomes a visible failure a human can retry rather than a silent forever-wait (RE297).

  • Log node_job_id convergence. POST /api/board/logs entries may carry an optional node_job_id alongside run_id — same nullable-string shape, not an FK. It rides through Relay.AgentLog.stamp/1 → Relay.Activity.LogSink.row/2 → activities.node_job_id, kept for W6's run panel to key log lines off a specific node-job.

  • The full outcome-file contract (RELAY_NODE_OUTCOME) runners must honor is Declaring an outcome below.

The foreach cursor (RE252)

A foreach node (foreach: card.tasks) iterates the card's tasks — the sub_tasks rows — and every execution under it carries the id of the task its iteration is bound to (node_executions.sub_task_id). The job payload hands that binding to the node as the vars task (title) and task_id (id), which a prompt references as {task} / {task_id}. These are the canonical flow-contract names since RE367: the legacy spellings card.sub_tasks, sub_tasks (in reads/writes) and {sub_task} / {sub_task_id} are rewritten by Schemas.Flow.Node.normalize_legacy/1 wherever a flow enters (Schemas.Flow.Node.changeset/2, Relay.Flows.Document.decode/1), and the RE367 data migration rewrote every stored flows / flow_versions row, so the engine only ever sees canonical names.

"Which task is next" is derived, never persisted: Relay.Runs.next_sub_task_id/1 returns the first sub_task in position order whose done is false, so a crashed-and-resumed run recomputes the same answer with no cursor column.

The card drawer reads the binding, not the cursor: Relay.Runs.in_flight_sub_task_id/1 returns current_sub_task_id/2 for the foreach node of the card's active run (nil with no active run, no flow row, or no foreach node), and RelayWeb.BoardLive recomputes it on drawer open and on every card/run refresh to auto-open that task with an AGENT IS HERE chip (RE356). It is a read; no engine path calls it.

Those sub_tasks rows are also addressable one at a time (RE355): GET /api/cards/:ref/tasks (summaries — id, title, done, position, never body), GET /api/cards/:ref/tasks/:id (adds body), POST /api/cards/:ref/tasks ({"tasks": [{title, body}, …]} — one atomic append after the last task, argument order), PATCH /api/cards/:ref/tasks/:id (title / body only) and DELETE /api/cards/:ref/tasks/:id (later tasks move up one), wrapped by relay tasks add|list and relay task show|update|rm. None of them replaces the set, so a surviving row keeps its id — and every node_executions.sub_task_id bound to it. done is not writable there; the toggle route below owns it. Nothing in the engine calls these routes yet: Relay.Runs.PlanTasks still seeds the rows at run start.

But done is not a private engine field. Three writers set it: the loop tail's check-off (RunServer.check_off_sub_task/3), the card drawer's checkbox, and relay check <ref> <id> (PATCH /api/cards/:ref/sub-tasks/:id) — which any agent in any node can call. So the derived cursor advances on exactly one thing: a when: :foreach_remaining edge, the flow author's explicit "next iteration". RunServer.binding_for/5 keys off the guard the engine reports (Engine.decide/4 returns {:transition, target, guard}), never off the target node:

Route into the next node Binding
when: :foreach_remaining edge the derived cursor — the loop tail already checked the finished task off inside the same transaction
when: :foreach_exhausted edge nil — the run has left the loop
unguarded edge into the foreach head inherit, falling back to the derived cursor when unbound (first entry from outside the loop, e.g. branch → implement)
anything else, including a {:retry, node} or a {:requeue, node} (RE267) inherit

The third row is the one that matters. A review's failure loop-back (spec_review/quality_review --failed--> implement) is not an advance: it re-enters the same iteration with the reviewer's detail attached as findings, whatever any other writer did to done in the meantime. Keying off the target node instead — as the engine did before RE252 — made a failed review re-derive the cursor, so with the in-flight task already checked off the re-run landed on the NEXT task, that task's findings were delivered to nothing, and the run marched on to final_review and smoke with the branch looking clean.

done stays writable while a run is live, deliberately: a human un-checking a box mid-run is a legitimate escape hatch. The residual risk is confined to the success path, where an external write can make Relay.Runs.remaining_sub_tasks/1 under-count and exit the loop early. The failure path never consults done at all.

Fan-out regions (RE437)

A fan_out node fires every outgoing edge at once; each edge starts a branch, and the paired fan_in (Schemas.Flow.regions/1) waits for all of them. Both are control nodes (Schemas.Flow.Node.control_types/0): RunServer resolves them as node_executions rows with an immediate outcome and no job.

  • Entering a region — a {:transition, fan_out, _}, or a start edge into one (Runs.start_run/3, which then starts the server with {:dispatch, job_ids}) — records the fan_out :succeeded, inserts one run_branches row per branch (Schemas.RunBranch, keyed by that fan_out execution and the branch key, :queued) and starts the first max_parallel by position. run.current_node stays the fan_out key while the region is open.
  • Starting a branch sets its row :running and enqueues the branch key's node with node_executions.run_branch_id set. Its prior_detail / findings come from the last outcome-bearing non-branch execution before the fan_out, re-derived from history, so a branch started later from the queue gets the same vars.
  • Inside a branch, Engine.decide/4 sees only that branch's history, so max_retries, max_loops, the visit cap and the failure-signature breaker count per branch (the retry findings and commit-guard baselines scope the same way). A transition to the fan_in ends the row with the node's outcome; a {:fail, _} ends it :failed with one :failure timeline entry, and the run carries on. Each ended row frees a slot for the next :queued one; a queued branch the live flow no longer has ends :failed (branch removed from the flow).
  • Joining — once every row is :ended — records the fan_in with Engine.join/1's result: :succeeded iff every branch succeeded, and one ## <branch>: <outcome> section per branch. A fan_in carrying foreach guards is the loop tail and checks the task off. It is then decided and routed like any other execution, so {prior_detail} / {findings} reach the next node.
  • Several active jobs. A run inside a region holds one job per running branch (Runs.active_jobs/1). Claiming an exclusive run's first job pins the run and its queued siblings to that runner. cancel_run/2 and close_run!/3 end every open branch (Runs.end_open_branches!/1); Runs.open_region/1 returns the latest entry's rows while any is open. diagnose/3 reports the first active job plus active_job_count.
  • Parking with siblings. A branch that parks (a question, an escalation edge, a :blocked) sets its row :parked and parks the run (park_effects/2 — card blocked, parked/needs_input), but revokes no sibling job. While the run is parked, Runs.report_accepted?/2 still takes a report for a job in a :running branch: it is decided and routed inside that branch (a next node's job is enqueued and dispatched, an ending ends the row, a park parks it — the card is only re-blocked, never asked twice), but no :queued branch starts and no join happens. RunServer starts for such a run only in :attach (Runs.attachable_while_parked?/1) and stops after each report, so a parked run holds no process; Runs.resume_run/2 also stops any lingering server before its parked → running transition.
  • Re-entry inside an open region. Every re-entry mode ({:reenter, _}, --at, advance) with Runs.open_region/1 non-empty goes branch by branch in one transaction: a :running row with an active job is left alone; a :running row without one (restart, hand-back, lost runner) and every :parked row get a fresh attempt of the same visit with the branch's own last failure as findings — a parked row resuming its own :needs_input session, never the run-level one — then the region is scheduled. Jobs are revoked per row, never run-wide. The RE410 outcome-less cap counts per branch and parks just that branch (and the run). Runs.boot_mode/1 always re-enters a run with an open region. With no region open, re-entering at a fan_out, a branch node or a fan_in opens a fresh region at the fan_out; Runs.retry_run/2 maps such a target to the fan_out itself.

Branches share the run's checkout: concurrent edits, commits and builds are the flow author's responsibility — Relay neither guards nor forbids them.

Run recovery surface (RLY-189)

A terminally failed run can be re-entered by a human — the branch, worktree, execution history and runner pin all survive, because retry revives the dead run in place rather than starting a new one. Re-entry (RunServer.handle_continue({:reenter, _})) never consults the flow's start edge, so the Code flow's destructive branch node is unreachable from a retry by construction, and finished commits cannot be thrown away.

Re-adoption (RE297). Deleting a flow nils runs.flow_id on every run of that row, and the reaper now fails such a run outright (D6 in failures.md) — so retry is the hatch a human reaches for on exactly that shape, and refusing it no_flow while the board plainly shows a flow working the card's stage was the dead end's last link. When flow_id is nil, retry re-adopts the enabled flow whose work lane is the card's current stage (Relay.Flows.working_flow/1 — the same lookup rejection re-entry uses) and re-enters at that flow's start node, since the node this run died on need not exist in a replacement graph. Two things follow only in this case: the run's flow_id/flow_key are rewritten to the adopted flow, and a pin to a dead runner is released rather than refused on (settle_retry_pin/3) — honouring it would revive the run straight back into pinned_runner_absent, refused every tick until the reaper failed it again. A pin whose machine is alive is kept: the worktree really is still there. no_flow survives for the genuinely unresolvable case — no enabled flow works the card's stage, so there is nothing to re-enter.

  • POST /api/runs/:id/retry (RelayWeb.Api.RunController.retry/2) — id-addressed.
  • POST /api/cards/:ref/retry (.retry_card/2) — ref-addressed alias resolving the card's most recent run; what relay retry <ref> calls. Both take an optional {"at": "<node_key>"} body and funnel into Relay.Runs.retry_run/2.
  • POST /api/board/restart-stalled (RelayWeb.Api.RunController.restart_stalled/2) — bulk-revives every restartable run on the token's board (RLY-228), returning 200 {"data": {"status": "ok", "restarted", "refused"}}. Board-scoped by the bearer token. Cards already in a terminal-type stage (Relay.Schemas.Stage.terminal_types/0) are excluded before this runs — a finished card is not stalled (RE247) — and are not counted refused.

Success is 200 {"data": {"status": "ok", "run_id", "node", "retries"}}. A refusal is 422 {"error": {"code", "message"}} where code is one of not_failed, awaiting_answer, active_run_exists, no_flow, unknown_node, runner_unavailable — the message names the specific status, node key or runner that blocked it. An unknown run/card, another board's run, or a card that has never run is 404. RLY-228 widened retry to an escalation park — a :parked/needs_input run that Relay.Runs.park_kind/1 classifies :escalation (its latest execution reported something other than :needs_input, so a node failure was routed to a human, not a question asked of one). A :question park is refused awaiting_answer instead. See "Telling A1 from A4" in failures.md for the classifier — park_kind/1 is the one place that distinction lives; do not re-derive it here.

A successful retry also clears the card's block: revive_run/4 calls clear_card_block/2, so a card sitting :failed or :needs_input behind the retried run returns to its unblocked status and the board stops showing a blocked card whose run is already live again.

The guard is split, because worktrees and branches are runner-side state Phoenix cannot see. Server-side, the endpoint refuses up front for the six reasons above — including an exclusive run whose pinned runner is absent or stale per Relay.Runs.runner_stale?/2, whose worktree is unreachable. Runner-side, branch existence stays with RLY-166's check_branch_attached and RLY-173's reattach_branch; a retried job whose branch was deleted fails there with a clear message. Neither half ever silently restarts from origin/main.

CLI: relay retry <ref> [--at NODE] [--json]. On a refusal it prints the server's message to stderr and exits non-zero.

Cancelling a run (RE309)

The stop half of the same surface. Relay.Runs.cancel_run/2 stops the run server, revokes any in-flight job (freeing the runner slot), transitions the run from Relay.Schemas.Run.active_statuses/0 to :cancelled, logs an :action timeline entry and broadcasts :run_finished. Both running and parked runs cancel, with no extra flag — restricting it would leave a wedged running run needing a production console, which is the hole this exists to close.

  • POST /api/runs/:id/cancel (RelayWeb.Api.RunController.cancel/2) — id-addressed.
  • POST /api/cards/:ref/cancel (.cancel_card/2) — ref-addressed alias resolving the card's active run via Relay.Runs.active_run/1 (retry's alias resolves the newest run instead, whatever its status); what relay cancel <ref> calls. Both take an optional {"reason": "…"} body and funnel into Relay.Runs.cancel_run/2.

Success is 200 {"data": {"status": "ok", "run_id", "ref", "previous_status", "node"}}, where previous_status and node are the run's values before the cancel. The single refusal is 422 {"error": {"code": "no_active_run", "message"}}, covering both shapes of "nothing to cancel" — no run row at all, and a run that raced to terminal between lookup and cancel — because they are the same fact to the caller. A card with no active run therefore gets a named 422, never a silent success. An unknown ref or run id, or a run on another board, is 404.

A :reason is folded into the timeline text as run cancelled — <reason>, trimmed and capped at @max_cancel_reason (200) characters; omitting it keeps the plain run cancelled. The :actor (:agent | {:user, id}) is logged with the entry, so "who killed this run?" is answerable: BoardLive's confirm-move dialog passes the signed-in user, the board-key API passes :agent, and the run Listener and RunnerReaper take the :agent default.

Cancelling never moves the card. A caller that wants the card elsewhere follows with POST /api/cards/:ref/move, whose 409 would_strand_run keeps meaning exactly what it says. That two-call sequence — relay cancel REF && relay move REF Review — is the supported way to unstick a card whose run must die.

CLI: relay cancel <ref> [--reason TEXT] [--json]. On a refusal it prints the server's message to stderr and exits non-zero. Self-cancel is permitted and documented, not guarded: a flow node cancelling its own ref revokes its own job, and the runner handles revocation gracefully.

Advancing past an already-committed task (RE310)

A foreach node re-entered onto a task whose work is already committed has no exit: the commit guard rewrites its succeeded to failed (it cannot move HEAD — its own earlier commit IS the work), the failure edge loops it back or parks it, and no answer a human can type changes either. Relay.Runs.advance_foreach/2 is the hatch: check the bound sub-task off through Relay.Cards.set_sub_task_done/3 and revive the run at the foreach head re-bound to the next task, or — when that was the last one — at the flow's when: :foreach_exhausted target.

  • POST /api/runs/:id/advance (RelayWeb.Api.RunController.advance/2) — id-addressed.
  • POST /api/cards/:ref/advance (.advance_card/2) — ref-addressed alias; what relay advance <ref> calls. Same success and refusal shapes as retry, and the same board scoping (another board's run is a 404).

Eligibility is deliberately wider than retry's: a :failed run, or a :parked run with parked_reason: :needs_input — either park kind. RE306's actual state was a :question park, which retry refuses awaiting_answer, so reusing retry's rule would leave this hatch unable to open in the exact state it exists for. :running, :parked/:runner_gone, :done and :cancelled are refused. The refusal codes add not_advanceable, no_foreach, no_exhausted_edge and not_bound to retry's list, through the same Relay.Runs.retry_refusal_code/1 pair; every refusal is checked before the check-off, so a refused advance writes nothing.

The engine change is one thing: RunServer's re-entry takes a binding MODE. Every existing re-entry inherits the node's current binding (which is what keeps a review-failed loop-back on the SAME task, RE252); the new {:advance_foreach, _} start mode re-derives it from Runs.next_sub_task_id/1, which — the stale task having just been checked off — is the next one. It is the only caller allowed to re-derive it.

CLI: relay advance <ref> [--json]. UI: a run-advance button on the run panel's :failed, :circuit and :parked banners, rendered only when Relay.Runs.advance_foreach_available?/1 holds — the LiveView computes that predicate and the component reads a boolean.

Scheduler / Listener authority split (RLY-200)

Before either owner below gets a say, Relay.Runs.Listener.reconcile_card/2 runs a first, unconditional rule: a still-active run (running/parked, any parked_reason) that is a leak — its card already sits in a terminal-type stage (Schemas.Stage.terminal_types/0) or has been archived (RE335; the Listener reacts to {:card_archived, card}) — is closed via Relay.Runs.cancel_run/2 with the reason Relay.Runs.leak_reason/1 returns, not resumed (RLY-233). This is what stops a parked :needs_input run from being resumed after its card reached Done or was archived — closing pre-empts the resume rules below. An archived card with no active run triggers no rule at all, so it is never re-entered after a rejection. Relay.Runs.RunnerReaper's 30s sweep (Relay.Runs.close_orphaned_runs/0) is the companion catch-up for anything the event path missed. Run dispatch (Relay.Runs.start_run/3) moves the card into the flow's work lane and inserts the run row in one transaction, so no committed state pairs an active run with a terminal pull stage — and the leak itself is judged from a single run → card → stage snapshot (Relay.Runs.leak_reason/1), never a card stage read apart from the run, so a concurrent Spec:Done → Plan dispatch between two reads can't get a freshly-dispatched run cancelled (RLY-233 / RE239). No grace window is needed.

A parked run has exactly one process allowed to resume it, keyed off Schemas.Run's parked_reason:

parked_reason owner resume shape
:needs_input Relay.Runs.Listener same node, WITH the stored session (--resume)
:claimed Relay.Runs.Listener fresh (a human may have changed anything)
:runner_gone Relay.Runs.Scheduler capacity-driven re-dispatch
nil / unknown nobody left untouched (mirrors the Listener's own fallback)

Resuming is not the only way a park can end. Relay.Runs.RunnerReaper is the one process allowed to end a park without resuming it: when the scheduler has refused to resume a run continuously for Relay.Runs.unresumable_after_s/0 (30 minutes), Relay.Runs.abandon_unresumable_runs/1 fails it (RE297). That give-up path is scoped to the refusal clock — it never pre-empts the owner above, which still holds the resume itself.

Relay.Runs.Scheduler.resume_runs/5 filters on this before its existing human/needs-input guards, so it never even considers a Listener-owned park; Scheduler.explain/2 mirrors the same split, surfacing :awaiting_listener_resume instead of :awaiting_capacity for a Listener-owned park so relay why stays honest. Without this split, both processes reacted to the same card event and whichever won decided whether a resumed agent kept its Claude session — the loser silently re-wrote the run underneath the winner.

Every running ↔ parked writer (Relay.Runs.resume_run/2, park_for_reclaim/1, park_claimed/1) is a guarded Repo.update_all keyed on the run's current status (the same pattern transition_job/3 already used for job-state writes), so a lost race — whichever process loses — returns a detected no-op ({:error, :not_parked} for resume) rather than a second silent write.

Listener boot sweep. Relay.Runs.Listener.init/1 reconciles every card holding a :parked run at startup ({:continue, :boot_reconcile}), so an answer or hand-back that arrived while the Listener was down is not lost — the scheduler is no longer a backstop for :needs_input/:claimed parks the way it was before this split.

Runner mode (relay start) (RLY-135, ADR 0006 card 05)

./relay start is the only runner mode: a thin, board-agnostic client of the node-job transport above. It knows the Relay REST API and how to execute a node-job; nothing else — every board-specific fact lives server-side as flow data.

Every iteration of the loop is the same three steps:

  1. Claim the next node-job from the server (a long-poll — cheap when idle).
  2. Run it — an agent node runs its harness's CLI headless (Claude Code, Codex, … — RE433); shell/gate nodes run shell.
  3. Report the typed outcome back to the server, which advances the flow (moving the card to the next stage when the flow lands there).

Step 3 is a contract a node author has to honor — see Declaring an outcome below.

Agent steps run headless Claude, which uses whatever authentication the local Claude CLI has — a Claude subscription (no ANTHROPIC_API_KEY needed) or, if ANTHROPIC_API_KEY is set, the metered API. Subscription rate limits are the ceiling; when hit, the step is throttled, not silently billed to the paid API.

  • Config. .relay/runner.json holds name (defaults to <checkout-dir>@<short-host>, e.g. relay@Jeremys-MBP; override per invocation with relay start --name foo). Upgrading past RE305 changes that default, and because the name is the exclusive-affinity pin key (see Node-job transport above), runs pinned under the old bare-hostname identity park :runner_gone instead of resuming. The hop happens unattended — auto-update re-execs at a job boundary, which is "nothing in flight", not "no run pinned to me" — so a defaulted runner prints a WARNING: at startup naming the pre-RE305 identity. It is gated on evidence the identity actually moved on this machine: an identity lock file for the bare hostname (acquire_singleton_lock writes one and never unlinks it), so a fresh install that never ran a pre-RE305 runner starts silent. Because that lock is never unlinked the gate stays true forever once it is true, so emitting also drops a .re305-warned marker beside the new identity's lock: the line is said once per (machine, board, new identity), not on every start and every re-exec. Keyed on the new name because the legacy one is the bare hostname and so machine-global — keyed there, the first checkout to start would eat the only telling and leave the others (often the one actually holding the pins) silent. A --dry-run prints the line but does not spend it. To adopt those pins, restart the one checkout that owns them with --name <hostname> — per invocation, not as a "name" key in .relay/runner.json, which is tracked in git and shared by every checkout, so a bare hostname there restores exactly the shared identity this default exists to split apart. Per run the recovery depends on where the run already is: while it is still running, a human baton clears the pin (Runs.park_claimed/1 nils pinned_runner_name, and it transitions from [:running] only), so handing the baton back re-dispatches it anywhere; once it has parked :runner_gone nothing clears the pin in place — the listener leaves that shape untouched, the scheduler's resume still targets the gone runner, and relay retry refuses runner_unavailable — so the way out is cancelling the run and letting the card dispatch fresh, losing that run's worktree state. The board already says Runner "X" is not currently connected. on the parked run. Also namespace (default exec), capacity: {shared_clean, exclusive}, base, poll_timeout, heartbeat_interval, and three optional per-card-worktree keys (RLY-231): cache_dir (a warm dep/build cache dir passed to the lifecycle hooks), worktrees (RE339 — {"prepare": path, "cleanup": path}, the project's per-worktree lifecycle hooks, defaulting to .relay/prepare-worktree.sh and .relay/cleanup-worktree.sh; a flat top-level prepare is still read as a deprecated alias for worktrees.prepare, which wins when both are set), and max_retained_failed (how many failed-run worktrees to keep for post-mortem before the oldest is evicted, default 3), and two auto-update keys (RE185): auto_update (default true) and auto_update_min_interval (seconds between update attempts, default 300). Missing file → sensible defaults, including the auto-update keys; capacity is the field a developer routinely edits. The loader (load_runner_config) strips full-line // comments (first non-whitespace characters //) before parsing, keeping each stripped line as a blank line so JSON error line numbers match the file; trailing // and /* */ are not supported, and a string value containing // is never altered. ./relay update creates the file if missing (RE383, ensure_runner_config): namespace = the sanitized checkout-dir name (fallback project), cache_dir = ~/.cache/relay/<namespace>, starter capacity shared_clean 3 / exclusive 2, the worktrees block commented out (PAUSE AT is per harness, set in Settings → Agents, so there is no limits key — RE436), and no name key (a tracked name would recreate the RE305 identity collision). The startup line names namespace <ns>.

    base is the trunk every worktree is baselined to — the ref a worktree is created at, hard-reset to on re-baseline, refreshed to when the shared tree goes idle, and handed to the prepare hook as RELAY_BASE. Default origin/main; set it to origin/master, upstream/main, or a long-lived integration branch when that is your trunk. It does not govern where a card's branch starts — that start point is written into the flow's branch node (git checkout -B {branch} origin/main) and has to be changed in the flow.

    Running more than one exclusive slot? Concurrent runs each work in their own worktree, so make sure they don't share mutable state — most importantly, give each run its own test database (or equivalent) so parallel test suites don't truncate each other. How you do that depends on your project's toolchain (the prepare hook below is where a project wires per-worktree isolation).

  • What the terminal tells you (RE305). Startup prints ONE line naming the runner, its version, the board it reached (display name + key), the URL, and the capacity it advertises — so a key pointed at the wrong board is visible immediately instead of looking identical to a correct one. If the board cannot be reached at startup the line says board UNREACHABLE, a WARNING: line names the URL and the underlying error (exactly the diagnostic for a wrong or expired RELAY_API_KEY), and the runner keeps polling — a transient outage at startup must not kill a long-running process.

    In the loop it prints claimed <REF> · <node> (run <id>, <isolation>) (or claimed talk turn for <REF>) the moment work arrives, and two throttled idle — … lines for the paths that were silent: the board offering no work, and every slot being busy — the latter naming what holds them (RE291 retained is a failed run's worktree kept for post-mortem, which holds its exclusive slot until reclaimed). Both are capped at one line per reason per IDLE_LOG_INTERVAL (300s, a module constant, deliberately not a config key), and a claim re-arms them, so a quiet runner costs ~2 lines per 5 minutes.

  • Single-process guarantee (RLY-193). Exactly one relay start may run per {server, name} (the pair the server keys a Runner on, name defaulting to <checkout-dir>@<short-host> — RE305, so two checkouts of one project on one machine no longer collide on identity, provided their directories are named differently) and per worktree namespace. At startup cmd_start takes two exclusive, non-blocking fcntl.flock locks — an identity lock under $RELAY_RUNNER_LOCK_DIR or ~/.relay/locks keyed on sha256(RELAY_URL + "\0" + name) (since name embeds the checkout directory, two clones on one host in differently-named directories now hash to different lock paths — RE305; default_runner_name/0 uses the directory's BASENAME, so same-named directories still collide and the identity lock refuses the second one by name), and a namespace lock at <ROOT>/.claude/worktrees/.<namespace>.lock — held for the life of the process by keeping their fds open. A second process for a colliding identity or a shared worktree namespace refuses to start (relay: already running: …, naming the holder's pid) rather than registering as the same runner. Because flock is released by the kernel on process death, a crashed runner leaves no stale lock (this is why a flock and not a pidfile). This is what makes the RLY-170 orphan recovery above sound: that recovery requeues a job the runner no longer reports running, which is only correct because a single identity can no longer be split across two live processes each beating a partial running list. Two runners on one host are supported when they are different checkouts — a distinct name gives a distinct identity lock, and a distinct ROOT gives a distinct namespace lock; what remains unsupported is two runners sharing one checkout and name.

  • ./relay update [--check] [--json] — non-interactive, writes only the six Relay-owned files, needs no TTY and no board key.

  • Worktree namespace (RLY-231: one worktree per card). RunnerPool maps every job's isolation onto worktrees under the exec-* namespace. shared_clean jobs share one reused exec-clean worktree (never reset per-job, only fast-forwarded to base when every shared slot is idle) — unchanged. exclusive jobs no longer draw from a fixed exec-work-1..N pool; each card gets its own worktree named <ns>-<ref> (e.g. exec-RLY-231), created on the card's first exclusive job and torn down when its run reaches a terminal run_state. Since the worktree's identity is the card's branch, cross-contamination between two cards is impossible by construction, and the binding is derivable from git worktree list rather than an in-memory map, so a restart re-derives it (RunnerPool.recover/0) instead of losing it.

    • Capacity is reinterpreted, not reshaped: capacity.exclusive is max_worktrees — the max number of concurrent active per-card worktrees a runner holds, not a fixed slot count. Advertised free exclusive = max_worktrees − active_count. The runners view's used for the exclusive chip is now derived from the runner's declared holdings (runners.held, RE311) rather than from its active jobs — which is exactly total − free as RunnerPool.capacity() computes it, so the two sides agree by construction. Counting active jobs made a bound-but-idle, talk-attached or retained worktree invisible, and that is what reported "runner available" while the runner had zero free exclusive slots. The runners page also lists every holding in a per-runner HELD WORKTREES section (RE337, Relay.Runs.list_runner_status/2's holdings: card, state, the card's active run and how long it has sat), with Release worktree (idle bound holdings only — Relay.Runs.release_action/1; disabled for running/talk, hidden for retained) and Cancel run (Relay.Runs.cancel_run/2). Relay.Runs.starvation/2 drives an amber banner above the queue when a queued exclusive job has waited past @awaiting_slot_grace_s, no connected runner has a free exclusive slot, and every held exclusive slot is bound (no live job) — so nothing will free a slot on its own. A roster with any running holding is not starved. Release is manual only; there is no auto-release.
    • Two states. Active: bound to a non-terminal run, counts toward max_worktrees, holds a MIX_TEST_PARTITION index. Retained: a failed run's leftover kept on disk (marked with the gitignored .relay-retained sentinel at its root) for post-mortem, up to max_retained_failed (default 3, oldest evicted first by mtime past that), not counted against capacity. done/cancelled remove the worktree immediately; a revoke (run_state == nil) touches nothing.
    • Several node jobs per tree (RE437) — concurrent node jobs per run. A fan_out's branches are node jobs of ONE run, and all of them run at once in the card's one worktree (one exclusive slot, not one per branch). The record's live is therefore a live count of in-flight node jobs, not a flag: assign increments it on every exclusive placement, release decrements it, and the terminal disposition above (or its deferral under a talk turn) runs only when it reaches 0 — the last branch out decides, so the first branch to finish never tears the tree down under its siblings. holdings() reports running while it is above 0. Each job's board-log lines are attributed to that job, not to the card (per-job attribution, see Talk turns below), and the first job's preparation finishes before any sibling runs in the tree (the prep wait, under Prepare hook).
    • Shared-checkout hazards — the flow author's responsibility, not Relay's. Concurrent branches share everything the worktree has, and Relay guards none of it (RE437 decision 3): same-file edits race and the last write wins; concurrent git commit contend for .git/index.lock, so one fails; and there is one MIX_TEST_PARTITION and one _build per tree, so two branches running mix compile / mix test at once rebuild under each other and truncate the same test database. A region whose branches only read (a review panel) is safe; one whose branches write must split the files between them and leave committing and building to a single node after the fan_in.
    • Never-detach. Once the run's branch node attaches refs/heads/{branch}, the runner never re-detaches that worktree again — the old mid-run reset-on-revoke path (below) is gone. A revoked exclusive job now just stops its subprocess and leaves the worktree active and bound, ready for the pinned resume to continue in it.
    • Prepare hook. On a reset (first job of a card, or reclaiming a retained worktree for a new run), RunnerPool.create_or_rebaseline/1 makes the worktree clean at base, then run_prepare_hook/3 warms it: it runs the hook RunnerPool._hook_path("prepare") resolves — worktrees.prepare from runner.json, else the deprecated flat prepare, else .relay/prepare-worktree.sh — if that file exists and is executable, else it is a no-op (a cold build, not an error). The hook receives [worktree, ref, branch, base, cache_dir] as both argv and env (RELAY_WORKTREE/RELAY_REF/RELAY_BRANCH/RELAY_BASE/ RELAY_CACHE_DIR) with cwd set to the new worktree; a nonzero exit fails the run fail-fast (its stderr becomes the node's failure detail) rather than silently running against a half-warmed tree. Relay's own hook, .relay/prepare-worktree.sh, is a boring first cut: it copies deps/, _build/, and assets/node_modules/ from cache_dir (or else the main checkout) into the fresh worktree via APFS clonefile copy-on-write (cp -Rc, falling back to plain cp -R), so mix deps.get is a no-op and mix compile only rebuilds the diff. When a run's first job is a branch, several of its jobs are placed on a tree that does not exist yet: exactly one gets reset and prepares it, and the rest block in execute_one on the pool's per-slot preparation event (RunnerPool.wait_prepared/1) until it finishes. They then run in the prepared tree, or — when the hook failed — each report failed with the same prepare hook failed: … detail instead of running (RE437).
    • Cleanup hook (RE339). Right before the runner deletes an intact per-card worktree, run_cleanup_hook/2 runs the project's cleanup hook so it can tear down what prepare set up (a dev server, a docker-compose stack, a per-worktree test database, a port reservation). It resolves worktrees.cleanup from runner.json, else .relay/cleanup-worktree.sh, and is skipped when that file does not exist or is not executable. Relay ships none of its own, because deleting the tree already removes everything its prepare hook copies. It gets the prepare hook's contract: [worktree, ref, branch, base, cache_dir] as argv and RELAY_* env, cwd = the worktree (still on disk). branch is read from the tree and is empty when HEAD is detached. When: every removal of a tree whose .git link is intact. That covers a done/cancelled release, a talk-worktree release, retained-over-cap eviction, every release_held retry, and recover()'s reclaim of retired <ns>-work-N slots, where RELAY_REF is empty. It does not fire on retain (a failed run's post-mortem tree is cleaned up later, when it is evicted), on a half-deleted tree (its cleanup already ran, and a .git-less cwd would send git into the main checkout), or on a rebaseline/reset or shared_clean refresh, since those trees are reused. Order: cleanup hook → _stop_processes_in → stash salvage → git worktree remove, so the hook stops its own services gracefully and the RE336 sweep catches the rest. Best-effort: unlike prepare, a nonzero exit, or running past CLEANUP_HOOK_TIMEOUT (120s, not configurable, because _teardown holds RunnerPool.lock), is killed if still running, forwarded as an error line naming the card's ref, and the removal goes ahead. The run's outcome never changes. The hook must be idempotent: if it succeeds and the git worktree remove after it fails, the retry finds the link still intact and runs the hook again.
    • Shared .git. Worktrees never get their own clone; git worktree add/remove/prune routes through git_worktree_with_retry, the same bounded-retry discipline as git fetch (RLY-224 §6), since concurrent per-card creates/teardowns race on the one shared ref db.
    • A failed teardown is retried, not forgotten (RE336). done/cancelled pop a worktree's record and free its partition only once the tree is actually gone. If git worktree remove --force fails, the record stays active, live=False, bound to its run and holding its partition, and is flagged teardown_failed. holdings() therefore keeps reporting it bound, the server keeps naming it in release_held (its runs are all terminal), and RunnerPool.release_held/2 retries the removal on every heartbeat. Each failure is forwarded as an error line carrying the card's ref, throttled to one per IDLE_LOG_INTERVAL per ref, and a later success logs once. Retention eviction follows the same rule. A card whose teardown is failing gets reset=True on its next run or talk turn (assign_talk re-baselines it too, or waits if something still occupies it), never reuse-as-is, because the tree may be half-deleted. A half-deleted tree (its .git link gone) is removed with rm -rf + prune, and create_or_rebaseline deletes and re-adds one rather than running git inside it, since git would walk up into the main checkout. If that delete leaves the directory behind, it stops the tree's processes, retries once, and then dies. It never falls through to reset_worktree on a .git-less path. Previously the record was popped before removal, so a failed removal left the tree on disk but invisible to release_held until a runner restart (recover()) rediscovered it.
    • In-tree processes are stopped before removal (RE336). Before git worktree remove (on the removal path only, never for a retained failed tree, which is a post-mortem left as is), _stop_processes_in finds every process whose current working directory is the worktree or inside it. It uses /proc/*/cwd on Linux and lsof -a -d cwd -Fpcn elsewhere, and excludes the runner and its ancestors. It sends each one SIGTERM, waits up to TEARDOWN_KILL_GRACE_S (5s), then SIGKILLs any survivors, and forwards one line per process (pid, command, card ref). It kills by cwd rather than by process group so that a server a smoke/acceptance node started with nohup/&, which escaped the node's group, is still caught. That was the observed trigger: a leftover phx.server made git worktree remove --force fail with "Directory not empty". Discovery is best-effort: if the tool is missing or errors, the step is skipped and the retry-and-log above covers the failure.
  • Per-node scratch (RLY-214). Alongside the worktree itself, every node gets RELAY_NODE_SCRATCH (scratch_path in ./relay): tmp/<REF>/<node>.md inside that same worktree, keyed only on (ref, node) so a re-queued job after a runner restart resolves the identical path. It sits under the checkout's own .gitignore, so it survives reset_worktree's salvage/stash/clean untouched and never gets committed. See ../../relay.md for the full contract, including why agents must not invent their own scratch path (RLY-177).

  • Test database per slot (RLY-213). Worktree isolation keeps two concurrent runs' files apart, but mix test for both would otherwise hit the same Postgres database — Ecto's SQL sandbox only isolates concurrent tests within one BEAM, not across two OS processes. RunnerPool.partition_for(slot) (./relay) derives MIX_TEST_PARTITION from a free-list index held by the worktree's registry record at the single point where a node's command launches (both _stream_shell and _stream_agent_job), so every step of a run — including the precommit gate — sees the same database: each active per-card worktree (e.g. exec-RLY-231) holds its own index for the run's lifetime, recycled on teardown; the shared exec-clean is always partition 0. config/test.exs already keys the database name off MIX_TEST_PARTITION.

  • Harness command templates (RE433). An agent node runs whatever command the claim's harness defines; ./relay names no harness (RunnerHarnessGuardTest greps it for the seeded keys and names). select_command(harness, resume_session) picks resume_command only when the job carries a session and the harness has one — otherwise command and a fresh start (a harness that cannot resume never sees the old session). expand_command(template, values) is pure: it shlex.splits the template first, then substitutes inside each token, so each placeholder fills exactly one argv element, the prompt never passes through a shell, and a substituted value is never re-scanned (a prompt containing {model} stays verbatim). The placeholders are HARNESS_PLACEHOLDERS — prompt model worktree ref effort subagent session, pinned to Schemas.Harness.placeholders/0 by the fixture's vocabulary.harness_placeholders. [ … ] marks an optional segment ([--effort {effort}] → tokens [--effort {effort}]): it is dropped whole when any placeholder in it is null/empty; such a placeholder outside a segment, or an unknown name, is a TemplateError, and run_node_job fails the node with that message without spawning. So does an agent job with no harness (this job carries no harness — the server and runner disagree; update both). The values are prompt (compose_node_prompt), model/effort (the claim's), worktree (the slot path), ref, subagent (the node's agent) and session. _stream_agent_job(argv, cwd, …, session_id_path=…) (formerly _stream_claude_job) runs the argv with stdin=subprocess.DEVNULL (codex exec reads a non-TTY stdin) and reads the session id from each JSON event at the harness's session_id_path with read_path (.session_id, .thread_id; latest non-empty wins, no path captures nothing). Talk is out of scope and stays on claude: run_talk_job expands the fixed TALK_COMMAND / TALK_RESUME_COMMAND with session path .session_id.

  • Harness inventory (RE433). harness_inventory(harnesses) reports, per board harness, {key, installed, version, signed_in}: installed = the template's executable is on PATH, version = the first line of <exe> --version, signed_in = the harness's signed_in_check exited 0 (null with no check or when not installed). Each probe has a 10 s timeout (HARNESS_PROBE_TIMEOUT_S) and degrades instead of raising. RunnerHeartbeat.refresh_harnesses fetches GET /api/harnesses and recomputes it — at startup before the first beat (so the first beat already carries harnesses and the server never reads the runner as Claude-only), then again only when a beat reply's harnesses_digest differs from the one held. Every later beat carries the inventory; a failed fetch leaves the digest alone so the next beat retries.

  • The claim/execute/report loop (cmd_start). Each iteration: advertise current free capacity per isolation class on a long-poll POST /api/node-jobs/claim (a read timeout is "no work", not an error) — the claim body is {runner, capacity, running, held, harnesses}, harnesses (RE436) being the installed harness keys minus those paused at a usage limit, omitted before the definitions are first fetched (see Usage limits); on a claim, hand the job to a worker thread bounded by the pool's free slots; the worker resets the slot if needed, runs the step (shell/gate via _stream_shell, agent via _stream_agent_job), and POSTs the typed outcome to /api/node-jobs/:id/outcome. --once drains a single claim→execute→report cycle and exits; --dry-run claims and mutates nothing (it only logs the capacity it would advertise); --interval overrides the configured poll timeout; SIGINT stops claiming new work and waits for in-flight workers to finish.

  • The heartbeat cannot wedge, and a failing one is visible (RE336). Every api() call gets a finite read timeout, API_TIMEOUT_S (30s), unless its caller passes its own. The long-poll claim and the startup board fetch pass their own and still get the socket.timeout re-raised. On a default call, a read timeout retries only for GET/PATCH or an idempotent POST. Anything else fails the call, since the server may already have processed it. Before this change a bare urlopen blocked forever on a half-open connection. One heartbeat POST caught that way by a server restart silently killed the runner's heartbeat for days, and with it release_held, revoke, capability refresh, usage reporting, auto-update and capacity advertisement. The roster still showed the runner fresh, because claims also stamp last_heartbeat. RunnerHeartbeat._beat now guards everything it does (capability scan, held_fn, the POST, the reply handlers), and _run guards the loop body as well. A failing beat forwards one error line naming the cause (die() raises Died, a SystemExit that carries its message), repeats at most once per IDLE_LOG_INTERVAL while failures continue, and logs one "heartbeat restored after Ns" line when a beat lands again.

  • Heartbeat-borne revoke. RunnerHeartbeat POSTs {runner, capacity, running: [job-ids], held: [{ref, state}]} to POST /api/node-jobs/heartbeat every heartbeat_intervals and reads {revoked: [job-ids], release_held: [{ref, status}], latest_runner_version} back (RLY-164, ref keying RE311), terminating each revoked job's live subprocess via its JobControl. release_held is the ref-scoped analogue of revoked: the runner advertises every per-card worktree it holds (held), and the server names the subset whose card has at least one run and no run left in Schemas.Run.active_statuses/0 — with the status needed to choose remove vs retain (Relay.Runs.releasable_held/2) — so RunnerPool.release_held/2 disposes of them within one heartbeat. Since RE337 release_held also carries human-requested releases: the runners page's Release worktree action appends the card ref to the runner row's runners.release_requests (Relay.Runs.request_worktree_release/4, accepted only while the runner's latest held reports the ref bound), and the reply (Relay.Runs.release_held/3) adds {ref, status: "cancelled"} — a remove disposition (Schemas.Runner.release_remove_status/0), never failed — for each pending request this beat still reports bound. The run stays parked; on resume the runner's assign() rebuilds the worktree from the branch. Clearing rule: every beat that reports held prunes the column in one atomic UPDATE (upsert_runner/2) — a request survives only while its ref is reported bound or talk (teardown deferred under a talk turn); absent (torn down) or running (the run resumed first) clears it, so a stale request can never tear down a resumed run's tree. A runner that goes silent keeps its requests until it beats again. No runner change was needed: RunnerPool.release_held already refuses a live or talk-attached tree and stashes dirty edits as auto-salvage <slot> (never restored). A card with zero runs is a talk-only worktree and is never named (ADR 0009 §2: a talk session's tree spans runs and must outlive them), and a retained tree is the human's post-mortem, the runner's own to evict. This is how taking the baton (ADR 0004, via park_claimed/1) or cancelling from the run panel stops a running agent without waiting on its next outcome POST — and, unlike the retired run-id-keyed channel, it also reaches a worktree the runner re-derived from disk after a restart. Never-detach (RLY-231): a revoked job of either isolation class leaves its worktree exactly as it was — an exclusive worktree is bound 1:1 to its card and stays active + attached to the branch for the pinned resume to continue in, and a revoked shared_clean job already left exec-clean untouched (it's shared by other concurrently running jobs and only ever fast-forwarded once every shared slot is idle). The old reset-on-revoke for exclusive jobs is gone: no reset is ever needed mid-run now that a worktree's identity is the card itself. Either way, no outcome is reported for a revoked job — the server already knows a revoked job never finished. The same beat carries sessions (RE358), each in-flight agent job's captured session id, read from its JobControl (talk jobs are never reported).

  • A newly claimed job supersedes the stale job it replaces (RE358). Before placement, cmd_start snapshots in_flight under its lock (each entry carries the job's run_id, node_id and kind) and calls supersede_stale_jobs: every other in-flight non-talk job of the claimed job's run at the same node is cancelled through its JobControl and joined, outside the lock, for up to SUPERSEDED_JOB_JOIN_S (30s; an error line names a job still running after that). A run has one active job per running branch (Runs.active_jobs/1): one outside a fan_out region, one per branch inside it (RE437), and branches are node-disjoint. So a same-run, same-node job is either steered (a Steer now re-enters the same node; revoked, but the heartbeat that says so may be up to an interval away) or already reported and still in its finally, where the cancel is a no-op. A same-run job at another node is a concurrent sibling branch and is never touched. This closes a real gap: a granted claim never carries revoked, and RunnerPool.assign reuses an exclusive worktree bound to the same run_id even while it is live, so a Steer now restart could otherwise start in the same tree as the claude -p it replaced. Talk jobs are never superseded — a talk turn may legitimately run beside its card's node job.

  • Auto-update (RE185). When the beat's latest_runner_version exceeds the running RUNNER_VERSION, maybe_auto_update fires from the claim loop — but only at a job boundary (nothing in flight), never under --once, and never more often than auto_update_min_interval, and never again after three failed updates in one process's life (a refused download or a failed install — a successful one never counts, so a runner that has been up for months and picked up ten releases is unaffected). It downloads ./relay from the board's scaffold endpoint ($RELAY_URL/api/scaffold/relay) through download_runner, which shares verify_runner_source with relay update — HTTPS, UTF-8, a leading #!, a RUNNER_VERSION parsed from the downloaded bytes (authoritative — a board ahead of what it serves is then harmless, not a chase), a compile() syntax check, and (for auto-update only) strictly newer. Verification is deliberately HTTPS + parse only: no checksum, no signature. Any rejection logs why and the running version keeps serving; a bad update can never leave a machine with a broken runner.

    The install writes the tracked ./relay in place (os.replace, so a concurrent ./relay card … never reads a half-written file). _safe_to_overwrite is what makes that tolerable: untracked or not-a-repo is fine, tracked-and-clean is fine, and tracked-and-dirty is allowed only when the file's sha256 matches a write recorded in ~/.relay/auto-update.json — otherwise the dirt is a human's uncommitted edit and the update is skipped with a one-time log line. That ledger is not a nicety: after the first auto-update the file is permanently dirty against HEAD, so a clean-only rule would wedge every later update. The accepted cost is that ./relay shows as modified in git status on the machine running the runner.

    The restart stops the heartbeat and the log forwarder, calls release_runner_locks() — required, because RLY-193's flocks belong to open file descriptions that survive execv while the re-exec'd image opens new descriptors and would die "already running" — then os.execvs with RELAY_UPDATED_FROM/RELAY_UPDATE_ATTEMPTS set. On the way back up check_update_handshake compares: newer, and it logs the transition and clears the failure count (the update took, so nothing before it was thrash); same or older, and auto-update is disabled for that process, leaving RLY-184's fail-stop as the behaviour. Three failed attempts disable it the same way. Opt out with "auto_update": false.

Usage limits (RE320 → RE436)

relay start stops claiming work for one harness once that harness's usage passes the PAUSE AT its board rule sets, and resumes by itself. RE320 did this for Claude Code alone, from hard-coded event shapes and per-runner .relay/runner.json limits; RE436 made it generic. Reading usage, and deciding when to pause, is now each harness's usage rule (Settings → Agents → Harnesses, served by GET /api/harnesses — see "Harness routing" above). ./relay names no CLI, event or window in executable code (bin/test_relay.py RunnerHarnessGuardTest).

  • Config. PAUSE AT lives only on the harness (windows[].pause_at). There is no per-runner override. .relay/runner.json limits is retired: load_runner_config drops it and logs LIMITS_IGNORED_LINE once; it never refuses to start. The starter config no longer writes it.

  • The book. cmd_start keeps a HarnessUsageBook (the global USAGE): one UsageLimiter per harness key. RunnerHeartbeat.refresh_harnesses hands each successful fetch's raw definitions to on_definitions, which calls HarnessUsageBook.rebuild. A limiter whose usage rule is unchanged is kept, so its readings survive a digest move caused by an unrelated edit; a changed rule starts afresh; a harness gone from the board is dropped.

  • Usage sources. A limiter is fed three ways, all through usage_reading (the rule evaluator):

    • output — the job's own stream. run_node_job passes the claimed job's harness limiter (job["harness"]["key"]) to _stream_agent_job(usage=…), which feeds every parsed event to it. There are no extra calls in the normal path. Claude Code's seed reads the rate_limit_event its stream-json emits on every call. Talk turns feed no limiter: they are out of RE436's scope, and the old shared limiter they fed is gone.
    • command — UsagePoller, a daemon thread ticking every USAGE_SLICE_S (5s), runs the rule's command through sh -c from the system temp dir every interval_min (run_usage_command). Each stdout line that parses as JSON is observed; the last readable one wins. Codex's seed reads the token_count line of its newest rollout file.
    • re-check — while a harness is paused and has no job in flight (the loop's in_flight entries record each job's harness key), the poller runs recheck_harness when recheck.interval_min has passed since its last reading or re-check. That expands the harness's own command template with prompt ok, the rule's recheck.model and the temp dir as {worktree}, runs it from the temp dir, and reads its stdout (output) or then runs the usage command (command). It returns a reason rather than raising (a template needing {ref}, a timeout, a CLI that refuses a non-git cwd), never un-pauses anything by itself, and a failure is logged once per pause per harness. Every usage command and re-check times out at USAGE_COMMAND_TIMEOUT_S (60s).
  • Pause. A window trips at used_pct >= pause_at. A line meeting the rule's pause_now.when is a refusal and trips regardless, until its reset (reason: "rejected"); a later readable line that is not a refusal ends it early. With several trips, UsageLimiter.paused reports the latest reset (a refusal wins a tie). Each reading and refusal is discarded at its reset. Before each claim the loop takes installed (the inventory's installed keys, None before the first fetch) and paused = USAGE.paused_keys(now):

    • with every installed harness paused, it claims nothing — node, gate or talk — and sleeps in USAGE_SLICE_S slices; jobs already running finish (RE320's contract, now for the all-paused case only);
    • otherwise it claims with harnesses = sorted(installed - paused) (omitted before the first fetch). The server ignores the claim's harnesses until it filters by it.

    It logs ⏸ <name> paused: <label> <used>% ≥ <pause_at>% · resumes HH:MM · re-checking every <N>m (or refused a call) once when a harness enters a pause and ▶ <name> usage back under its pause-at — claiming its nodes again once when it leaves.

  • RE267 resume-at. When _stream_agent_job's limiter reads a refusal with a reset from the job's own stream, it sets report.rejected / report.rejected_resets_at; blocked_resume_at sends that as the outcome's resume_at. ClaudeRunReport no longer parses any usage event itself.

  • Heartbeat. Each harnesses[] entry carries usage: UsageLimiter.wire(now) — {source, read_at, windows: [{label, used_pct, resets_at}], paused: {label, used_pct, pause_at, resets_at, reason} | null} with reason in Schemas.Runner.rate_limit_reasons/0 (limit | rejected, mirrored as USAGE_PAUSE_REASONS) — or {source: "none", read_at: null, windows: [], paused: null} for a harness the book does not know. The top-level rate_limit is no longer sent. Beats continue while paused, so the runner stays fresh. The server stores each reading on its runners.harnesses entry via Schemas.Runner.normalize_harnesses/1. That normalizer is total: an unknown source drops the whole usage and keeps the entry; a malformed window is dropped, and at most 20 are kept; a malformed paused (unknown reason, non-positive resets_at) becomes nil. The next beat's paused: null clears a pause; a beat without harnesses leaves the stored readings alone. The roster derives "rate limited" from these readings (see the Rate-limited roster bullet).

Talk turns (RE268, ADR 0009)

The claim loop's worker() branches on job.get("kind"): a "talk" job runs execute_talk instead of execute_one, and reject() reports a rejected talk job through report_talk_outcome (POST /api/talk/turns/:id/outcome) rather than the node-job outcome route — a talk job has no run_id for that route to accept. No second thread: it is the same claim/execute/report loop, one more branch.

  • Worktree. A talk turn always runs in the card's own exclusive per-card worktree <ns>-<ref> — never the shared <ns>-clean tree, which other cards' jobs are using. RunnerPool.assign_talk/1 attaches to a live worktree a node job already holds (dirty reads are the point — the tree may be mid-edit, and that is often exactly what is being asked about), reuses a retained failed one as-is (post-mortem is what people ask about), or creates one on demand for a card the flow engine would never itself dispatch.

    A node job and a talk turn can occupy the SAME worktree record at once, so occupancy is tracked per occupant: live for the node job, and a talk_users count for talk turns — a count, not a flag, because two turns can legitimately overlap on one tree: Stop finalises a turn server-side at once, but the runner only learns of the revoke on its next heartbeat (15s), so a person who hits Stop and immediately retypes has turn 1 still running when turn 2 is claimed into the same tree. A flag did not count the second occupant, and turn 1's release then stashed and force-removed the tree turn 2 was answering in. RunnerPool.assign_talk/1 and release_talk/1 only ever touch talk_users, never live — an earlier version shared one live flag between both occupants, which let either tear the tree down (or believe it idle) out from under the other: a node job's release() finishing while a talk turn was still streaming would run git worktree remove --force mid-answer and stash the person's uncommitted edits; a talk turn ending would clear live under a still- running node job, defeating the release path's "never touch a worktree with a live job" guarantee. release()/release_held/2 (which replaced the run-id-keyed release_run/2 in RE311, keeping both of its guards verbatim) DEFER a terminal disposition (pending_finish) while any other occupant (live, or talk_users > 0) is still present; release_talk/1 finishes it once the last one leaves — last occupant out tears down. The deferred disposition is read only after those early returns, so a non-final release cannot discard it.

    A talk-created worktree carries run_id: None, but so does a worktree recover() re-derives after a restart — the exact case release_held/2's ref keying exists to reach (RE311) — so run_id alone can no longer gate the skip. release_held/2 instead checks the talk marker: a talk-created record is skipped outright, because its tree spans runs and must outlive them, while a recovered one is not, so a release_held entry can still tear down a tree bound to nothing that was never talk-only. Once such a tree goes fully idle (its talk turn ends and no node job ever adopted it), release_talk/1 retires it — releasing its partition and marking it retained, evictable oldest-first exactly like a failed run's leftover — so a card that was only ever talked to, never run, does not permanently consume an exclusive slot (and keep the poller from ever idling). A retained tree a talk turn has reattached to is excluded from that eviction while it is live.

    Retirement is gated on the record's own talk marker, not on run_id is None alone (RE268): recover() rebuilds every restart-adopted per-card worktree as {"run_id": None, "state": "active", ...} too — the identical shape, with no talk marker. Retiring on run_id alone mistook a recovered worktree for talk-only and retired (stashed + hard-reset) it out from under its resuming job.

    RunnerPool.assign/1 (the node-job path) likewise refuses to reclaim a retained tree a talk turn has reattached to (talk_users > 0) rather than re-baselining it out from under that turn's still-reading claude process — the same "refuse rather than steal a live worktree" precedent it already applies to a tree bound to a different live run.

    Restart durability of the talk marker (accepted gap, RE311). That marker is in-memory only: recover() rebuilds every adopted tree with no talk key, so after a restart a talk-created tree on a card that HAS runs (all terminal) is named by the server and torn down, which ADR 0009 §2 otherwise forbids. The exposure is narrow — an idle talk-only tree has already been retained by _retire_talk_only_locked, and retained IS disk-marked and survives — so it takes a talk turn that was live at crash time, whose claude process died with the runner anyway; teardown stashes any dirty edits (_teardown salvages via git stash push -u) rather than discarding them. Marking talk on disk the way RETAINED_MARKER is would close it, but the marker would have to be written after the tree is created (in the talk worker, not assign_talk/1) and CLEARED the moment a run adopts the tree — a stale marker would make release_held/2 skip that tree forever, which is the very slot leak this card removed. Left as a documented gap rather than a half-durable marker.

    Adopting the card's own tree (RE311). assign/1 reuses an active record when it is bound to this run, has an unknown binding (run_id: None, the recover() shape), OR is idle and bound to an earlier run of the same card — the last case is cancel-then-retry: a revoke deliberately leaves the tree active, idle and bound to the cancelled run, and the release channel only frees it on a later beat (~15s). Refusing there placed nothing for the bounded retry window and then failed the retry with a misleading "no free 'exclusive' slot advertised", so a card's retry could fail for holding its own worktree. The refusal is now exactly what its comment always claimed: a genuinely LIVE run, or a deferred pending_finish.

    Bounded, and retried: while a node job's terminal disposition sits deferred (pending_finish), the record stays state="active" bound to the OLD run_id. If a NEW run for the same card is dispatched before the talk turn ends, assign/1 refuses it — a deferred disposition still owns the tree, and adopting it would let release_talk/1 tear the tree down under the new run. (This is now the ONLY reason assign/1 refuses an idle tree: since RE311 an idle active record bound to an EARLIER run of the same card is adopted, because the tree's identity is the card's branch. See "Adopting the card's own tree" below.) That refusal used to be reported as a failed job, so someone asking "why did this fail?" during a retry failed the retry. The claim loop now retries placement for a bounded window (PLACEMENT_ATTEMPTS × PLACEMENT_RETRY_S, ~10s, ./relay) before rejecting, which covers one talk turn handing the tree back; only a persistent miss is treated as genuine capacity exhaustion. See failures.md E2t.

    A talk turn that creates its own worktree fresh (the card's run is done/cancelled and was torn down, or the card was never run) lands on the card's own branch when one exists, via checkout_talk_branch/2 right after create_or_rebaseline — not left detached at origin/main. Falls back to the existing detached-at-base state when there is no branch yet or it has not reached the remote. It prefers the LOCAL branch when one already exists and never resets it — checkout -B <branch> origin/<branch> force-moves the branch ref to the remote tip, which would silently discard commits a run made and never got to push — and falls back to creating the local branch from the remote only when there is no local ref to lose. A failed checkout (e.g. the branch is already checked out in another worktree) is logged rather than swallowed, so it doesn't silently fall back to a detached tree describing none of the card's work.

  • Prompt. talk_prompt(job) is built in product code, not a .claude/agents/*.md definition — a recorded exception to ADR 0006 (ADR 0009 §5): Talk is a property of Relay itself and must behave identically on every connected repo. It is the preamble (names the pane, the read-only rule, and ./relay why/runs/card) + the card's seed fields + the human's text, spliced in verbatim and last — never passed through render(), so a person's {ref}-shaped typing can never reach into the var namespace.

  • Event mapping. _stream_agent_job (then _stream_claude_job) gained an on_event=None callback, invoked with each parsed stream-json event as it arrives — this is what lets a turn's transcript stream while the turn is still working, not all at once at the end. talk_events_from(ev) maps one event to the transcript lines it produces: assistant text → an :out line, a tool_use block → one dim :tool line naming its target, an errored result → an :error line. Anything else (including an unrecognised event type) produces nothing — a mangled or unfamiliar event must never cost the turn. The tool-use "brief" (command/file_path/path/pattern/description, truncated to 140 chars) is one shared _tool_brief helper, used by both this and the console/board-log renderer (_print_claude_event).

    _stream_agent_job also gained mirror=True; a talk turn passes mirror=False so its events print locally but are never also forwarded to the board-log mirror. Two things make that necessary: the tag run_talk_job builds is ref-clean (f"[{ref}] (talk) ", "talk" outside the brackets — an earlier f"[{ref} talk] " made _ref_from_tag extract a ref that does not exist, e.g. "DE3 talk", spamming POST /api/board/logs under a phantom card), and a talk job's worker deliberately registers no node_job_attribution. Board-log attribution is per worker thread (RE437, replacing the ref-keyed NODE_JOB_IDS/RUN_IDS maps): forward() stamps a line with the run_id/node_job_id of the node job the calling thread runs for that ref, so a talk turn's lines carry neither, and concurrent fan-out branches on one card never clobber — or, finishing, erase — each other's ids.

  • Delivery. TalkEventSender batches and POSTs lines to /api/talk/turns/:id/events at-least-once — deliberately unlike LogForwarder, which drops by construction (enqueue on a full queue, swallowed _send errors). Each line's client_seq is per-turn and monotonic from 1, which is what makes a retried batch idempotent server-side (Relay.Talk.append_events/2 dedupes on it). A failed POST retries with backoff up to max_attempts (no sleep after the last attempt — the decision to give up is already made by then); exhausting the budget sets failed and run_talk_job calls send_best_effort to re-arm and try once more to deliver a single visible :error line saying the transcript could not be fully delivered, then reports the turn failed rather than silently reporting a done turn with missing output.

    batch=40 is a size cap, not a delay: a typical short turn (a couple of tool lines, a couple of text blocks) never reaches it, so without a second trigger every turn's transcript arrived in one POST at the very end — defeating the point of the on_event streaming hook (RE268). enqueue also flushes whatever is already pending once it has waited flush_interval (default 1s), checked before the new line joins it, so a slow trickle of events posts as it arrives instead of all landing in one batch together.

  • Stop. Arrives as an ordinary revoke on the existing heartbeat — RunnerHeartbeat already terminates a revoked job's subprocess via its JobControl; run_talk_job checks control.cancelled() after the process exits and reports stopped (not failed) with whatever partial output was already delivered. No talk-specific channel was added.

RUNNER_VERSION 33 → 39 for this change (34 shipped the initial worker, 35 the occupancy/ retirement/attribution hardening above, 36 the recovery/retained-tree/streaming/branch fixes from a follow-up review, 37 the branch-checkout non-destructiveness fix above, 38 the failure-line-into-the-transcript fix from the whole-branch review, 39 the talk_users occupancy count, the bounded placement retry, and the rejected-turn transcript line).

One version floor (talk included). Talk originally shipped with its own, higher floor (38, then 39): Relay.Runs.min_runner_version/0 was 21 at the time and was not raised for Talk, because a runner without Talk is not worse than a stopped one for the flow work it still does correctly. RE311 then raised the base floor past it (it is 63 today), and RE414 removed the separate talk floor. Today talk_capable?/1 inside claim_next_job/1 gates talk claims on min_runner_version/0, narrowing the claim to NodeJob.flow_kinds() for anything below it (over HTTP such a runner is refused 409 runner_outdated first, so the gate is defense-in-depth for in-process callers). Without it a pre-Talk runner would happily claim the first (deliberately unpinned, capacity-exempt) turn on a card, KeyError on the isolation key a talk payload does not carry, and reject it to the flow-only outcome route — a 404 that leaves the turn claimed forever and wedges Talk for the whole board (failures.md D4t).

Declaring an outcome

An agent node must declare its verdict before it exits, by running:

relay outcome <outcome> [--detail TEXT|@file]

./relay writes the JSON to $RELAY_NODE_OUTCOME (set per node; the verb refuses to run outside a node), so a detail containing quotes or newlines cannot corrupt the file. detail becomes the context handed to the next node. Which outcomes exist and what each one does to the run and the card is state.md's "Node outcomes" table — the schema owns that set, not this page.

Five rules sit on top of it:

  • Silence is failure — but silence is not nothing. A node that exits without declaring is reported failed whatever its exit code: a node that did nothing is indistinguishable from one that exited early, so it must never route past its own gate. The detail, however, names any uncommitted work the node left in its worktree (RE298), because "did not declare" and "did nothing" are different failures and the retry receives this detail as its findings. Without it the work is invisible — the retry redoes it from scratch and the next job on that slot sweeps it into a stash.
  • A success claim must be backed by a commit — or by proof the commit already exists. On a node marked expects_commits, a succeeded that left HEAD unmoved is rewritten to failed before finalize (failures.md A6). The one way out is relay outcome succeeded --no-changes (RE310): an ASSERTION that the work was already committed, which the engine honours only when it can show from this run's own history that this node already produced a commit for the thing it is bound to — the same node and, under a foreach, the same sub-task. On a node's first visit for that binding the two baselines coincide, so the claim is always rejected and default-deny survives. The claim is recorded on the execution (no_changes) whether accepted or rejected, and a rejected one fails with (no_changes_unproven: <node>).
  • A declared card write must actually land. On a node declaring writes, a succeeded that left one of those card fields blank is rewritten to failed before finalize (failures.md A10). reads is never checked at run time — it is advisory.
  • Asking a human wins. If the node moved the card to needs_input, that is the outcome even if the node also declared something else.
  • An agent that could not run is blocked, not failed (RE308). When claude -p exits non-zero and its own stream shows the environment refused it — the assistant event's error tag (authentication_failed/billing_error → auth, rate_limit → usage limit), the result's api_error_status (401/403, 429), a rate_limit_event with status: "rejected" seen in this job's stream, or else a phrase from CLAUDE_AUTH_FAILURE_SIGNATURES/CLAUDE_USAGE_LIMIT_SIGNATURES in its last words — determine_agent_outcome reports blocked with agent could not run: <reason> (a usage limit adds · resets HH:MM). The engine parks without spending retries (failures.md A11). The check runs after "asking a human wins" and after a declared outcome file, so a declared verdict is never reclassified. blocked is not one of relay outcome's choices, and an outcome file declaring it is reported failed, so an agent cannot park itself around its own retry budget. An unclassified non-zero exit stays failed, its detail now agent exited non-zero: <the stream's last words>. When the classification is a usage limit AND this job's stream carried a rejected rate_limit_event with resetsAt, blocked_resume_at returns that epoch, run_node_job returns it as its 6th element, and outcome_body sends it as resume_at (the key is omitted otherwise, so every other body is unchanged). The server then requeues the node rather than parking it — up to 3 consecutive waits — while the runner's own RE320 pause holds its claiming until the reset (RE267). Auth, billing_error and a phrase-only usage limit send no resume_at and park as before.

The reminder is appended to every agent node's prompt automatically, so the requirement travels with every invocation. shell and gate nodes are exempt — their exit status is already an unambiguous verdict.

What an agent node's prompt is made of

./relay composes an agent node's prompt from up to four parts, in this order (compose_node_prompt):

  1. the node's own run, rendered — {ref}, {branch}, {relay} and the rest substituted from the job's vars;
  2. the findings block, appended only when the job carries a non-blank vars["findings"] (RE251);
  3. the guidance block, appended only when the job carries a non-blank vars["guidance"] (RE358);
  4. the outcome contract, rendered, appended to every agent node.

shell and gate nodes get part 1 only: their run is a command line, not a prompt.

The findings block (RE251)

On a loop-back the engine sets findings on the next job's payload — the reviewer's detail on a transition, and the ORIGINATING findings plus this attempt's failure detail on a retry (RunServer.apply_decision/4). Substitution alone was never enough to deliver it: it only fires for placeholders a template contains, and no shipped flow node contains {findings}, so every fix node in the system was told to fix findings it was never handed. Appending the block in the runner makes it universal — every agent node, every flow, every repo, and nothing for a flow author to remember.

The block states that this is a loop-back and the findings are the subject of the run, carries the findings, requires the node to account for every one of them in its outcome detail (fixed, or rebutted with a reason), and makes "no finding needs a change" an escalation (needs-input) rather than a succeeded. It points at the outcome contract below it for the exact commands, so there is one rendered copy of each command per prompt.

The findings text is spliced verbatim and never rendered. Findings are free-form reviewer prose: a reviewer who writes {ref} must see those characters reach the model, and reviewer text must never be able to reach into the var namespace. The static wrapper carries no placeholders.

The guidance block (RE358)

Human comments and rejection notes written while a card is being worked reach every agent node. The server derives them live, never snapshotting them into run.context: Relay.Runs.guidance_for/2 selects them and Relay.Runs.render_guidance/2 renders them into vars["guidance"] (a string, or nil when there are none).

  • The window opens at the earlier of the card's open rejection's rejected_at and the run's inserted_at (just the run's when no rejection is open), inclusive — timestamps are second-precision and the rejection's own note lands in the same second. So a note a human wrote between a reject and the hand-back still counts on the run the hand-back starts.
  • The entries are user comments whose kind is in Schemas.Comment.guidance_kinds/0 (:comment, :changes_requested, :steer) plus any :changes_requested note whatever its actor. Agent :comments and every :question are excluded. The rejection note is already a :changes_requested comment, so it is never appended a second time from card.rejection. Entries are ordered oldest → newest (inserted_at, then id).
  • The rendering is one entry per comment, separated by a blank line: a header <author> · <YYYY-MM-DD HH:MM> UTC (tagged · CHANGES REQUESTED or · STEER by kind), the body verbatim, and — when the note carries images — (N images attached — see ./relay card <ref>), since images don't travel in a prompt.
  • Derived at insert, refreshed at claim. build_payload/4 computes it for every job it builds (agent, shell and gate alike). In the pull model the claim is the real dispatch — a job is inserted when its run starts or advances but may be claimed much later — so claim_next_job/4 recomputes it on the job it hands out and stores it back on the row (refresh_job_guidance!/1, inside the claim transaction; talk jobs are untouched). A note added after a reject but before the runner picks the job up still reaches it.

In the runner, guidance_block/1 wraps the text in GUIDANCE_CONTRACT_HEAD / GUIDANCE_CONTRACT_TAIL. The head frames the notes as instructions for this run and says later entries supersede earlier ones — including the CHANGES REQUESTED note that sent the card back; the tail asks for one short line per entry in the outcome detail (followed, superseded, or not applicable) and makes an unresolvable conflict a needs-input escalation. Like the findings block, the guidance text is spliced verbatim and never rendered — a human who types {ref} gets those characters, and human prose can never reach the var namespace — and the static wrapper carries no placeholders. An older runner ignores the var, which is the pre-RE358 behaviour, so @min_runner_version was not raised.

Steer now (RE358)

The drawer's Steer now is Relay.Runs.steer/4: a human's note that stops the agent node working the card and restarts that same node with the note in its guidance.

  • The comment comes first, always. The steer is saved as a :steer comment (Relay.Activity.add_comment/2) in its own transaction before anything else happens, so the words survive whatever follows. A blank steer with no images is a changeset error and nothing else happens.
  • Steered re-entry. When the card's run is :running, the call goes through the run's RunServer ({:steer, actor, body, image_ids}, serialized with outcome reports). If the run's current node is an agent node with an active job whose attempt has no outcome yet — claimed or still queued, so a steer is never lost to a payload built before it — then in one transaction the server stamps that attempt outcome: :steered (no detail), revokes its job (revoke_active_jobs/1; the runner kills the process when its heartbeat's revoked list names it) and enters the same node, same visit, attempt + 1, same sub_task_id. The new job's payload carries resume_session = the stopped attempt's session_id (so claude --resume keeps the conversation when one was captured — see "The session survives the kill" below) plus the stopped job's findings and prior_detail. After commit: {:node_finished, run, steered}, {:node_started, run, new}, dispatch. The run's status is never written — it stays :running.
  • :steered is server-only. It is in Schemas.NodeExecution.server_only_outcomes/0, so the outcome API refuses it like any unknown outcome (outcomes/0, the wire contract pinned by the runner contract fixture, is unchanged) and no edge may route on it. The engine reads a steered row as itself, never as a degraded :failed, so it spends no max_loops budget; and since any outcome ends RE410's outcome-less streak, steering repeatedly never trips the RE410 cap (the steered re-entry doesn't consult it anyway). A later re-entry still carries the last real failure as findings — last_failure_detail/1 skips server-only rows as it skips :blocked. The Run tab shows a steered attempt as ↪ with a "steered" chip; it is not a spent attempt, and its restart reads "session resumed" when a session was kept.
  • Degrade to a note. No running run (none, parked, or one that stopped mid-call), a shell or gate node, or no outcome-less active attempt: nothing is revoked and steer/4 returns {:ok, comment, :noted}. The steer still reaches the next agent node through its guidance.
  • Receipts land at claim. refresh_job_guidance!/1 (the claim-time guidance refresh) stamps comments.delivered_execution_id = job.node_execution_id on every :steer it just rendered that has no receipt yet — for agent jobs only (a shell or gate job never delivers a steer), and first delivery wins. steer_receipts/1 reads them back for the drawer (node, session_kept?, and the attempt's outcome detail as its reply once it reports); steer_state/1 says whether the card's AI-held run is on an agent node and whether a restart is still waiting to be claimed. A flow-job claim emits {:run_changed, card_id} so the drawer refetches.
  • The session survives the kill. A revoked job reports no outcome, so its session id would be lost. Instead the runner reports every running agent job's captured session on each heartbeat (sessions: {job-id: session-id}), and Relay.Runs.record_job_sessions/2 writes it onto that job's NodeExecution.session_id — which is exactly what the steer reads as resume_session. So the hand-off is beat → NodeExecution.session_id → resume_session: a steer after at least one beat has carried the session resumes it; a steer before the first such beat restarts on a fresh session (the guidance still carries the note).
  • The restart never shares a worktree with the process it replaced. When the runner claims the restarted job, it first supersedes the run's still-running stale job at that node — cancels it and waits for it to exit (supersede_stale_jobs, see "Runner mode" above) — before placing the new one. The revoke alone was not enough: it only arrives on the next heartbeat, and RunnerPool.assign reuses an exclusive worktree bound to the same run even while it is live.

Agent node → .claude/agents definition

A flow node of type agent may name an agent (e.g. plan-implementer). The server carries it in the job payload (Relay.Runs.build_payload/4 → the claim response's agent), and ./relay fills the harness template's {subagent} placeholder with it (RE433). The Claude Code harness carries it as an optional [--agent {subagent}] segment, so a node with an agent runs claude -p … --agent <name> and a node without one drops the segment: the agent file supplies the system prompt, the node's run string stays the user prompt. A harness whose template has no {subagent} simply ignores the node's agent (.claude/agents on non-Claude harnesses is out of RE433's scope). An unknown name makes the CLI fail loudly rather than silently fall back to the default agent (verified against CLI 2.1.214), which is the property that makes this safe to depend on. A node with no agent invokes exactly as it did before RLY-139.

Fallback if --agent ever regresses: delegate by name from the node's run prompt — "Use the spec-reviewer subagent to review …". It works today with zero new plumbing and needs no schema change.

Escalating a plan-mandated finding (RLY-190)

A reviewer that finds a defect the code implements faithfully because plan.md mandates it does not return failed into the fix loop. The implementer is instructed to follow the plan, so neither side can yield: the loop burns its budget until loop_budget_exhausted or the circuit breaker ends the run with a message describing the symptom and not the cause. Instead the reviewer raises needs-input and stops without declaring an outcome, which parks the run for a human. spec-reviewer, quality-reviewer and final-reviewer each carry this as a third verdict beside Approve/Pass and Fix; plan-implementer uses the same route when the plan tells it to build something it can see is wrong.

This needed no engine change. needs_input is decided before any edge is consulted (Relay.Runs.Engine), so it consumes no max_loops budget, does not increment the visit count, and is not degraded to :failed. The command itself reaches every agent node for free — ./relay appends its outcome contract, already rendered, to every agent prompt (see What an agent node's prompt is made of).

The human's answer, not the plan, is authoritative for the remainder of the run. This is a deliberate decision, and the tempting alternative — have the human edit the card's plan and treat the edited plan as authoritative — does not work:

  • the plan is materialized once, to the per-ref $RELAY_PLAN path, by the Code flow's branch node (Relay.Flows.DefaultLibrary), and no later node ever re-writes it.
  • sub_tasks are seeded from card.plan at run start only, and are deliberately never re-materialized on re-entry so that done-state isn't wiped (Relay.Runs).
  • Re-entry replays the same node, same visit, fresh attempt with the agent's claude session resumed (RunServer.enter_same_node!), and the prompt is byte-identical to the parked attempt's — build_payload exposes no answer variable.

So editing the plan mid-run changes nothing any node reads. The answer arrives as a card comment (Cards.answer_input/3) and the resumed reviewer reads it with relay card <ref>. plan.md and card.plan stay stale by design; any lasting plan correction is a follow-up card, not a mid-run mutation.

Escalation is deliberately rare: Fix remains the default, and a reviewer may escalate only when it can quote the plan text that mandates the defect — the test being "can the implementer act on this without contradicting the plan?". A reviewer that escalated because a finding was merely hard would convert a self-healing loop into a human queue. On resume the reviewer resolves rather than re-parking: "fix it anyway" returns Fix with the authorization quoted verbatim (which the implementer is instructed to treat as outranking plan.md for that task), "waive it" returns Approve with the waiver and follow-up recorded.

Where the contract's text lives — inline vs. .claude/agents/references/ (RE423). An agent definition IS its system prompt: the .md file is loaded whole at invocation (--agent <name>) and has no include mechanism, so nothing in a sibling file is ever passively in context. A references/ file reaches the model only when the agent reads it at runtime, with a Read or Bash tool call, from the card checkout it runs in. That fixes the rule for what goes where:

Inline what the agent needs in order to decide; put in references/ what it needs only after it has decided. An agent file is always in context. A reference file reaches the model only when the agent reads it at runtime, so its content must be something the agent only needs once it already knows it needs it.

  • Escalation. The decision rule (### Escalate sparingly, the quote-the-plan test) and the **Escalate** verdict in ## Decide stay inline in each of the three reviewers — a reviewer that never reads the rule can never decide to escalate. The mechanics of how to escalate (what the question must say, the two options, what to do on resume) live once, in .claude/agents/references/escalating.md, which each reviewer points at. plan-implementer carries its own, role-specific escalation text and does not point at references/escalating.md.
  • Re-review. The trigger (a fix commit on top, or a findings block in the prompt), each reviewer's own blocking rule and its own Pass/Approve line stay inline, because they decide the verdict. How to conduct a re-review once the agent knows it is in one — re-check only the findings, don't re-run the checklist, never re-raise a rebutted finding — lives once, in .claude/agents/references/re-review.md.

Inline by design — these paragraphs appear word for word in more than one agent file on purpose, and are the only such reviewer prose: (1) the re-review trigger paragraph, in all three reviewers; (2) the blocking bullets and the "→ Approve" line shared by quality-reviewer and final-reviewer; (3) the ### Escalate sparingly decision rule shared by quality-reviewer and final-reviewer; (4) the one-line pointers to the two reference files.

Probe evidence (2026-10-08, claude --version → 2.1.294 (Claude Code)). Each of spec-reviewer, quality-reviewer and final-reviewer was run headless with the runner's exact flags (claude -p … --permission-mode auto --verbose --output-format stream-json --agent <name>, cwd = the card checkout) and asked — without naming the path — to open the escalation reference, then the re-review reference, its definition points to and reply with the first line. All six runs read the right file and replied with its heading:

Agent escalation reference re-review reference
spec-reviewer Read .claude/agents/references/escalating.md Read .claude/agents/references/re-review.md
quality-reviewer Bash head -n 1 .claude/agents/references/escalating.md Bash head -n 1 .claude/agents/references/re-review.md
final-reviewer Bash head -n 1 .claude/agents/references/escalating.md Bash head -n 1 .claude/agents/references/re-review.md

Replies: # Escalating a plan-mandated finding and # Re-reviewing after a fix pass. An earlier version of this page said a references/ file "would never reach the model"; that is wrong for any agent that reads it with a tool, and the probe is what settled it.

Shipping caveat for the relay-* skills. The three relay-* skills still cannot rely on a sibling file — not because of how they load, but because of how they ship: Relay.Scaffold.items/0 names SKILL.md only, so a reference file next to one would silently vanish from every scaffolded repo (which is why test/relay/onboard_skill_test.exs pins the relay-onboard directory to SKILL.md alone). Agents are unaffected: Relay.Scaffold ships no agents at all — every agent is the repo's own, per ADR 0010. test/relay/agents/escalation_contract_test.exs pins the markers so an edit can't silently drop the contract, and fails on any references/ path an agent file names that does not exist (a dangling pointer).

Operating invariants

If you build your own runner or your own node behavior, honor these — break one and cards corrupt each other's work:

  1. One agent per working directory at a time. A git checkout (or branch/file edit) is global to the directory — two agents on two branches in one directory overwrite each other. Serialize (one card at a time), or give each agent its own clone or git worktree. Don't run the runner and an interactive session in the same working tree at once.
  2. State lives on the board, never in the working tree. Many cards are in flight, moving back and forth between stages; a card may be specced now and planned days later while others pass through. Nothing durable may depend on what's currently checked out or on a shared repo-root scratch file.
  3. Each card owns its branch — check it out at the start of a step, commit at the end. Every step must be self-contained: begin by checking out the card's branch (from its branch field), end by committing (never leave uncommitted changes for the next card to inherit).
  4. Work travels with the card. The spec is the card's spec; the acceptance criteria its acceptance_criteria; the plan its plan field. Materialize these into the branch just-in-time (at the per-card $RELAY_PLAN path), never via a shared file another card would clobber.

Readiness, ordering, WIP and failure routing are not on this list: they are decided server-side by the scheduler and the engine (see "Dispatch is server-side" above, state.md and failures.md). A runner that tries to decide them locally will disagree with the server.


Sources of truth: ./relay, .relay/runner.json, bin/test_relay.py, lib/relay_web/controllers/api/node_job_controller.ex, lib/relay/runs.ex, lib/relay_web/controllers/api/board_controller.ex, lib/relay_web/controllers/api/run_controller.ex, lib/relay_web/controllers/api/runner_controller.ex, lib/relay_web/controllers/api/flow_metrics_controller.ex, lib/relay_web/controllers/api/flow_controller.ex, lib/relay_web/controllers/api/talk_controller.ex, lib/relay/talk.ex, lib/relay/scaffold.ex, lib/relay_web/controllers/api/scaffold_controller.ex.