The runner: how work physically gets done
Today's system. ADR 0006 landed a server-side
flow engine + thin runner for every stage — Spec (RLY-136), Plan (RLY-138), and Code
(RLY-139, this doc's most recent cutover). The legacy board-runner (relay watch,
relay_config.json, .claude/workflows/execute-plan.js) is deleted; there is no
fallback dispatcher to describe. The runner lives on a developer machine — it needs the
checkout, git worktrees, and the claude CLI — and talks to the deployed app only through
the board-key REST API.
./relay (Python, single file) is two things:
- A CLI for every card operation an agent needs (
card,move,comment,needs-input,approve, …) — the surface documented in../../relay.md. relay start— the only runner mode: a poll loop that claims node-jobs from the server and runs them (see "Runner mode" below).
Dispatch is server-side
A sketch of a Code flow in this model (edges labeled with the outcome that routes them):
flowchart LR
start([start]) --> impl["agent: implement task<br/>(runs repo skills)"]
impl -- succeeded --> review["agent: spec + quality review"]
review -- failed --> fixf["agent: fix findings"]
fixf -- succeeded --> review
review -- succeeded --> pre{"gates: mix precommit<br/>+ mix test.browser"}
pre -- failed --> fix["agent: final fix"]
fix -- succeeded --> pre
pre -- succeeded --> smoke["agent: final review → smoke"]
smoke -- failed --> fix
smoke -- needs_input --> human{{"human answers<br/>(implicit pause — card blocked)"}}
human --> smoke
smoke -- succeeded --> acceptance["agent: acceptance"]
acceptance -- succeeded --> resync["shell: rebase onto origin/main"]
resync -- succeeded --> reverify{"gates: mix precommit<br/>+ mix test.browser"}
resync -- failed --> resync_fix["agent: rebaser (parks on semantic conflict)"]
resync_fix -- succeeded --> reverify
reverify -- succeeded --> merge["shell: push · PR · auto-merge"]
reverify -- failed --> resync_fix
merge -- succeeded --> deploy["shell: await merge + deploy"]
merge -- failed --> resync
deploy -- failed --> ghfix["agent: ci-fixer"]
ghfix -- succeeded --> resync
deploy -- succeeded --> post["agent: post checklist"]
post -- succeeded --> done([done])
A run rebases onto origin/main twice — once before the expensive review/smoke/acceptance
tail and once immediately before merge, each followed by mix precommit and mix test.browser — so a busy board
moving main under a long run no longer strands the work at merge (RLY-192); real conflicts
route to the rebaser agent, which parks for a human on a semantic conflict. After merge queues the
auto-merge, deploy (bin/await_deploy.sh) waits for the PR to merge and main's CI to deploy
it, so a card reaches Review only once its change is live; post runs last so its summary
describes what shipped. The authoritative graph is
docs/designs/flows/code.json.
The RE board: PR-less Code, then a Deploy stage (RE408)
The graph above is the default library, which other boards still run. The RE board (this repo's
own board) overrides two flows, checked in at
.relay/flows/code.json and
.relay/flows/deploy.json and pushed with
./relay flow-push <key> <file>. Neither file has a version, so a push is last-write-wins.
test/relay/flows/re_board_flow_documents_test.exs runs both through
Relay.Flows.upsert_from_document/3, the same path PUT /api/flows/:key takes.
flowchart LR
rebrowser["… rebrowser"] -- succeeded --> merge["shell: bin/ship_to_main.sh<br/>(squash, fast-forward push)"]
merge -- failed --> resync["resync"]
merge -- succeeded --> await_ci["shell: bin/await_main_ci.sh<br/>(main's CI on that commit)"]
await_ci -- succeeded --> post["agent: post checklist"]
await_ci -- failed --> github_fix["agent: ci-fixer"]
github_fix -- succeeded --> resync
post -- succeeded --> codedone[["Code:Done"]]
codedone --> checkout["shell: checkout -B {branch} origin/main"]
checkout -- succeeded --> fly["shell: deploy_fly.sh"]
fly -- succeeded --> ios["shell: deploy_ios.sh"]
ios -- succeeded --> android["shell: deploy_android.sh"]
android -- succeeded --> review[["Review"]]
checkout & fly & ios & android -- failed --> human{{"needs_input"}}
-
Code (
Plan:Done → Code → Code:Done) opens no PR.mergeisbin/ship_to_main.sh {ref}: it squashesorigin/main..HEADinto one<REF> <card title>commit and pushes it tomainas a plain fast-forward, never with force. A rejected push fails intoresync(max 2 loops).await_ci(bin/await_main_ci.sh) then waits forci.yml's push run on that exact commit, re-running failed jobs once for flakes. Only a green run moves on topostandCode:Done, so the Deploy flow never picks up a commit CI hasn't passed. A red one goes togithub_fix, theci-fixeragent (max 2 loops). It commits a fix on top oforigin/main, and the flow loops back throughresyncand the gates to ship that fix as a new commit. The PR-eradeploynode is gone. -
Flutter gate (RE424). The Code flow's
fluttergate (bin/flutter_gate.sh) runs betweenprecommitandbrowser. It diffsflutter/againstgit merge-base origin/main HEADand exits 0 immediately when the branch didn't touch it; otherwise it runsbin/flutter_validate.sh(flutter pub get,flutter analyze, a non-mutatingdart formatcheck,flutter test), and a failure goes tofinal_fix(max 2 loops), which re-enters atprecommit. A detection error validates rather than skips. The gate is not repeated in the pre-merge tail (reverify/rebrowser): a rebase breaks Flutter only when main also changedflutter/, and the Deploy flow'sios/androidstill validate before any store upload. The board runs the pushed flow, so a change to the file takes effect only after./relay flow-push code .relay/flows/code.json(then/relay-doctor, interactively). -
Deploy (
deployflow,Code:Done → Deploy → Review,exclusive, stage WIP 1) only deploys what is already onmain. Its four shell nodes (max_retries: 1, each parking onneeds_inputwhen it fails, with no AI fixer) run on our own runners:checkout:{relay} git-fetch && git checkout -B {branch} origin/main.fly:bin/deploy_fly.sh. This deploy always runs, and it bakesGIT_SHA/BUILT_ATinto the image.ios:bin/deploy_ios.sh {ref}, which goes to TestFlight.android:bin/deploy_android.sh {ref}, which goes to Google Play.
iosandandroidskip with exit 0 when the card's commit didn't touchflutter/(bin/card_touched_flutter.sh), and Android also skips until its credentials exist (RLY-103). -
Secrets come from the runner's own environment, which shell nodes inherit. On the RE board that is
.envrc.local, loaded by direnv in the shell that ranrelay start. Each deploy script fails fast naming any secret it's missing. -
Why
checkout -Band not--detach. On anexclusiveflow the runner re-attaches the card's branch before every node, and it refuses a detached HEAD unless the command containscheckout -B. Pointing the card's branch atorigin/mainkeeps that guard happy and runs every deploy on exactly what is onmain. This is harmless, since the branch's work already shipped as the squash.
CI still runs the tests on pushes to main and on PRs. Its deploy jobs (ci.yml's deploy and
flutter-deploy.yml's push: trigger) are commented out, not deleted, and
flutter-deploy.yml's workflow_dispatch still works for a manual store build.
A card in any AI-enabled stage is dispatched by Relay.Runs.Scheduler (folding over every
enabled Flow on the board, rightmost flow stage position first; each flow pulls from the
stage before its own in board order, worked out by Relay.Flows.neighbours/2) straight to the
node-job engine (Relay.Runs) — no per-stage config file, no board-runner poll loop.
relay start claims the resulting NodeJob rows over the node-job REST API (below) and
runs whatever node it is handed; it knows nothing about stages, columns, or which flow a
job belongs to. Board-specific facts (stages, prompts, per-node budgets) live entirely in
Flow/Flow.Node/Flow.Edge rows, seeded from
docs/designs/flows/ and editable in Settings › Stages (each stage row's FLOW band).
A broken board shape pauses a flow (RE430). Relay.Flows.Shape.problems/2 is the one shape
rule: the column before a flow's stage must be a :queue or :done column, and there must be a
column after it. Scheduler.Server.build_snapshot/2 reads every flow of the board (enabled and
disabled, Relay.Flows.list_flow_snapshots/1 — still one query), keeps the enabled ones as the
snapshot's flows, and computes the snapshot's problems in memory from the same ordered
stages and the full flow list (a disabled flow is still named as an upstream). An enabled flow
with a problem is paused: plan/1 starts no fresh run for it but still resumes its parked
in-flight runs, and a run already going finishes and lands from board order exactly as
before — nothing on the landing or resume path checks the shape, and Relay.Runs.start_run/2
is not gated. The server subscribes to {:stages_changed, board_id} on the board topic and
treats it like a card event (a full, debounced reconcile, never the quiescent skip), so fixing
the board un-pauses the flow on the very next reconcile.
Shared-budget arbitration: rightmost flow wins ties. Relay.Runs.Capacity keys free
slots runner_id => %{shared_clean: n, exclusive: n} per isolation class, not per
flow (capacity.ex:5-7), and Relay.Runs.Scheduler.plan/1 threads one shared capacity
accumulator through its fold, sorted rightmost flow stage (flows.stage_id) position first
(scheduler.ex:38-45, rule documented at scheduler.ex:9-13). So when two flows share an
isolation class and both have eligible cards under scarce capacity, the flow closer to Done
draws first — intended WIP discipline, not starvation, even though under real scarcity it
looks like the leftward flow is being starved. Pinned by
test/relay/runs/scheduler_test.exs and exercised live over the REST API by
test/relay_web/api/plan_flow_e2e_test.exs / test/relay/runs/code_flow_e2e_test.exs.
The capacity store (Relay.Runs.Capacity, ETS) is keyed by runner id; each entry carries
the runner's board id and is evicted by the reaper's reclaim sweep once the runner goes stale
(RE402). The scheduler snapshot is assembled from five narrow reads (stages, the
Cards.list_scheduler_cards/2 projection, active runs, runners, every flow — RE402, RE430) and is
board-scoped: Scheduler.Server.build_snapshot/2
keeps a capacity entry only for a runner of this board that Relay.Runs.counting_runner?/1
accepts (not :gone) — an allow-list, so another board's runner or an orphaned ETS entry
contributes nothing to dispatch or to relay why's capacity evidence (RE338). Every snapshot
capacity key is therefore a runner Relay.Runs.list_runner_status/2 names. What remains
unscoped is one physical machine registered on two boards: that is two runner rows advertising
the same slots, so two boards' schedulers can both count the same free slot. The runner's own
live capacity is the final backstop, so an over-assigned job waits there rather than
double-booking (YAGNI: no multi-board reservation yet).
Side channels
- Log mirror: every feed line is queued to a background
LogForwarderthread that batchesPOST /api/board/logs(best-effort: drops on full queue, swallows all errors) — landing inActivity.LogSink→ the card timeline, andAgentLog→ the live log sheet. - Runner heartbeat:
RunnerHeartbeatposts{runner, capacity, running: [job-ids], held: [{ref, state}], harnesses, sessions}toPOST /api/node-jobs/heartbeateveryheartbeat_intervals (RLY-164) and reads back{revoked: [job-ids], release_held: [{ref, status}], want_capabilities, runner_outdated, required_version, latest_runner_version, harnesses_digest}(the digest since RE433 — see "Harness routing" below). It terminates each revoked job's live subprocess via itsJobControl(see "Node-job transport" and "Runner mode" below). The advertisedcapacityis the runner's configured per-class total and this route is its single writer (RE311) — the claim'scapacityis a live FREE count, passed toRelay.Runs.claim_next_job/4as an argument and never persisted, because one column carrying both meanings made the roster flip-flop between the total and the free count while the board sat deadlocked.runningis the jobs the runner believes it holds, so the server can name the ones it no longer considers live.held(RE311) is every per-card worktree the runner holds, each with astatefromSchemas.Runner.holding_states/0(bound|retained|running|talk);release_heldis its ref-keyed analogue ofrevoked— the subset whose card's runs have all ended server-side, named with the status (done/failed/cancelled,Relay.Runs.releasable_held/2) that chooses remove vs retain. It replaces RLY-218's retired run-id-keyed release channel, which structurally could not see a worktree adopted byrecover()after a restart — itsrun_idis unknown — so a run cancelled while the runner was down leaked its exclusive slot permanently. Since RE337release_heldalso carries human-requested releases: the runners page's Release worktree action appends the card ref to the runner row'srunners.release_requests(Relay.Runs.request_worktree_release/4, accepted only while the runner's latestheldreports the refbound), and the reply (Relay.Runs.release_held/3) adds{ref, status: "cancelled"}— a remove disposition (Schemas.Runner.release_remove_status/0), neverfailed— for each pending request this beat still reportsbound. The run stays parked; on resume the runner'sassign()rebuilds the worktree from the branch. Clearing rule: every beat that reportsheldprunes the column in one atomic UPDATE (upsert_runner/2) — a request survives only while its ref is reportedboundortalk(teardown deferred under a talk turn); absent (torn down) orrunning(the run resumed first) clears it, so a stale request can never tear down a resumed run's tree. A runner that goes silent keeps its requests until it beats again. No runner change was needed:RunnerPool.release_heldalready refuses a live or talk-attached tree and stashes dirty edits asauto-salvage <slot>(never restored). The samerunninglist also refreshes card liveness (RLY-226,Runs.refresh_running_card_liveness/2): the server stampsagent_heartbeat_aton the cards whose reported job is still active, the positive complement of the revoke query, so a live-but-quiet agent never falsely reads:staleinCards.health/1.latest_runner_version(RE185/RE304) is theRUNNER_VERSIONof the./relaythe app itself serves —Relay.Runs.latest_runner_version/0, delegating toRelay.Scaffold.runner_version/0, read frompriv/scaffold/relayat runtime. Truthful by construction. It is a target, distinct fromrequired_version's floor; a runner withauto_updateon updates itself against it. Eachharnesses[]inventory entry carries that harness'susage(RE436) —{source, read_at, windows, paused}, its latest readings and its pause if any — read fresh on every beat (see Usage limits below). RE320's top-levelrate_limitis no longer sent.sessions(RE358) is{"<job-id>": "<session-id>"}— the session each in-flight agent job's stream has captured so far (JobControl.note_session, fed by_stream_agent_job'son_session), sent only when at least one job has one.Relay.Runs.record_job_sessions/2writes each onto that job'sNodeExecution.session_id— board-scoped and id-parsed likerevoked_among/2, and only for an active flow job whose execution has no outcome yet (a beat can race the outcome POST). Anything malformed is ignored; the beat still answers 200. See "Steer now" below for why it exists. - Run ids: each runner worker tags its log lines with the claimed job's
run_id(RLY-112) so a card's timeline can group lines by run.
Observability surface (RLY-177)
Read-only endpoints answering "why isn't this card moving?" without an fly ssh console
Ecto query, board-scoped like every other /api route (a ref/id on another board 404s,
never 403s):
GET /api/cards/:ref/diagnosis(RelayWeb.Api.DiagnosisController.show/2) — one verdict plus the evidence behind it, produced byRelay.Runs.diagnose/3, the boundary-safe facade the web layer calls (Relay.Runs' exports list is unchanged by this surface, sodocs/architecture/domain.mdneeded no edit). It in turn callsRelay.Runs.Scheduler.explain/2, which replaysScheduler.plan/1's real dispatch decision — sharing its predicate functions — rather than reimplementing it, so the verdict cannot drift from what actually dispatches. A card the pulling flow would take while that flow is paused by a broken board shape diagnosesflow_paused(RE430):detailisRelay.Flows.Shape.paused_detail/1andevidence.problemisRelay.Flows.Shape.wire/1of the problem (flow_key,kind,what,why,fixes,columns); every other verdict carriesevidence.problem: null. It is checked afterno_enabled_flowand beforeblocked_by_dependencies, so earlier reasons (human-owned, needs input, a run in flight) still win.GET /api/cards/:ref/runs(RelayWeb.Api.RunController.index/2) — the card's runs newest-first withnode_executionspreloaded, composingRelay.Runs.list_runs_for_card/1.detailandfailure_detailare serialized in full, never truncated — the exact text a failing review's findings need to be readable for.GET /api/runners(RelayWeb.Api.RunnerController.index/2) — composesRelay.Runs.list_runner_status/2(no second runner read): advertised capacity per isolation class, last heartbeat, the tri-statefreshness(Relay.Runs.runner_freshness/2;stale?is thefreshness != :freshconvenience flag),version/outdated(Relay.Runs.runner_outdated?/1— orthogonal to freshness, since a refused runner can still be beating normally),display_state(:gone > :stale > :outdated > :rate_limited > :fresh),rate_limit(Relay.Runs.active_rate_limit/2— set only while every installed harness on the runner is paused, as the pause that lifts first:{harness, window, used_pct, pause_at, resets_at, reason}, nil once it resets; RE320, per harness since RE436),harnesses(the stored per-harness inventory and usage readings,[]for a runner that never reported), and the jobs each runner currently holds.- Web: the Runners view (
/board/:slug/runners,RelayWeb.BoardRunnersLive) — the sameRelay.Runs.list_runner_status/2roster rendered one panel per machine, plus (RE307) the board-wide active queue fromRelay.Runs.list_queue/2: everyqueuedorclaimednode job on the board, both kinds (:nodeand:talk), claimed or not, ordered asRelay.Runs.claim_next_job/1will hand them out, withRelay.Runs.stopped_work/2's verdict above the rows. It renders whether or not a runner is connected — an empty roster is exactly when a stacked queue matters. Read-only and deliberately no new endpoint and no new CLI verb:POST /api/node-jobs/claimstays the only path a job is handed out on, andlist_queue/2is the function a futurerelay queuewould render. Each runner panel's HARNESSES section (RE436) joinslist_runner_status/2'sharnesseswith the board's harness definitions and agents (Relay.Runs.runner_harness_rows/4, read once per refresh): one row per board harness the runner reported, with its version, signed-in state, one usage bar per window the harness's USAGE setting defines and its own paused line. A harness removed from the board stops showing, and a harness the runner has not reported yet is not shown. - Rate-limited roster (RE320, per harness since RE436). A runner pauses per harness, and the
server derives "rate limited" from the readings stored in
runners.harnesses.Relay.Runs.harness_pause/2reads one entry's live pause: the entry isinstalled, itsusage.pausedis set, and that pause'sresets_atis still ahead. A stored pause past its reset reads as not paused before the next beat clears it.Relay.Runs.runner_rate_limited?/2is the one predicate: the runner reported at least one installed harness and every installed one is paused (a nil or[]inventory is not rate limited). A partly paused runner is not rate limited: it still counts as connected and claims its other harnesses' work, and the claim'sharnesseskeeps the paused one's jobs away from it (see Harness routing).Relay.Runs.active_rate_limit/2returns the pause that lifts first, as%{harness, window, used_pct, pause_at, resets_at, reason}, since that is when the runner can claim again;list_board_runners/1selectsharnessesfor it. RE436 retired the runner-widerunners.rate_limitcolumn,Schemas.RunnerRateLimitand the heartbeat's top-levelrate_limit.Scheduler.capacity_diagnosis/1returns:runner_rate_limitedwhen the roster has live runners, not all of them are outdated, and every live, current runner is paused. Its evidence carriesresumes_at(the earliest reset) andrate_limited_runners. The roster-blaming reasons live once inScheduler.roster_blocking_reasons/0.Relay.Runs.stopped_work/2returns the new reason for the board and Runners-page banner (warning tint).Relay.Runs.diagnose/3(relay why,GET /api/cards/:ref/diagnosis) returns verdict:runner_rate_limitedfor a live run whose job is queued behind that roster; its sentence isRelay.Runs.rate_limited_run_detail/1("… every connected runner is paused at its usage limits"). Rate-limited runners are also excluded from:job_awaiting_slot's "connected runners". The card face readsRelay.Runs.roster_rate_limit/2(one runners query through the same diagnosis) intersected withRelay.Runs.queued_run_ids/1on BoardLive's health tick, and showsRate limited · resumes …without the 5-minute stall threshold. The card drawer does not calldiagnose/3(a full snapshot per render); it reads the same per-card:run_face_metaentry, renderingrate_limited_run_detail/1as a Run-tab banner and swapping the Activity tab's "gone quiet" chip for the rate-limited label. Copy comes fromRelay.Runs.rate_limit_phrase/1(codex 5h 93% / 90%, orclaude-code refused (5h)for a refusal) andresume_time_label/1. No new PubSub topic. GET /api/version(RelayWeb.Api.VersionController.show/2) — the git SHA the running app was built from, baked in at image build time (Dockerfile'sfinalstage, fed bybin/deploy_fly.sh'sflyctl deploy --build-argin the RE board's Deploy flow; for a manual or other-board deploy, the commented-outdeployjob in.github/workflows/ci.ymlpasses the same build args when it is restored). Unauthenticated, on the plain:apipipeline — it leaks nothing a deploy does not.GET /api/flows/:key/metrics(RelayWeb.Api.FlowMetricsController.metrics/2) — the per-node rollup for a flow over a?window=7d|30d|allwindow (default30d): ascope("flow"|"card"), the echoedcardref (nullin flow scope), asummarystat band (total_runs,completed,completed_pct,total_spend,median_end_to_end,total_end_to_end) and anodesarray (runs,duration_p50/p95,cost_p50/p95—nulluntil runners report spend —duration_total,cost_total,attempts_mean,verdict_split,loop_laps, and the RE345 idle-gap keyswait_p50/p95,wait_total,wait_count,held_p50/p95,held_total,held_count, and the RE348 span keyswork_total,rework_total,work_count,rework_count,rewind_total,rewind_count). The nodes come fromRelay.Runs.node_metrics_for_flow/2, which merges inRelay.Runs.node_waits_for_flow/2: a node's gap is the time from the previous execution in the same run finishing to this one starting, and it is held when that previous execution endedneeds_input/blocked(Schemas.NodeExecution.holding_outcomes/0) or the card logged aneeds_input/input_answeredactivity inside the gap (an escalation park, whose predecessor staysfailed— RE348), otherwise hand-off wait; a run's first node has none, and the window applies to the execution the gap precedes. The span keys come fromRelay.Runs.execution_spans_for_flow/2: an execution is work when it is in the first visit of its(node, sub_task)binding (so foreach tasks 2..N are work) before that visit's firstfailedattempt, and rework otherwise — later visits, retries after a failure, and every execution of a fix-role node. Rewind (rewind_total/rewind_count, fix nodes only) is the non-fix rework charged to the fix that opened the charge, closed by the next work or fix execution.work_total + rework_total == duration_total. An optional?card=<ref>(RE235) scopes every figure to one card's node executions across ALL of its runs of that flow:?window=is then ignored, the eight percentile keys (duration_*,cost_*,wait_*,held_*p50/p95) andmedian_end_to_endcome backnull(a percentile over one card is noise), and an unknown, unparseable or other-board ref is a404. Read-only, board-scoped.GET /api/flows/:key/audit(RelayWeb.Api.AuditController.audit/2) — run-history health findings for a flow over a?window=7d|30d|allwindow (default30d), composingRelay.Runs.audit/2(Relay.Runs.Audit.findings/2overRelay.Runs.recent_runs_for_flow/2). Returnsflow_key, the echoedwindow, therunscount examined, and afindingsarray ofseverity/check/flow_key/node_key/run_id/summary/evidence/fix. Four checks today:findings_dropped(a foreach cursor advanced past a failed review),verdict_flipped(a retry turnedfailedintosucceededat the samegit_sha),planner_not_migrated(RE368 — one WARNING per flow when a run's tasks were seeded by the legacy plan-parse fallback, read fromruns.tasks_from_plan; names the affected-card count and most recent refs, fix/relay-doctor), andoutcomeless_attempts(RE410 — one WARNING per{run, node, visit}with at leastRelay.Runs.max_outcomeless_reentries/0executions that never reported an outcome: something keeps killing the node's job). The check ids are pinned asvocabulary.audit_checksin the runner contract fixture. The other half ofrelay audit— CI parity — is computed in./relay, because the server has no checkout of any board's repo. Read-only, board-scoped, advisory.GET /api/flows(RelayWeb.Api.FlowController.index/2) — every flow on the board, fully serialized in stablekeyorder. One round trip is allrelay doctor(RLY-240) needs.GET /api/flows/:key(RelayWeb.Api.FlowController.show/2) — one flow as the canonicalRelay.Flows.Document(RLY-241):key,version,enabled,isolation, the trigger as the one stage name the flow works in,{"stage": "<name>"}(portable across boards), and the orderednodes/edgesarrays. Sparse — nil fields and schema defaults are omitted. Beside the document sits a read-only"derived": {"pulls_from", "lands_on"}block (RE429,Relay.Flows.Document.derived/1): the stage names worked out from current board order,nullat either end../relay flowprints it aspulls_from → stage → lands_on. A read-only"problem"(RE430) sits beside it:null, orRelay.Flows.Shape.wire/1of the flow's broken board shape —flow_key,kind,what,why,fixes(action+label) andcolumns(stage_id,name,type,mark)../relay flowmarks an enabled flow with a problemPAUSEDandrelay flow KEYprints its What / Why / Fix verbatim. Every key set, plusShape.kinds/0/Shape.fix_actions/0, is pinned intest/fixtures/runner_contract.json("flows",vocabulary.shape_problem_kinds/shape_fix_actions). 404s an unknown key.PUT /api/flows/:key(RelayWeb.Api.FlowController.update/2) — upsert a flow from a document viaRelay.Flows.upsert_from_document/3, in one transaction:Schemas.Flow.changeset/2's graph validation,save_definition/2's version-bump semantics (an unchanged push bumps nothing — pull → push is a genuine no-op), andenable_flow/1/disable_flow/1's arming rules. The response renders through the same view asGET,derivedincluded.derivedon input is accepted and ignored (a pulled file pushes back unchanged); a legacy three-key trigger is accepted, reading onlyworks_in.201on create,200on update. Refusals:422 invalid_document/key_mismatch/unknown_stages/stage_occupied(the named stage already holds another key's flow —error.stage/error.flowname it) /invalid, and409 stale_versionwhen the document carries aversionthat no longer matches (absentversion= last-write-wins).- CLI:
./relay why REF/./relay runs REF/./relay runners/./relay version/./relay flow-stats KEY/./relay flow [KEY]/./relay flow-push KEY FILE/./relay audit [KEY], documented in../../relay.md.
Bootstrap surface (RE304, ADR 0010)
The board serves the scaffold again. Two routes on the unauthenticated /api pipeline,
alongside /api/version:
GET /api/scaffold(RelayWeb.Api.ScaffoldController.manifest/2) — the manifest:{"version": "<12 hex>", "items": [{"path", "sha256", "bytes"}, …]}; therelayitem also carriesunbaked_sha256(RE443, see Hashing below).GET /api/scaffold/*path(.show/2) — the raw bytes of one manifest entry, 404 for anything else.Relay.Scaffold.fetch/2checks a static allowlist, so this is not a general file server and traversal is impossible by construction.
Unauthenticated because the board's install script (/b/<slug>/install) downloads ./relay
before anything has authenticated, and because these files were published openly regardless.
Exactly five files are Relay-owned (Relay.Scaffold.items/0): ./relay, the three
.claude/skills/relay-{update,doctor,onboard}/SKILL.md, and relay.md. .claude/skills is the
manifest's canonical skill dir (Relay.Scaffold.skill_dir/0, RE457, the ONE Elixir definition);
Relay.Scaffold.items/1 maps the skill items to another LLM's dir (.agents/skills for Codex)
without changing the served manifest. (RE443 retired the
fourth, setup skill: the board's setup wizard hands out the install command instead.) They are
never user-edited, which is what lets ./relay update overwrite them unconditionally — no provenance
ledger, no per-file diff prompt. Everything else in a project (agents, other skills,
AGENTS.md/CLAUDE.md, .relay/runner.json, flow documents) is out of scope and is never
served by this surface; the flow documents and .relay/runner.json are wired by
/relay-onboard, which works step 5 of the board's setup checklist (5.1–5.8). The one local exception
(RE383): when .relay/runner.json is missing, ./relay update renders a starter one from the
runner's own template — no endpoint serves it — and reports it under the config key
(created, missing under --check, or present). It is never overwritten, never part of
the scaffold version or manifest, never touched by auto-update, and not behind the in-app-repo
--force refusal. --base REF sets the base of a runner.json the update creates.
The project's LLM layout (RE457). .relay/runner.json carries two optional, user-owned
keys: skill_dir (default .claude/skills) and pointer_file (default CLAUDE.md) — the
defaults are ./relay's DEFAULT_SKILL_DIR / DEFAULT_POINTER_FILE, pinned against the board by
setup.runner_config_defaults in the runner contract fixture, as ./relay's POINTER_LINE is by
setup.pointer_line (Relay.Setup.pointer_line/0). The manifest keeps its canonical
.claude/skills/… paths; ./relay update maps every item under .claude/skills/ to
<skill_dir>/… (local_scaffold_path) for its comparison, its install and its report — a skill
is still fetched from its manifest path but written, hashed and reported at the local one, and
the report carries skill_dir. ./relay update --skill-dir DIR --pointer-file FILE (either flag
alone works) switches an existing project to another LLM's layout: it rewrites only those two
runner.json keys in place, installs the skills under the new dir, removes Relay's three old skill
files (reported as obsolete, removed on apply — never anything else in the old dir), and
appends the pointer line to the new pointer file (listed in written). It commits nothing; every
path and the runner.json rewrite are checked before anything is written, and the app-repo
refusal applies. With --starter-only, the same flags only set the two keys of a runner.json it
creates. Board Settings → Agents shows the switch command as a flash when "Make default" moves the
board to an LLM with another skill dir (Relay.Setup.llm_switch_notice/2); the board never
changes repo files.
./relay update --starter-only [--base REF] [--skill-dir DIR --pointer-file FILE] (RE443) writes only the user-owned starter
files, each only when missing: that .relay/runner.json and a generic
.relay/prepare-worktree.sh (mode 755, render_prepare_hook() — the copy-from-cache-or-main-
checkout warm-up, generalized over a WARM="" list the project fills in; empty, it copies
nothing and exits 0). No network, no scaffold work, not refused in the Relay app repo, never an
overwrite. Report: {"config": "created"|"present", "hook": "created"|"present", "applied": true}. The install script runs it right after downloading ./relay, before ./relay update.
The served ./relay carries its board's URL (RE443). The committed file has a placeholder
line BOARD_URL = "" right after RUNNER_FINGERPRINT; ScaffoldController.show/2 serves it
via Relay.Scaffold.fetch("relay", board_url: RelayWeb.Endpoint.url()), which rewrites that one
line to BOARD_URL = "<url>" (Relay.Scaffold.bake_board_url/2 / board_url_line/1, the ONE
Elixir regex; ./relay's bake_board_url / board_url_of / BOARD_URL_RE are the ONE Python
parse; the contract fixture's scaffold.board_url_line_example pins the two). The URL is the
endpoint's configured one, never the request's Host, because this text gets executed.
- Lookup:
relay_url()—RELAY_URLwhen set and non-empty, else the bakedBOARD_URL, else it dies pointing at the install command. Every URL-needing call site goes through it; a node's{url}variable uses the non-dyingconfigured_relay_url(). - Hashing: the manifest describes the served bytes —
Relay.Scaffold.manifest(board_url: RelayWeb.Endpoint.url())sets therelayitem'ssha256/bytesto the baked copy's, because every runner up to v89 verifies a download with a raw sha256 against that field, and addsunbaked_sha256, the built placeholder copy's (the build never bakes)../relayv90+ hashesrelaywith itsBOARD_URLreset to""(runner_sha256) and compares it withmanifest_sha(item)—unbaked_sha256, orsha256from a board that predates the key — both forscaffold_statusand for verifying a downloaded body before install; otherwise a baked local copy would read as changed forever. The scaffold version is derived from the built (unbaked) shas, so it is one value per build on every board. The contract fixture pinsunbaked_sha256. - Fingerprint:
FINGERPRINT_MASKmasksRUNNER_VERSION|RUNNER_FINGERPRINT|BOARD_URL, so a baked copy fingerprints the same as the committed one. - Carried across rewrites:
install_runnerkeeps the existing file's non-emptyBOARD_URLover the one it is installing, so neither./relay updatenor auto-update ever drops (or repoints) a project's board.
The install handshake (RE443). GET /b/:slug/install (RelayWeb.InstallController, a
pipeline-free public route) serves a POSIX sh script rendered by RelayWeb.InstallScript.render/2
from lib/relay_web/install_script.sh.eex, for any slug in Schemas.Board.slug_format/0 (whether
or not that board exists; anything else is a plain 404). It bakes RelayWeb.Endpoint.url/0 and the
format-checked slug — never the request's Host — plus the check ids from
Relay.Setup.preflight_checks/0 and the CLIs from Relay.Setup.onboarding_llms/0. Run in a repo
as curl -fsSL <host>/b/<slug>/install | sh with RELAY_API_KEY exported, it:
- runs the local checks (
git_repo—pwdis the work tree's top level;remote—origin;default_branch—origin/HEAD, else the current branch;python3) andcommand -vfor each onboarding CLI, using onlycurl git python3 uname mktemp mkdir chmod mv rm cat grep printf; POST /api/setup/preflight(board key) — the key's first authed call, which ticks setup step 3 and marks step 4waitingorfailedwith the fix text. The answer'sllmcarriesharness/label/cli/skill_dir/skill_path/pointer_file, and (RE457) itsrelay_filesisRelay.Scaffold.items(llm.skill_dir), the five Relay-owned paths under that LLM's skill dir. It prints the board's answer, oneok/!!line per check; on any failure, the fixes andfix_command, then exits 1 having written nothing;- (RE457) prints a "Writing:" preview — one line per
relay_filespath, then the pointer file (only when its line is absent),.relay/runner.json,.relay/prepare-worktree.shand.gitignore (+N lines)— and downloads onlyrelayfrom/api/scaffold/relay(to a temp dir, then./relay, mode 755). It appends the pointer line (Relay.Setup.pointer_line/0, rendered into the script) to the LLM'spointer_fileand the three.gitignoreentries — each only if absent — then runs./relay update --starter-only --base origin/<branch> --skill-dir <llm.skill_dir> --pointer-file <llm.pointer_file>andRELAY_URL=<board> ./relay update --json, which lays downrelay.md, the three skills underskill_dirand.relay/scaffold.json, every body checked against the manifest's sha256. A failed update is recovered by re-running the install; POST /api/setup/installed(board key) withfiles:relay, then every other path in the update report'swritten(once, in report order), then the pointer file,.relay/runner.json,.relay/prepare-worktree.shand.gitignore. That finishes step 4 and activates step 5; the script then printsNext: open <cli> here and run /relay-onboard.
The key travels only in the Authorization header and is never written to a file. Both
/api/setup/* endpoints refuse a key from another board (403 wrong_board); preflight also
refuses a board still on step 2 (409 step_2_pending). A re-run is safe: steps already past are
left alone and nothing is duplicated. A board with no checklist still gets the install (its LLM is
the default agent's), with no checklist writes. RelayWeb.ApiAuth assigns :current_api_key
(from Relay.ApiKeys.authenticate_key/1) for this. Every /api/setup/* route works the key's own
setup (Relay.Setup.target_for_key/1, RE446): a key minted by Connect a machine reports into
that machine's checklist (step 3 reads "minted for <name> · first authed call …"), never the
board's, and a board key never touches a machine's.
Onboarding reports (RE445). After the install, step 5 is worked by the /relay-onboard skill
through two more board-key routes on RelayWeb.Api.SetupController, rendered by
RelayWeb.Api.SetupJSON: GET /api/setup answers {"data": checklist} — the board name,
connect (true for a connected machine's checklist, RE446: the skill then does 5.1 and a
pull-only 5.6 check), the template, the onboarding LLM (llm: cli, harness, label,
pointer_file, skill_dir, skill_path — pinned as setup.llm_keys in the runner contract
fixture, beside RE457's setup.pointer_line and setup.runner_config_defaults), current_id / current_sub_id / current
(Relay.Setup.reportable_id/1, the one step a report may move), first_flow_stage, first_card_stage (the
column that flow pulls from, where 5.8's first card goes), the
position counts, the 5.3 interview catalog, the 5.4 change_decisions and
missing_file_resolutions (Relay.Setup.missing_file_resolutions/0), referenced_files (RE447:
the .claude/ agent and skill files the board's flows name, with their repo paths and used_by
{flow, node} pairs — computed in the controller from Relay.Flows.referenced_files/1 over
Flows.list_flows/1, [] on a template board; the skill lists the missing ones at 5.1 and
resolves each at the start of 5.4, reporting missing files k of N resolved), and every step
(string statuses and actors, ISO 8601 datetimes, step 5's sub-steps in children). POST /api/setup/steps/:step takes {"status": "waiting"|"done"|"failed"|"skipped", "evidence"?, "error"?, "data"?} and calls Relay.Setup.advance/4, answering the same body. Refusals: 404 no_checklist ("no guided setup on this board"), 409 not_current with the current step in
error.current, 422 invalid for a bad body. The board also ticks 5.5 (every flow pushed through
PUT /api/flows/:key and enabled) and 5.8 (a card created in first_card_stage, or in the first flow stage) itself. The
wire shape is pinned by the "setup" section of test/fixtures/runner_contract.json.
The skill reaches those routes only through two ./relay verbs (v91+). ./relay setup status
calls GET /api/setup and prints the checklist (--json / --field give it raw).
./relay setup report STEP --status S [--evidence] [--error] [--data JSON] calls POST /api/setup/steps/STEP with only the fields given. --status choices are SETUP_REPORT_STATUSES,
pinned equal to Relay.Setup.advance_statuses/0 by the contract fixture. --data must be a JSON
object, checked before the request. Exit codes: 0 success; 3
(SETUP_NO_CHECKLIST_EXIT, used by nothing else) on 404 no_checklist from either route — the
skill's signal to stop and point at /relay-doctor; 1 on 409 not_current, with a stderr line
naming not_current and the server's current step, and on any other API failure.
The version is derived, never maintained. Relay.Scaffold.version/1 is the first 12 hex
characters of the sha256 of the sorted "<path>:<sha256>" lines, so it changes exactly when
content changes and cannot be forgotten. The client only compares it for equality; "N versions
behind" is deliberately unsupported.
mix relay.build_scaffold writes priv/scaffold/ (a gitignored build artifact) — from mix setup, from the test alias, and from the Dockerfile after mix compile and before
mix release, because a release ships priv/ but ships neither relay nor .claude/.
Consequence, accepted deliberately: publishing is coupled to deploying. A skill fix reaches projects only when the app ships. In exchange, the whole publish/marker/drift apparatus is gone.
./relay update is the one mechanism for getting those files onto disk. The work list is
the verdict: every item is hashed against the manifest on every run, and "current" means
nothing needs writing — never a version comparison. That is what makes a deleted or edited
file come back, and what keeps the RE185 steady state honest, since auto-update rewrites
./relay in place without touching .relay/scaffold.json, so the marker legitimately lags the
bytes. Applying with an empty work list reconciles the marker; --check reports and writes
nothing, ever; --json on either. The runner rewrites itself through RE185's verified
installer (verify_runner_source + install_runner's atomic os.replace and write ledger),
and every body is additionally checked against the manifest's sha256 before it touches disk.
relay update also removes what an older layout left behind (RE319): a bin/relay that
declares the pre-rename version constant is Relay's, so after installing ./relay the update
deletes it (and bin/ when that leaves it empty) and reports it as removed (--check
reports it as obsolete and writes nothing). A bin/relay without that line is someone else's
and is never touched.
An old bin/relay running the upgrade installs ./relay without the executable bit: its
installer gives mode 755 only to its own path and writes every other item at 644. The one-time
step is chmod +x relay && ./relay update, which also removes the leftover, and the commit must
carry relay as mode 755. From then on the new CLI keeps it that way: scaffold_status counts a
./relay whose bytes match but which is not executable as changed, so relay update reinstalls
it through install_runner and restores 0o755.
Node-job transport (RLY-134, ADR 0006 card 04)
The first slice of ADR 0006's target shape: a pure REST transport on top of the runs engine
(W5, Relay.Runs), board-key auth like the rest of /api, no scheduling/dispatch policy —
that stays server-side.
-
POST /api/node-jobs/claim(RelayWeb.Api.NodeJobController.claim/2) — upserts the advertising runner (a claim doubles as a liveness touch, viaRelay.Runs.upsert_runner/2) then atomically claims the oldest eligiblequeuedNodeJob(Relay.Runs.claim_next_job/1,SELECT … FOR UPDATE SKIP LOCKED). Long-polls up to ~25s on theboard:<id>:runstopic when nothing is immediately claimable (?wait=0short-polls instead); serialises the rawrun+ resolvedvarsW5 already stored, never a worktree path. Eligibility is three-way (ADR 0006 §5, RE311): a job is offered when it is pinned to the requesting runner (runner_name= its name), OR when that runner declares it holds that card's worktree (the job isexclusiveand itsvars.refappears in the claim'sheldwith a state inSchemas.Runner.active_holding_states/0), OR when it fits the advertised free capacity the claim carries. The middle clause exists because the pin is only a proxy for the holding and the proxy can go missing:settle_retry_pin/3's:readoptedbranch releases a pin to a machine that is provably not answering (RE297), the job is inserted unpinned, and the machine then returns and adopts<ns>-<ref>— consuming the very slot the unpinned job needed to be offered through. The bypass is exclusive-only: ashared_cleanjob runs in the shared worktree and must still respect shared capacity. Pinning is persisted on the run:runs.pinned_runner_nameis set when a runner claims anexclusiverun's job (Relay.Runs.maybe_pin_run/2), kept through a:runner_gonepark (so the resume returns to the holder), and cleared by a human-baton park (Relay.Runs.park_claimed/1, so the hand-back resume re-offers anywhere).Relay.Runs.exclusive_holder/2reads that column to pin each successive job, andRelay.Runs.active_runs/1resolves it to the runner row id so the scheduler resumes a parked exclusive run on its holder (RLY-199) — one column, two readers. The runner'snameis therefore a durable, run-affecting key, not a label: renaming a running runner strands every run pinned to the old name (the resume targets a row nothing beats again, andretryrefuses it as "not connected") because runner rows are never pruned. RE305 changed the default name from the bare hostname to<checkout-dir>@<short-host>, so a defaulted runner warns loudly at startup and names the legacy identity — see the Config bullet below. -
Setup side effects (RE446). Both endpoints also drive the board's guided-setup checklist (
Relay.Setup):heartbeat/2callsSetup.observe_heartbeat/2with the authed key and the post-upsert runner, so a board waiting on step 6 ticks it on the first heartbeat (machine and per-harness sign-in as the evidence) — or, for a connected machine's key, that machine's step 6 (recording itshost/runner_name), never the board's; a granted claim callsSetup.observe_claim/2fromRelay.Runs.claim_next_job/4after theFOR UPDATE SKIP LOCKEDtransaction commits, so a board waiting on step 7 finishes setup on its first claim — a setup write never rolls a claim back and the setup board lock never nests in the job lock. Both are no-ops returning:okon every other board (oneexists?query), so neither response changes. (A parked run whose holder advertisesexclusive: 0can still be handed its own resume — the runner keeps polling while it holds bound slots viaRunnerPool.has_bound_slots/0.) -
A process that predates the rename is refused, never served (RE319). The wire's identity key is
runner, renamed as a hard cut with no alias. A claim or heartbeat whose body has norunnerobject but carries the pre-rename identity key gets 409runner_outdated— the same code a merely old runner gets, pinned intest/fixtures/runner_contract.json(claim_refused.outdated) — withrequired: Relay.Runs.min_runner_version/0and a message telling the operator to install./relay(relay update) and restart with./relay start. Nothing is written for it. The check is one private function inRelayWeb.Api.NodeJobController; a body that carriesrunneris current whatever else it carries. -
Version negotiation (RLY-184). Every claim and heartbeat carries
runner.version, theRUNNER_VERSIONthe running./relaydeclares.claim/2compares it againstRelay.Runs.min_runner_version/0and answers 409runner_outdated(withrequiredandrunning) instead of handing out work — claim is the only call that dispenses jobs, so that is the load-bearing check.nilcounts as outdated: a runner sending no version predates the card.heartbeat/2deliberately still succeeds for an outdated runner — the beat is how it stays on the roster and how revokes still reach it — and its reply carriesrunner_outdated/required_version. A refused runner stays alive, advertises{"shared_clean": 0, "exclusive": 0}so nothing queues behind it, finishes in-flight work, and wears anOUTDATEDbadge on the runners view until a human restarts it. Auto-update (RE185). The heartbeat reply also carrieslatest_runner_version, and a runner withauto_updateon downloads that version from the board's/api/scaffold(RE304) and re-execs at a job boundary, so being refused is normally self-healing. The fail-stop below is unchanged and remains the floor: when auto-update is off, refused, or does not take, the runner still stops loudly. -
Paused plan (RE460). When billing is on and the board's owner's plan is paused, every board-key
/api/*call — claim and heartbeat included — is refused 402plan_paused(RelayWeb.PlanPaused, sent byRelayWeb.Plugs.RequireActivePlan), and the board'sScheduler.Serverdispatches nothing../relaymirrors the code asPLAN_PAUSED_CODE(pinned byclaim_refused.plan_pausedin the contract fixture) and never retries the 402. A one-shot command exits 1 withrelay: This board's plan is paused. Manage billing at <board>/account/billing.relay startclaims withsoft_paused=True: it logs onePAUSED: …line on the transition, waitspoll_timeoutbetween claims (the 402 answers instantly, so this is what keeps it from hot-looping), and logsplan active again — claiming resumedon the first claim reply that isn't paused. The heartbeat also posts withsoft_paused=True, so a refused beat counts as landed and starts no failure streak. -
POST /api/node-jobs/:id/outcome(.outcome/2) —Relay.Runs.get_claimed_job/2(board- scoped) returns a three-way result: aclaimedjob runsRelay.Runs.report_outcome/2against the closed outcome set (422unknown_outcomeon a bad value); an already-finalized (:done) job is first-writer-wins — 200 with the run's recordedrun_state, ignoring the resent payload, so a retried outcome POST after a dropped response never turns finished work into a failure (RLY-202); and only a:queued(reassigned) or:revoked(zombie) job answers 409conflict. A run whose server can't start answers 503run_server_unavailable: the outcome is not recorded, the job stays claimed and the run:running, and the runner's idempotent retry re-POSTs it (RE412). The runner'sreport_outcomeis the one API call with a retry budget (RE449): after the usualAPI_MAX_ATTEMPTSit keeps retrying a transient failure (a connection error, a read timeout or a 5xx —_retryable) with exponential backoff capped atOUTCOME_BACKOFF_CAP_Sfor up toOUTCOME_RETRY_BUDGET_S(10 min), logging one line on entering that stretch and one on leaving it, then gives up as before. Throughout, the worker stays inin_flight, so the heartbeat keeps listing the job inrunningand the server doesn't reclaim it; a 4xx (the 409 above, a 422) still fails at once, and every other call keeps theAPI_MAX_ATTEMPTSbound. A job of a run that is not:runningis refused like a revoked one — except a fan-out sibling's while another branch waits on a human (Runs.report_accepted?/2, RE437). The body's optionalresume_at(RE267, ISO-8601 UTC) is read only with outcomeblocked— an unparseable value is dropped to nil, never refused — and makes the engine requeue the node until that reset instead of parking it (failures.md A11b). The node outcomes (including the runner-onlyblocked, RE308) and what each does to the run and the card are tabulated in the state reference. -
Harness routing (RE433).
Relay.Runs.build_payload/4resolves everyagentnode'sllm(nil = the board default) to a board agent withRelay.Agents.resolve/2, and the job payload — hence the node claim — carries top-levelharness({key, command, resume_command, session_id_path}, the template the runner expands),model(the agent's model) andeffort(the node's raw value; the runner drops it for a template without{effort}). All three are null on shell/gate nodes.Relay.Runs.insert_job!/3copiesharness.keyontonode_jobs.harness_key, so the RE267 requeue copy keeps it. A red agent (Relay.Agents.red?/1) or anllmnaming no agent on the board still gets a job, but its payload carriesrefusal(Relay.Agents.red_agent_detail/1/unknown_agent_detail/2):RunServer's one dispatch seam never announces it and records the attempt:blockedwith that detail and noresume_at, so the run parks, the card goesneeds_inputwith the refusal on its timeline, and no retry is spent.refusalis never sent on a claim. The claim filter: a job with aharness_keyis offered only to a runner whose reportedrunners.harnesseslists that keyinstalled: true(Schemas.Runner.installed_harness_keys/1; a runner that never reported is treated as having onlyRelay.Agents.default_harness_key/0,claude-code), and a job whose payload hasrefusalis offered to nobody — both apply to pinned jobs too. Shell/gate/talk jobs have noharness_keyand pass. Inventory + definitions: the heartbeat may carry top-levelharnesses—[{key, installed, version, signed_in, usage}], normalized bySchemas.Runner.normalize_harnesses/1(malformed entries dropped; a malformedusagedegrades to nil, see Usage limits) and absent-means-untouched likeheld— and its reply always carriesharnesses_digest(Relay.Agents.harnesses_digest/1, SHA-256 of the board's definitions in wire shape, free of ids and timestamps).GET /api/harnesses(RelayWeb.Api.HarnessController, board key) returns{harnesses: [Relay.Agents.harness_wire/1 …], digest}; the runner refetches it when the digest moves. The claim's harness set (RE436): a claim may carryharnesses, the keys the runner will run now (installed minus paused at a usage limit), normalized bySchemas.Runner.normalize_claim_harnesses/1and passed toRelay.Runs.claim_next_job/4, which narrows the installed keys to that intersection. Like the claim'sheldit is request-scoped and never written to the row; an absent one leaves the claim unnarrowed. Each harness carries its usage rule (RE436) asusage—Schemas.HarnessUsage.wire/1:{source, match, command, interval_min, used_unit, resets_unit, windows: [{label, used_path, resets_path, pause_at}], pause_now: {when, resets_path} | null, recheck: {model, interval_min} | null},sourceone ofSchemas.HarnessUsage.sources/0(output/command/none). Paths and conditions are a jq subset evaluated bySchemas.HarnessUsage.Ruleand, line for line, by./relay'susage_readingand friends; both pass the contract fixture'susage_rule_cases. Seeds: Claude Code reads its stream'srate_limit_event(5h / 7d), Codex polls its newest rollout file'stoken_countline (5h / wk), Gemini CLI isnone; theAddUsageToHarnessesmigration backfilled existing boards' seeded harnesses from a frozen copy. TheAddHarnessRoutingmigration pointed already-queued/claimed agent jobs at a frozen Claude Code harness with modelopus. -
Talk rides the same claim, a different transport (RE268 / ADR 0009). Every
POST /api/node-jobs/claimreply now carrieskind("node"or"talk"), so the runner can branch without a second endpoint. A"talk"claim carries exactly{id, kind, ref, turn_id, prompt, author, branch, seed, resume_session}—RelayWeb.Api.NodeJobController.claim_payload/1's talk-only clause — never the flow shape'srun_id/node_id/vars. A talk turn's outcome does not go throughPOST /api/node-jobs/:id/outcome: that route finalises a job through the run lifecycle viaRelay.Runs.get_claimed_job/2, which is flow-only (kind in NodeJob.flow_kinds()) and 404s a talk job on purpose. InsteadRelayWeb.Api.TalkController, board-scoped by the same board-key auth, offers two routes:POST /api/talk/turns/:id/eventsappends a batch of transcript lines (at-least-once — a replayedclient_seqis accepted and stored once, perRelay.Talk.append_events/2) andPOST /api/talk/turns/:id/outcomeends the turn (done/stopped/failed, 422unknown_statuson anything else) viaRelay.Talk.finish_turn/3. -
The wire contract is pinned by a fixture.
test/fixtures/runner_contract.jsonis generated from these routes bytest/relay_web/controllers/api/runner_contract_test.exs(never hand-edited) and read bybin/test_relay.py, so both suites assert against one shape instead of each side's idea of the other (RLY-176). Renaming a claim field, or changing what the outcome/heartbeat bodies carry, breaks CI on the next run. Regenerate withRELAY_WRITE_CONTRACT_FIXTURE=1 mix test test/relay_web/controllers/api/runner_contract_test.exs, which rewrites the file and still fails so the diff gets reviewed. -
Runner heartbeat superset.
BoardController.heartbeat/2's/api/board/heartbeatroute carries an independent, additive branch: a beat carryingname+capacitycallsRelay.Runs.upsert_runner/2, writing/refreshing a durableSchemas.Runnerrow ({board_id, name}, capacity map,last_heartbeat) — the durable row is what feeds the Runners view (RLY-167). A capacity-less beat never touches theRunnertable. -
Capability inventory (RLY-182). The runner heartbeat may carry an optional
capabilitiespayload —{"agents": [...], "skills": [...]}, the names this machine can actually resolve from its repo.claude/, the user-level~/.claude/, and the CLI's built-in agents../relay'scollect_capabilities()enumerates BOTHskills/<name>/SKILL.mdandcommands/<name>.md, because a slash command can live in either (/write-planlives only incommands/). It rides send-on-change, not every beat: the runner hashes the inventory each beat and includes the key only when the hash differs from the last successfully-acknowledged send, so a failed POST retries on the next beat. The server persists it onrunners.capabilities, where null means never reported and is deliberately distinct from{}(reported, and empty). Because a beat that omits the key must never erase a stored value,Relay.Runs.upsert_runner/2builds itson_conflictreplace list dynamically. For the case where the server genuinely has none (recreated row, or a runner predating this change), the heartbeat response carrieswant_capabilities: true; the runner clears its cached hash and resends on the next beat.Relay.Runs.preflight_flow/1reads the stored inventory. -
Runner liveness + reclaim.
Relay.Runs.RunnerReaper(supervised, seeruntime.md) periodically callsRelay.Runs.reclaim_stale_runners/0. A runner that goes quiet is not the same as a runner that is gone, so the sweep applies three liveness windows (RE449), each defined once onRelay.Runsand measured from the runner'slast_heartbeat:Window Length Effect once silent past it stale threshold ( Relay.Runs.runner_stale?/2)max(60s, 2 × interval)the runner stops counting: freshness :gone, itsRelay.Runs.Capacityentry is evicted (Capacity.clear/1), it is not offered work. No job is touched.Relay.Runs.shared_reclaim_after_s/03 min its in-flight shared_cleanjobs go back toqueued, unpinned, so another runner can take themRelay.Runs.exclusive_reclaim_after_s/010 min its exclusiveruns are parked (Relay.Runs.park_for_reclaim/1,parked_reason: :runner_gone) and their jobs revoked — never requeued, since an exclusive run is pinned to one runner's worktreeEach reclaim window is applied as
max(window, the runner's stale threshold), so a long-interval runner is never reclaimed before it is even stale; every comparison is a strict>. Shared work is reclaimed sooner because another runner can pick it up; an exclusive run can only resume on the same machine, so it waits longer. The sweep stays idempotent (its active-state query skips work an earlier sweep already reclaimed).Reattach for free. Inside a window the run stays
:runningand the job:claimedby that runner, so a runner back from a network blip simply resumes beating: its heartbeat lists the job inrunningand nothing happens — no new attempt, no second execution, the agent is never killed. A runner that restarted during the outage and lost the job in-process is handled by the RLY-170 orphan path below (Relay.Runs.requeue_orphaned_jobs/3): its first beat omits the job, so the job is requeued — pinned back to that runner when exclusive, unpinned when shared."Is it live?" checks (
working_run_ids/2, the roster's freshness, preflight, the scheduler's counting filter) all stay on the stale threshold. A:gonerunner's advertised capacity is dropped from the scheduler snapshot (Scheduler.Server.build_snapshot/2, which keeps only this board'sRelay.Runs.counting_runner?/1runners — RE338) — afterreserve_active_runshas debited its still-:runningheld runs — so the planner never resumes a pinned run onto a machine that has stopped counting; without this a parked exclusive run oscillates resume↔reap forever andrelay whymisreports it as "dispatchable" (RLY-199). The same reaper tick also callsRelay.Runs.close_orphaned_runs/0— a companion sweep, not a runner-liveness check — closing any run still active while its card already sits in a terminal-type stage (RLY-233) or has been archived (RE335 — the board's Archive button is unguarded, and an archived card's parked run would otherwise hold itsexclusiveworktree forever). This is safe to treat as an unambiguous leak because run dispatch (Relay.Runs.start_run/3) now moves the card into the flow's work lane and inserts the run row in one transaction: no committed state ever has an active run sitting on a terminal-type stage except a genuine leak. The tick's third sweep,Relay.Runs.abandon_unresumable_runs/0, applies the clock to the refusals the scheduler recorded: a parked run whose resume has been refused continuously forRelay.Runs.unresumable_after_s/0(30 minutes) is failed, so a run the planner keeps allowing but can never place becomes a visible failure a human canretryrather than a silent forever-wait (RE297). -
Log
node_job_idconvergence.POST /api/board/logsentries may carry an optionalnode_job_idalongsiderun_id— same nullable-string shape, not an FK. It rides throughRelay.AgentLog.stamp/1→Relay.Activity.LogSink.row/2→activities.node_job_id, kept for W6's run panel to key log lines off a specific node-job. -
The full outcome-file contract (
RELAY_NODE_OUTCOME) runners must honor is Declaring an outcome below.
The foreach cursor (RE252)
A foreach node (foreach: card.tasks) iterates the card's tasks — the sub_tasks rows — and
every execution under it carries the id of the task its iteration is bound to
(node_executions.sub_task_id). The job payload hands that binding to the node as the vars
task (title) and task_id (id), which a prompt references as {task} / {task_id}. These are
the canonical flow-contract names since RE367: the legacy spellings card.sub_tasks,
sub_tasks (in reads/writes) and {sub_task} / {sub_task_id} are rewritten by
Schemas.Flow.Node.normalize_legacy/1 wherever a flow enters (Schemas.Flow.Node.changeset/2,
Relay.Flows.Document.decode/1), and the RE367 data migration rewrote every stored flows /
flow_versions row, so the engine only ever sees canonical names.
"Which task is next" is derived, never persisted: Relay.Runs.next_sub_task_id/1 returns
the first sub_task in position order whose done is false, so a crashed-and-resumed run
recomputes the same answer with no cursor column.
The card drawer reads the binding, not the cursor: Relay.Runs.in_flight_sub_task_id/1 returns
current_sub_task_id/2 for the foreach node of the card's active run (nil with no active run, no
flow row, or no foreach node), and RelayWeb.BoardLive recomputes it on drawer open and on every
card/run refresh to auto-open that task with an AGENT IS HERE chip (RE356). It is a read; no
engine path calls it.
Those sub_tasks rows are also addressable one at a time (RE355):
GET /api/cards/:ref/tasks (summaries — id, title, done, position, never body),
GET /api/cards/:ref/tasks/:id (adds body), POST /api/cards/:ref/tasks
({"tasks": [{title, body}, …]} — one atomic append after the last task, argument order),
PATCH /api/cards/:ref/tasks/:id (title / body only) and DELETE /api/cards/:ref/tasks/:id
(later tasks move up one), wrapped by relay tasks add|list and relay task show|update|rm. None
of them replaces the set, so a surviving row keeps its id — and every node_executions.sub_task_id
bound to it. done is not writable there; the toggle route below owns it. Nothing in the engine
calls these routes yet: Relay.Runs.PlanTasks still seeds the rows at run start.
But done is not a private engine field. Three writers set it: the loop tail's check-off
(RunServer.check_off_sub_task/3), the card drawer's checkbox, and relay check <ref> <id>
(PATCH /api/cards/:ref/sub-tasks/:id) — which any agent in any node can call. So the derived
cursor advances on exactly one thing: a when: :foreach_remaining edge, the flow author's
explicit "next iteration". RunServer.binding_for/5 keys off the guard the engine reports
(Engine.decide/4 returns {:transition, target, guard}), never off the target node:
| Route into the next node | Binding |
|---|---|
when: :foreach_remaining edge |
the derived cursor — the loop tail already checked the finished task off inside the same transaction |
when: :foreach_exhausted edge |
nil — the run has left the loop |
| unguarded edge into the foreach head | inherit, falling back to the derived cursor when unbound (first entry from outside the loop, e.g. branch → implement) |
anything else, including a {:retry, node} or a {:requeue, node} (RE267) |
inherit |
The third row is the one that matters. A review's failure loop-back
(spec_review/quality_review --failed--> implement) is not an advance: it re-enters the
same iteration with the reviewer's detail attached as findings, whatever any other writer
did to done in the meantime. Keying off the target node instead — as the engine did before
RE252 — made a failed review re-derive the cursor, so with the in-flight task already checked
off the re-run landed on the NEXT task, that task's findings were delivered to nothing, and the
run marched on to final_review and smoke with the branch looking clean.
done stays writable while a run is live, deliberately: a human un-checking a box mid-run is a
legitimate escape hatch. The residual risk is confined to the success path, where an external
write can make Relay.Runs.remaining_sub_tasks/1 under-count and exit the loop early. The
failure path never consults done at all.
Fan-out regions (RE437)
A fan_out node fires every outgoing edge at once; each edge starts a branch, and the paired
fan_in (Schemas.Flow.regions/1) waits for all of them. Both are control nodes
(Schemas.Flow.Node.control_types/0): RunServer resolves them as node_executions rows with
an immediate outcome and no job.
- Entering a region — a
{:transition, fan_out, _}, or a start edge into one (Runs.start_run/3, which then starts the server with{:dispatch, job_ids}) — records thefan_out:succeeded, inserts onerun_branchesrow per branch (Schemas.RunBranch, keyed by thatfan_outexecution and the branch key,:queued) and starts the firstmax_parallelby position.run.current_nodestays thefan_outkey while the region is open. - Starting a branch sets its row
:runningand enqueues the branch key's node withnode_executions.run_branch_idset. Itsprior_detail/findingscome from the last outcome-bearing non-branch execution before thefan_out, re-derived from history, so a branch started later from the queue gets the same vars. - Inside a branch,
Engine.decide/4sees only that branch's history, somax_retries,max_loops, the visit cap and the failure-signature breaker count per branch (the retry findings and commit-guard baselines scope the same way). A transition to thefan_inends the row with the node's outcome; a{:fail, _}ends it:failedwith one:failuretimeline entry, and the run carries on. Each ended row frees a slot for the next:queuedone; a queued branch the live flow no longer has ends:failed(branch removed from the flow). - Joining — once every row is
:ended— records thefan_inwithEngine.join/1's result::succeedediff every branch succeeded, and one## <branch>: <outcome>section per branch. Afan_incarrying foreach guards is the loop tail and checks the task off. It is then decided and routed like any other execution, so{prior_detail}/{findings}reach the next node. - Several active jobs. A run inside a region holds one job per running branch
(
Runs.active_jobs/1). Claiming an exclusive run's first job pins the run and its queued siblings to that runner.cancel_run/2andclose_run!/3end every open branch (Runs.end_open_branches!/1);Runs.open_region/1returns the latest entry's rows while any is open.diagnose/3reports the first active job plusactive_job_count. - Parking with siblings. A branch that parks (a question, an escalation edge, a
:blocked) sets its row:parkedand parks the run (park_effects/2— card blocked,parked/needs_input), but revokes no sibling job. While the run is parked,Runs.report_accepted?/2still takes a report for a job in a:runningbranch: it is decided and routed inside that branch (a next node's job is enqueued and dispatched, an ending ends the row, a park parks it — the card is only re-blocked, never asked twice), but no:queuedbranch starts and no join happens.RunServerstarts for such a run only in:attach(Runs.attachable_while_parked?/1) and stops after each report, so a parked run holds no process;Runs.resume_run/2also stops any lingering server before itsparked → runningtransition. - Re-entry inside an open region. Every re-entry mode (
{:reenter, _},--at, advance) withRuns.open_region/1non-empty goes branch by branch in one transaction: a:runningrow with an active job is left alone; a:runningrow without one (restart, hand-back, lost runner) and every:parkedrow get a fresh attempt of the same visit with the branch's own last failure asfindings— a parked row resuming its own:needs_inputsession, never the run-level one — then the region is scheduled. Jobs are revoked per row, never run-wide. The RE410 outcome-less cap counts per branch and parks just that branch (and the run).Runs.boot_mode/1always re-enters a run with an open region. With no region open, re-entering at afan_out, a branch node or afan_inopens a fresh region at thefan_out;Runs.retry_run/2maps such a target to thefan_outitself.
Branches share the run's checkout: concurrent edits, commits and builds are the flow author's responsibility — Relay neither guards nor forbids them.
Run recovery surface (RLY-189)
A terminally failed run can be re-entered by a human — the branch, worktree, execution
history and runner pin all survive, because retry revives the dead run in place rather
than starting a new one. Re-entry (RunServer.handle_continue({:reenter, _})) never consults
the flow's start edge, so the Code flow's destructive branch node is unreachable from a
retry by construction, and finished commits cannot be thrown away.
Re-adoption (RE297). Deleting a flow nils runs.flow_id on every run of that row, and the
reaper now fails such a run outright (D6 in failures.md) — so retry is the hatch a
human reaches for on exactly that shape, and refusing it no_flow while the board plainly shows a
flow working the card's stage was the dead end's last link. When flow_id is nil, retry re-adopts
the enabled flow whose work lane is the card's current stage (Relay.Flows.working_flow/1 — the
same lookup rejection re-entry uses) and re-enters at that flow's start node, since the node
this run died on need not exist in a replacement graph. Two things follow only in this case: the
run's flow_id/flow_key are rewritten to the adopted flow, and a pin to a dead runner is
released rather than refused on (settle_retry_pin/3) — honouring it would revive the run straight
back into pinned_runner_absent, refused every tick until the reaper failed it again. A pin whose
machine is alive is kept: the worktree really is still there. no_flow survives for the genuinely
unresolvable case — no enabled flow works the card's stage, so there is nothing to re-enter.
POST /api/runs/:id/retry(RelayWeb.Api.RunController.retry/2) — id-addressed.POST /api/cards/:ref/retry(.retry_card/2) — ref-addressed alias resolving the card's most recent run; whatrelay retry <ref>calls. Both take an optional{"at": "<node_key>"}body and funnel intoRelay.Runs.retry_run/2.POST /api/board/restart-stalled(RelayWeb.Api.RunController.restart_stalled/2) — bulk-revives every restartable run on the token's board (RLY-228), returning200 {"data": {"status": "ok", "restarted", "refused"}}. Board-scoped by the bearer token. Cards already in a terminal-type stage (Relay.Schemas.Stage.terminal_types/0) are excluded before this runs — a finished card is not stalled (RE247) — and are not countedrefused.
Success is 200 {"data": {"status": "ok", "run_id", "node", "retries"}}. A refusal is
422 {"error": {"code", "message"}} where code is one of not_failed, awaiting_answer,
active_run_exists, no_flow, unknown_node, runner_unavailable — the message names the
specific status, node key or runner that blocked it. An unknown run/card, another board's
run, or a card that has never run is 404. RLY-228 widened retry to an escalation park —
a :parked/needs_input run that Relay.Runs.park_kind/1 classifies :escalation (its latest
execution reported something other than :needs_input, so a node failure was routed to a human,
not a question asked of one). A :question park is refused awaiting_answer instead. See
"Telling A1 from A4" in failures.md for the classifier — park_kind/1 is the one
place that distinction lives; do not re-derive it here.
A successful retry also clears the card's block: revive_run/4 calls clear_card_block/2, so a
card sitting :failed or :needs_input behind the retried run returns to its unblocked status
and the board stops showing a blocked card whose run is already live again.
The guard is split, because worktrees and branches are runner-side state Phoenix cannot
see. Server-side, the endpoint refuses up front for the six reasons above — including an
exclusive run whose pinned runner is absent or stale per Relay.Runs.runner_stale?/2,
whose worktree is unreachable. Runner-side, branch existence stays with RLY-166's
check_branch_attached and RLY-173's reattach_branch; a retried job whose branch was
deleted fails there with a clear message. Neither half ever silently restarts from
origin/main.
CLI: relay retry <ref> [--at NODE] [--json]. On a refusal it prints the server's message to
stderr and exits non-zero.
Cancelling a run (RE309)
The stop half of the same surface. Relay.Runs.cancel_run/2 stops the run server, revokes
any in-flight job (freeing the runner slot), transitions the run from
Relay.Schemas.Run.active_statuses/0 to :cancelled, logs an :action timeline entry and
broadcasts :run_finished. Both running and parked runs cancel, with no extra flag —
restricting it would leave a wedged running run needing a production console, which is
the hole this exists to close.
POST /api/runs/:id/cancel(RelayWeb.Api.RunController.cancel/2) — id-addressed.POST /api/cards/:ref/cancel(.cancel_card/2) — ref-addressed alias resolving the card's active run viaRelay.Runs.active_run/1(retry's alias resolves the newest run instead, whatever its status); whatrelay cancel <ref>calls. Both take an optional{"reason": "…"}body and funnel intoRelay.Runs.cancel_run/2.
Success is 200 {"data": {"status": "ok", "run_id", "ref", "previous_status", "node"}},
where previous_status and node are the run's values before the cancel. The single
refusal is 422 {"error": {"code": "no_active_run", "message"}}, covering both shapes of
"nothing to cancel" — no run row at all, and a run that raced to terminal between lookup and
cancel — because they are the same fact to the caller. A card with no active run therefore
gets a named 422, never a silent success. An unknown ref or run id, or a run on another
board, is 404.
A :reason is folded into the timeline text as run cancelled — <reason>, trimmed and
capped at @max_cancel_reason (200) characters; omitting it keeps the plain
run cancelled. The :actor (:agent | {:user, id}) is logged with the entry, so
"who killed this run?" is answerable: BoardLive's confirm-move dialog passes the
signed-in user, the board-key API passes :agent, and the run Listener and
RunnerReaper take the :agent default.
Cancelling never moves the card. A caller that wants the card elsewhere follows with
POST /api/cards/:ref/move, whose 409 would_strand_run keeps meaning exactly what it
says. That two-call sequence — relay cancel REF && relay move REF Review — is the
supported way to unstick a card whose run must die.
CLI: relay cancel <ref> [--reason TEXT] [--json]. On a refusal it prints the server's
message to stderr and exits non-zero. Self-cancel is permitted and documented, not guarded:
a flow node cancelling its own ref revokes its own job, and the runner handles revocation
gracefully.
Advancing past an already-committed task (RE310)
A foreach node re-entered onto a task whose work is already committed has no exit: the
commit guard rewrites its succeeded to failed (it cannot move HEAD — its own earlier commit
IS the work), the failure edge loops it back or parks it, and no answer a human can type changes
either. Relay.Runs.advance_foreach/2 is the hatch: check the bound sub-task off through
Relay.Cards.set_sub_task_done/3 and revive the run at the foreach head re-bound to the next
task, or — when that was the last one — at the flow's when: :foreach_exhausted target.
POST /api/runs/:id/advance(RelayWeb.Api.RunController.advance/2) — id-addressed.POST /api/cards/:ref/advance(.advance_card/2) — ref-addressed alias; whatrelay advance <ref>calls. Same success and refusal shapes as retry, and the same board scoping (another board's run is a404).
Eligibility is deliberately wider than retry's: a :failed run, or a :parked run with
parked_reason: :needs_input — either park kind. RE306's actual state was a :question park,
which retry refuses awaiting_answer, so reusing retry's rule would leave this hatch unable to
open in the exact state it exists for. :running, :parked/:runner_gone, :done and
:cancelled are refused. The refusal codes add not_advanceable, no_foreach,
no_exhausted_edge and not_bound to retry's list, through the same
Relay.Runs.retry_refusal_code/1 pair; every refusal is checked before the check-off, so a
refused advance writes nothing.
The engine change is one thing: RunServer's re-entry takes a binding MODE. Every existing
re-entry inherits the node's current binding (which is what keeps a review-failed loop-back on
the SAME task, RE252); the new {:advance_foreach, _} start mode re-derives it from
Runs.next_sub_task_id/1, which — the stale task having just been checked off — is the next one.
It is the only caller allowed to re-derive it.
CLI: relay advance <ref> [--json]. UI: a run-advance button on the run panel's :failed,
:circuit and :parked banners, rendered only when Relay.Runs.advance_foreach_available?/1
holds — the LiveView computes that predicate and the component reads a boolean.
Scheduler / Listener authority split (RLY-200)
Before either owner below gets a say, Relay.Runs.Listener.reconcile_card/2 runs a first,
unconditional rule: a still-active run (running/parked, any parked_reason) that is a leak —
its card already sits in a terminal-type stage (Schemas.Stage.terminal_types/0) or has been
archived (RE335; the Listener reacts to {:card_archived, card}) — is closed via
Relay.Runs.cancel_run/2 with the reason Relay.Runs.leak_reason/1 returns, not resumed
(RLY-233). This is what stops a parked :needs_input run from being resumed after its card
reached Done or was archived — closing pre-empts the resume rules below. An archived card with no
active run triggers no rule at all, so it is never re-entered after a rejection.
Relay.Runs.RunnerReaper's 30s sweep (Relay.Runs.close_orphaned_runs/0) is the companion
catch-up for anything the event path missed. Run dispatch (Relay.Runs.start_run/3) moves the
card into the flow's work lane and inserts the run row in one transaction, so no committed state
pairs an active run with a terminal pull stage — and the leak itself is judged from a single
run → card → stage snapshot (Relay.Runs.leak_reason/1), never a card stage read apart from the
run, so a concurrent Spec:Done → Plan dispatch between two reads can't get a freshly-dispatched
run cancelled (RLY-233 / RE239). No grace window is needed.
A parked run has exactly one process allowed to resume it, keyed off Schemas.Run's
parked_reason:
parked_reason |
owner | resume shape |
|---|---|---|
:needs_input |
Relay.Runs.Listener |
same node, WITH the stored session (--resume) |
:claimed |
Relay.Runs.Listener |
fresh (a human may have changed anything) |
:runner_gone |
Relay.Runs.Scheduler |
capacity-driven re-dispatch |
nil / unknown |
nobody | left untouched (mirrors the Listener's own fallback) |
Resuming is not the only way a park can end. Relay.Runs.RunnerReaper is the one process
allowed to end a park without resuming it: when the scheduler has refused to resume a run
continuously for Relay.Runs.unresumable_after_s/0 (30 minutes),
Relay.Runs.abandon_unresumable_runs/1 fails it (RE297). That give-up path is scoped to the
refusal clock — it never pre-empts the owner above, which still holds the resume itself.
Relay.Runs.Scheduler.resume_runs/5 filters on this before its existing human/needs-input
guards, so it never even considers a Listener-owned park; Scheduler.explain/2 mirrors the
same split, surfacing :awaiting_listener_resume instead of :awaiting_capacity for a
Listener-owned park so relay why stays honest. Without this split, both processes reacted
to the same card event and whichever won decided whether a resumed agent kept its Claude
session — the loser silently re-wrote the run underneath the winner.
Every running ↔ parked writer (Relay.Runs.resume_run/2, park_for_reclaim/1,
park_claimed/1) is a guarded Repo.update_all keyed on the run's current status (the same
pattern transition_job/3 already used for job-state writes), so a lost race — whichever
process loses — returns a detected no-op ({:error, :not_parked} for resume) rather than a
second silent write.
Listener boot sweep. Relay.Runs.Listener.init/1 reconciles every card holding a
:parked run at startup ({:continue, :boot_reconcile}), so an answer or hand-back that
arrived while the Listener was down is not lost — the scheduler is no longer a backstop for
:needs_input/:claimed parks the way it was before this split.
Runner mode (relay start) (RLY-135, ADR 0006 card 05)
./relay start is the only runner mode: a thin, board-agnostic client of the
node-job transport above. It knows the Relay REST API and how to execute a node-job;
nothing else — every board-specific fact lives server-side as flow data.
Every iteration of the loop is the same three steps:
- Claim the next node-job from the server (a long-poll — cheap when idle).
- Run it — an agent node runs its harness's CLI headless (Claude Code, Codex, … — RE433);
shell/gatenodes run shell. - Report the typed outcome back to the server, which advances the flow (moving the card to the next stage when the flow lands there).
Step 3 is a contract a node author has to honor — see Declaring an outcome below.
Agent steps run headless Claude, which uses whatever authentication the local Claude CLI has — a
Claude subscription (no ANTHROPIC_API_KEY needed) or, if ANTHROPIC_API_KEY is set, the
metered API. Subscription rate limits are the ceiling; when hit, the step is throttled, not
silently billed to the paid API.
-
Config.
.relay/runner.jsonholdsname(defaults to<checkout-dir>@<short-host>, e.g.relay@Jeremys-MBP; override per invocation withrelay start --name foo). Upgrading past RE305 changes that default, and because the name is the exclusive-affinity pin key (see Node-job transport above), runs pinned under the old bare-hostname identity park:runner_goneinstead of resuming. The hop happens unattended — auto-update re-execs at a job boundary, which is "nothing in flight", not "no run pinned to me" — so a defaulted runner prints aWARNING:at startup naming the pre-RE305 identity. It is gated on evidence the identity actually moved on this machine: an identity lock file for the bare hostname (acquire_singleton_lockwrites one and never unlinks it), so a fresh install that never ran a pre-RE305 runner starts silent. Because that lock is never unlinked the gate stays true forever once it is true, so emitting also drops a.re305-warnedmarker beside the new identity's lock: the line is said once per (machine, board, new identity), not on every start and every re-exec. Keyed on the new name because the legacy one is the bare hostname and so machine-global — keyed there, the first checkout to start would eat the only telling and leave the others (often the one actually holding the pins) silent. A--dry-runprints the line but does not spend it. To adopt those pins, restart the one checkout that owns them with--name <hostname>— per invocation, not as a"name"key in.relay/runner.json, which is tracked in git and shared by every checkout, so a bare hostname there restores exactly the shared identity this default exists to split apart. Per run the recovery depends on where the run already is: while it is still running, a human baton clears the pin (Runs.park_claimed/1nilspinned_runner_name, and it transitions from[:running]only), so handing the baton back re-dispatches it anywhere; once it has parked:runner_gonenothing clears the pin in place — the listener leaves that shape untouched, the scheduler's resume still targets the gone runner, andrelay retryrefusesrunner_unavailable— so the way out is cancelling the run and letting the card dispatch fresh, losing that run's worktree state. The board already saysRunner "X" is not currently connected.on the parked run. Alsonamespace(defaultexec),capacity: {shared_clean, exclusive},base,poll_timeout,heartbeat_interval, and three optional per-card-worktree keys (RLY-231):cache_dir(a warm dep/build cache dir passed to the lifecycle hooks),worktrees(RE339 —{"prepare": path, "cleanup": path}, the project's per-worktree lifecycle hooks, defaulting to.relay/prepare-worktree.shand.relay/cleanup-worktree.sh; a flat top-levelprepareis still read as a deprecated alias forworktrees.prepare, which wins when both are set), andmax_retained_failed(how many failed-run worktrees to keep for post-mortem before the oldest is evicted, default 3), and two auto-update keys (RE185):auto_update(defaulttrue) andauto_update_min_interval(seconds between update attempts, default 300). Missing file → sensible defaults, including the auto-update keys; capacity is the field a developer routinely edits. The loader (load_runner_config) strips full-line//comments (first non-whitespace characters//) before parsing, keeping each stripped line as a blank line so JSON error line numbers match the file; trailing//and/* */are not supported, and a string value containing//is never altered../relay updatecreates the file if missing (RE383,ensure_runner_config):namespace= the sanitized checkout-dir name (fallbackproject),cache_dir=~/.cache/relay/<namespace>, starter capacityshared_clean3 /exclusive2, theworktreesblock commented out (PAUSE AT is per harness, set in Settings → Agents, so there is nolimitskey — RE436), and nonamekey (a tracked name would recreate the RE305 identity collision). The startup line namesnamespace <ns>.baseis the trunk every worktree is baselined to — the ref a worktree is created at, hard-reset to on re-baseline, refreshed to when the shared tree goes idle, and handed to the prepare hook asRELAY_BASE. Defaultorigin/main; set it toorigin/master,upstream/main, or a long-lived integration branch when that is your trunk. It does not govern where a card's branch starts — that start point is written into the flow'sbranchnode (git checkout -B {branch} origin/main) and has to be changed in the flow.Running more than one
exclusiveslot? Concurrent runs each work in their own worktree, so make sure they don't share mutable state — most importantly, give each run its own test database (or equivalent) so parallel test suites don't truncate each other. How you do that depends on your project's toolchain (the prepare hook below is where a project wires per-worktree isolation). -
What the terminal tells you (RE305). Startup prints ONE line naming the runner, its version, the board it reached (display name + key), the URL, and the capacity it advertises — so a key pointed at the wrong board is visible immediately instead of looking identical to a correct one. If the board cannot be reached at startup the line says
board UNREACHABLE, aWARNING:line names the URL and the underlying error (exactly the diagnostic for a wrong or expiredRELAY_API_KEY), and the runner keeps polling — a transient outage at startup must not kill a long-running process.In the loop it prints
claimed <REF> · <node> (run <id>, <isolation>)(orclaimed talk turn for <REF>) the moment work arrives, and two throttledidle — …lines for the paths that were silent: the board offering no work, and every slot being busy — the latter naming what holds them (RE291 retainedis a failed run's worktree kept for post-mortem, which holds its exclusive slot until reclaimed). Both are capped at one line per reason perIDLE_LOG_INTERVAL(300s, a module constant, deliberately not a config key), and a claim re-arms them, so a quiet runner costs ~2 lines per 5 minutes. -
Single-process guarantee (RLY-193). Exactly one
relay startmay run per{server, name}(the pair the server keys aRunneron,namedefaulting to<checkout-dir>@<short-host>— RE305, so two checkouts of one project on one machine no longer collide on identity, provided their directories are named differently) and per worktree namespace. At startupcmd_starttakes two exclusive, non-blockingfcntl.flocklocks — an identity lock under$RELAY_RUNNER_LOCK_DIRor~/.relay/lockskeyed onsha256(RELAY_URL + "\0" + name)(sincenameembeds the checkout directory, two clones on one host in differently-named directories now hash to different lock paths — RE305;default_runner_name/0uses the directory's BASENAME, so same-named directories still collide and the identity lock refuses the second one by name), and a namespace lock at<ROOT>/.claude/worktrees/.<namespace>.lock— held for the life of the process by keeping their fds open. A second process for a colliding identity or a shared worktree namespace refuses to start (relay: already running: …, naming the holder's pid) rather than registering as the same runner. Becauseflockis released by the kernel on process death, a crashed runner leaves no stale lock (this is why a flock and not a pidfile). This is what makes the RLY-170 orphan recovery above sound: that recovery requeues a job the runner no longer reports running, which is only correct because a single identity can no longer be split across two live processes each beating a partialrunninglist. Two runners on one host are supported when they are different checkouts — a distinct name gives a distinct identity lock, and a distinctROOTgives a distinct namespace lock; what remains unsupported is two runners sharing one checkout and name. -
./relay update [--check] [--json]— non-interactive, writes only the six Relay-owned files, needs no TTY and no board key. -
Worktree namespace (RLY-231: one worktree per card).
RunnerPoolmaps every job'sisolationonto worktrees under theexec-*namespace.shared_cleanjobs share one reusedexec-cleanworktree (never reset per-job, only fast-forwarded to base when every shared slot is idle) — unchanged.exclusivejobs no longer draw from a fixedexec-work-1..Npool; each card gets its own worktree named<ns>-<ref>(e.g.exec-RLY-231), created on the card's first exclusive job and torn down when its run reaches a terminalrun_state. Since the worktree's identity is the card's branch, cross-contamination between two cards is impossible by construction, and the binding is derivable fromgit worktree listrather than an in-memory map, so a restart re-derives it (RunnerPool.recover/0) instead of losing it.- Capacity is reinterpreted, not reshaped:
capacity.exclusiveismax_worktrees— the max number of concurrent active per-card worktrees a runner holds, not a fixed slot count. Advertised freeexclusive=max_worktrees − active_count. The runners view'susedfor theexclusivechip is now derived from the runner's declared holdings (runners.held, RE311) rather than from its active jobs — which is exactlytotal − freeasRunnerPool.capacity()computes it, so the two sides agree by construction. Counting active jobs made a bound-but-idle, talk-attached or retained worktree invisible, and that is what reported "runner available" while the runner had zero free exclusive slots. The runners page also lists every holding in a per-runner HELD WORKTREES section (RE337,Relay.Runs.list_runner_status/2'sholdings: card, state, the card's active run and how long it has sat), with Release worktree (idleboundholdings only —Relay.Runs.release_action/1; disabled forrunning/talk, hidden forretained) and Cancel run (Relay.Runs.cancel_run/2).Relay.Runs.starvation/2drives an amber banner above the queue when a queued exclusive job has waited past@awaiting_slot_grace_s, no connected runner has a free exclusive slot, and every held exclusive slot isbound(no live job) — so nothing will free a slot on its own. A roster with anyrunningholding is not starved. Release is manual only; there is no auto-release. - Two states. Active: bound to a non-terminal run, counts toward
max_worktrees, holds aMIX_TEST_PARTITIONindex. Retained: afailedrun's leftover kept on disk (marked with the gitignored.relay-retainedsentinel at its root) for post-mortem, up tomax_retained_failed(default 3, oldest evicted first by mtime past that), not counted against capacity.done/cancelledremove the worktree immediately; a revoke (run_state == nil) touches nothing. - Several node jobs per tree (RE437) — concurrent node jobs per run. A
fan_out's branches are node jobs of ONE run, and all of them run at once in the card's one worktree (one exclusive slot, not one per branch). The record'sliveis therefore a live count of in-flight node jobs, not a flag:assignincrements it on every exclusive placement,releasedecrements it, and the terminal disposition above (or its deferral under a talk turn) runs only when it reaches 0 — the last branch out decides, so the first branch to finish never tears the tree down under its siblings.holdings()reportsrunningwhile it is above 0. Each job's board-log lines are attributed to that job, not to the card (per-job attribution, see Talk turns below), and the first job's preparation finishes before any sibling runs in the tree (the prep wait, under Prepare hook). - Shared-checkout hazards — the flow author's responsibility, not Relay's. Concurrent
branches share everything the worktree has, and Relay guards none of it (RE437
decision 3): same-file edits race and the last write wins; concurrent
git commitcontend for.git/index.lock, so one fails; and there is oneMIX_TEST_PARTITIONand one_buildper tree, so two branches runningmix compile/mix testat once rebuild under each other and truncate the same test database. A region whose branches only read (a review panel) is safe; one whose branches write must split the files between them and leave committing and building to a single node after thefan_in. - Never-detach. Once the run's
branchnode attachesrefs/heads/{branch}, the runner never re-detaches that worktree again — the old mid-run reset-on-revoke path (below) is gone. A revoked exclusive job now just stops its subprocess and leaves the worktree active and bound, ready for the pinned resume to continue in it. - Prepare hook. On a reset (first job of a card, or reclaiming a retained worktree for
a new run),
RunnerPool.create_or_rebaseline/1makes the worktree clean at base, thenrun_prepare_hook/3warms it: it runs the hookRunnerPool._hook_path("prepare")resolves —worktrees.preparefromrunner.json, else the deprecated flatprepare, else.relay/prepare-worktree.sh— if that file exists and is executable, else it is a no-op (a cold build, not an error). The hook receives[worktree, ref, branch, base, cache_dir]as both argv and env (RELAY_WORKTREE/RELAY_REF/RELAY_BRANCH/RELAY_BASE/RELAY_CACHE_DIR) withcwdset to the new worktree; a nonzero exit fails the run fail-fast (its stderr becomes the node's failure detail) rather than silently running against a half-warmed tree. Relay's own hook,.relay/prepare-worktree.sh, is a boring first cut: it copiesdeps/,_build/, andassets/node_modules/fromcache_dir(or else the main checkout) into the fresh worktree via APFS clonefile copy-on-write (cp -Rc, falling back to plaincp -R), somix deps.getis a no-op andmix compileonly rebuilds the diff. When a run's first job is a branch, several of its jobs are placed on a tree that does not exist yet: exactly one getsresetand prepares it, and the rest block inexecute_oneon the pool's per-slot preparation event (RunnerPool.wait_prepared/1) until it finishes. They then run in the prepared tree, or — when the hook failed — each reportfailedwith the sameprepare hook failed: …detail instead of running (RE437). - Cleanup hook (RE339). Right before the runner deletes an intact per-card worktree,
run_cleanup_hook/2runs the project's cleanup hook so it can tear down what prepare set up (a dev server, a docker-compose stack, a per-worktree test database, a port reservation). It resolvesworktrees.cleanupfromrunner.json, else.relay/cleanup-worktree.sh, and is skipped when that file does not exist or is not executable. Relay ships none of its own, because deleting the tree already removes everything its prepare hook copies. It gets the prepare hook's contract:[worktree, ref, branch, base, cache_dir]as argv andRELAY_*env, cwd = the worktree (still on disk).branchis read from the tree and is empty when HEAD is detached. When: every removal of a tree whose.gitlink is intact. That covers adone/cancelledrelease, a talk-worktree release, retained-over-cap eviction, everyrelease_heldretry, andrecover()'s reclaim of retired<ns>-work-Nslots, whereRELAY_REFis empty. It does not fire on retain (afailedrun's post-mortem tree is cleaned up later, when it is evicted), on a half-deleted tree (its cleanup already ran, and a.git-less cwd would send git into the main checkout), or on a rebaseline/reset or shared_clean refresh, since those trees are reused. Order: cleanup hook →_stop_processes_in→ stash salvage →git worktree remove, so the hook stops its own services gracefully and the RE336 sweep catches the rest. Best-effort: unlike prepare, a nonzero exit, or running pastCLEANUP_HOOK_TIMEOUT(120s, not configurable, because_teardownholdsRunnerPool.lock), is killed if still running, forwarded as anerrorline naming the card's ref, and the removal goes ahead. The run's outcome never changes. The hook must be idempotent: if it succeeds and thegit worktree removeafter it fails, the retry finds the link still intact and runs the hook again. - Shared
.git. Worktrees never get their own clone;git worktree add/remove/pruneroutes throughgit_worktree_with_retry, the same bounded-retry discipline asgit fetch(RLY-224 §6), since concurrent per-card creates/teardowns race on the one shared ref db. - A failed teardown is retried, not forgotten (RE336).
done/cancelledpop a worktree's record and free its partition only once the tree is actually gone. Ifgit worktree remove --forcefails, the record staysactive,live=False, bound to its run and holding its partition, and is flaggedteardown_failed.holdings()therefore keeps reporting itbound, the server keeps naming it inrelease_held(its runs are all terminal), andRunnerPool.release_held/2retries the removal on every heartbeat. Each failure is forwarded as anerrorline carrying the card's ref, throttled to one perIDLE_LOG_INTERVALper ref, and a later success logs once. Retention eviction follows the same rule. A card whose teardown is failing getsreset=Trueon its next run or talk turn (assign_talkre-baselines it too, or waits if something still occupies it), never reuse-as-is, because the tree may be half-deleted. A half-deleted tree (its.gitlink gone) is removed withrm -rf+prune, andcreate_or_rebaselinedeletes and re-adds one rather than running git inside it, since git would walk up into the main checkout. If that delete leaves the directory behind, it stops the tree's processes, retries once, and then dies. It never falls through toreset_worktreeon a.git-less path. Previously the record was popped before removal, so a failed removal left the tree on disk but invisible torelease_helduntil a runner restart (recover()) rediscovered it. - In-tree processes are stopped before removal (RE336). Before
git worktree remove(on the removal path only, never for a retainedfailedtree, which is a post-mortem left as is),_stop_processes_infinds every process whose current working directory is the worktree or inside it. It uses/proc/*/cwdon Linux andlsof -a -d cwd -Fpcnelsewhere, and excludes the runner and its ancestors. It sends each one SIGTERM, waits up toTEARDOWN_KILL_GRACE_S(5s), then SIGKILLs any survivors, and forwards one line per process (pid, command, card ref). It kills by cwd rather than by process group so that a server a smoke/acceptance node started withnohup/&, which escaped the node's group, is still caught. That was the observed trigger: a leftoverphx.servermadegit worktree remove --forcefail with "Directory not empty". Discovery is best-effort: if the tool is missing or errors, the step is skipped and the retry-and-log above covers the failure.
- Capacity is reinterpreted, not reshaped:
-
Per-node scratch (RLY-214). Alongside the worktree itself, every node gets
RELAY_NODE_SCRATCH(scratch_pathin./relay):tmp/<REF>/<node>.mdinside that same worktree, keyed only on(ref, node)so a re-queued job after a runner restart resolves the identical path. It sits under the checkout's own.gitignore, so it survivesreset_worktree's salvage/stash/clean untouched and never gets committed. See../../relay.mdfor the full contract, including why agents must not invent their own scratch path (RLY-177). -
Test database per slot (RLY-213). Worktree isolation keeps two concurrent runs' files apart, but
mix testfor both would otherwise hit the same Postgres database — Ecto's SQL sandbox only isolates concurrent tests within one BEAM, not across two OS processes.RunnerPool.partition_for(slot)(./relay) derivesMIX_TEST_PARTITIONfrom a free-list index held by the worktree's registry record at the single point where a node's command launches (both_stream_shelland_stream_agent_job), so every step of a run — including theprecommitgate — sees the same database: each active per-card worktree (e.g.exec-RLY-231) holds its own index for the run's lifetime, recycled on teardown; the sharedexec-cleanis always partition0.config/test.exsalready keys the database name offMIX_TEST_PARTITION. -
Harness command templates (RE433). An agent node runs whatever command the claim's
harnessdefines;./relaynames no harness (RunnerHarnessGuardTestgreps it for the seeded keys and names).select_command(harness, resume_session)picksresume_commandonly when the job carries a session and the harness has one — otherwisecommandand a fresh start (a harness that cannot resume never sees the old session).expand_command(template, values)is pure: itshlex.splits the template first, then substitutes inside each token, so each placeholder fills exactly one argv element, the prompt never passes through a shell, and a substituted value is never re-scanned (a prompt containing{model}stays verbatim). The placeholders areHARNESS_PLACEHOLDERS—prompt model worktree ref effort subagent session, pinned toSchemas.Harness.placeholders/0by the fixture'svocabulary.harness_placeholders.[ … ]marks an optional segment ([--effort {effort}]→ tokens[--effort{effort}]): it is dropped whole when any placeholder in it is null/empty; such a placeholder outside a segment, or an unknown name, is aTemplateError, andrun_node_jobfails the node with that message without spawning. So does an agent job with noharness(this job carries no harness — the server and runner disagree; update both). The values areprompt(compose_node_prompt),model/effort(the claim's),worktree(the slot path),ref,subagent(the node'sagent) andsession._stream_agent_job(argv, cwd, …, session_id_path=…)(formerly_stream_claude_job) runs the argv withstdin=subprocess.DEVNULL(codex execreads a non-TTY stdin) and reads the session id from each JSON event at the harness'ssession_id_pathwithread_path(.session_id,.thread_id; latest non-empty wins, no path captures nothing). Talk is out of scope and stays onclaude:run_talk_jobexpands the fixedTALK_COMMAND/TALK_RESUME_COMMANDwith session path.session_id. -
Harness inventory (RE433).
harness_inventory(harnesses)reports, per board harness,{key, installed, version, signed_in}:installed= the template's executable is onPATH,version= the first line of<exe> --version,signed_in= the harness'ssigned_in_checkexited 0 (null with no check or when not installed). Each probe has a 10 s timeout (HARNESS_PROBE_TIMEOUT_S) and degrades instead of raising.RunnerHeartbeat.refresh_harnessesfetchesGET /api/harnessesand recomputes it — at startup before the first beat (so the first beat already carriesharnessesand the server never reads the runner as Claude-only), then again only when a beat reply'sharnesses_digestdiffers from the one held. Every later beat carries the inventory; a failed fetch leaves the digest alone so the next beat retries. -
The claim/execute/report loop (
cmd_start). Each iteration: advertise current free capacity per isolation class on a long-pollPOST /api/node-jobs/claim(a read timeout is "no work", not an error) — the claim body is{runner, capacity, running, held, harnesses},harnesses(RE436) being the installed harness keys minus those paused at a usage limit, omitted before the definitions are first fetched (see Usage limits); on a claim, hand the job to a worker thread bounded by the pool's free slots; the worker resets the slot if needed, runs the step (shell/gate via_stream_shell, agent via_stream_agent_job), and POSTs the typed outcome to/api/node-jobs/:id/outcome.--oncedrains a single claim→execute→report cycle and exits;--dry-runclaims and mutates nothing (it only logs the capacity it would advertise);--intervaloverrides the configured poll timeout; SIGINT stops claiming new work and waits for in-flight workers to finish. -
The heartbeat cannot wedge, and a failing one is visible (RE336). Every
api()call gets a finite read timeout,API_TIMEOUT_S(30s), unless its caller passes its own. The long-poll claim and the startup board fetch pass their own and still get thesocket.timeoutre-raised. On a default call, a read timeout retries only for GET/PATCH or anidempotentPOST. Anything else fails the call, since the server may already have processed it. Before this change a bareurlopenblocked forever on a half-open connection. One heartbeat POST caught that way by a server restart silently killed the runner's heartbeat for days, and with itrelease_held, revoke, capability refresh, usage reporting, auto-update and capacity advertisement. The roster still showed the runner fresh, because claims also stamplast_heartbeat.RunnerHeartbeat._beatnow guards everything it does (capability scan,held_fn, the POST, the reply handlers), and_runguards the loop body as well. A failing beat forwards oneerrorline naming the cause (die()raisesDied, aSystemExitthat carries its message), repeats at most once perIDLE_LOG_INTERVALwhile failures continue, and logs one "heartbeat restored after Ns" line when a beat lands again. -
Heartbeat-borne revoke.
RunnerHeartbeatPOSTs{runner, capacity, running: [job-ids], held: [{ref, state}]}toPOST /api/node-jobs/heartbeateveryheartbeat_intervals and reads{revoked: [job-ids], release_held: [{ref, status}], latest_runner_version}back (RLY-164, ref keying RE311), terminating each revoked job's live subprocess via itsJobControl.release_heldis the ref-scoped analogue ofrevoked: the runner advertises every per-card worktree it holds (held), and the server names the subset whose card has at least one run and no run left inSchemas.Run.active_statuses/0— with the status needed to choose remove vs retain (Relay.Runs.releasable_held/2) — soRunnerPool.release_held/2disposes of them within one heartbeat. Since RE337release_heldalso carries human-requested releases: the runners page's Release worktree action appends the card ref to the runner row'srunners.release_requests(Relay.Runs.request_worktree_release/4, accepted only while the runner's latestheldreports the refbound), and the reply (Relay.Runs.release_held/3) adds{ref, status: "cancelled"}— a remove disposition (Schemas.Runner.release_remove_status/0), neverfailed— for each pending request this beat still reportsbound. The run stays parked; on resume the runner'sassign()rebuilds the worktree from the branch. Clearing rule: every beat that reportsheldprunes the column in one atomic UPDATE (upsert_runner/2) — a request survives only while its ref is reportedboundortalk(teardown deferred under a talk turn); absent (torn down) orrunning(the run resumed first) clears it, so a stale request can never tear down a resumed run's tree. A runner that goes silent keeps its requests until it beats again. No runner change was needed:RunnerPool.release_heldalready refuses a live or talk-attached tree and stashes dirty edits asauto-salvage <slot>(never restored). A card with zero runs is a talk-only worktree and is never named (ADR 0009 §2: a talk session's tree spans runs and must outlive them), and aretainedtree is the human's post-mortem, the runner's own to evict. This is how taking the baton (ADR 0004, viapark_claimed/1) or cancelling from the run panel stops a running agent without waiting on its next outcome POST — and, unlike the retired run-id-keyed channel, it also reaches a worktree the runner re-derived from disk after a restart. Never-detach (RLY-231): a revoked job of either isolation class leaves its worktree exactly as it was — an exclusive worktree is bound 1:1 to its card and stays active + attached to the branch for the pinned resume to continue in, and a revokedshared_cleanjob already leftexec-cleanuntouched (it's shared by other concurrently running jobs and only ever fast-forwarded once every shared slot is idle). The old reset-on-revoke for exclusive jobs is gone: no reset is ever needed mid-run now that a worktree's identity is the card itself. Either way, no outcome is reported for a revoked job — the server already knows a revoked job never finished. The same beat carriessessions(RE358), each in-flight agent job's captured session id, read from itsJobControl(talk jobs are never reported). -
A newly claimed job supersedes the stale job it replaces (RE358). Before placement,
cmd_startsnapshotsin_flightunder its lock (each entry carries the job'srun_id,node_idandkind) and callssupersede_stale_jobs: every other in-flight non-talk job of the claimed job's run at the same node is cancelled through itsJobControland joined, outside the lock, for up toSUPERSEDED_JOB_JOIN_S(30s; an error line names a job still running after that). A run has one active job per running branch (Runs.active_jobs/1): one outside a fan_out region, one per branch inside it (RE437), and branches are node-disjoint. So a same-run, same-node job is either steered (a Steer now re-enters the same node; revoked, but the heartbeat that says so may be up to an interval away) or already reported and still in itsfinally, where the cancel is a no-op. A same-run job at another node is a concurrent sibling branch and is never touched. This closes a real gap: a granted claim never carriesrevoked, andRunnerPool.assignreuses an exclusive worktree bound to the samerun_ideven while it islive, so a Steer now restart could otherwise start in the same tree as theclaude -pit replaced. Talk jobs are never superseded — a talk turn may legitimately run beside its card's node job. -
Auto-update (RE185). When the beat's
latest_runner_versionexceeds the runningRUNNER_VERSION,maybe_auto_updatefires from the claim loop — but only at a job boundary (nothing in flight), never under--once, and never more often thanauto_update_min_interval, and never again after three failed updates in one process's life (a refused download or a failed install — a successful one never counts, so a runner that has been up for months and picked up ten releases is unaffected). It downloads./relayfrom the board's scaffold endpoint ($RELAY_URL/api/scaffold/relay) throughdownload_runner, which sharesverify_runner_sourcewithrelay update— HTTPS, UTF-8, a leading#!, aRUNNER_VERSIONparsed from the downloaded bytes (authoritative — a board ahead of what it serves is then harmless, not a chase), acompile()syntax check, and (for auto-update only) strictly newer. Verification is deliberately HTTPS + parse only: no checksum, no signature. Any rejection logs why and the running version keeps serving; a bad update can never leave a machine with a broken runner.The install writes the tracked
./relayin place (os.replace, so a concurrent./relay card …never reads a half-written file)._safe_to_overwriteis what makes that tolerable: untracked or not-a-repo is fine, tracked-and-clean is fine, and tracked-and-dirty is allowed only when the file's sha256 matches a write recorded in~/.relay/auto-update.json— otherwise the dirt is a human's uncommitted edit and the update is skipped with a one-time log line. That ledger is not a nicety: after the first auto-update the file is permanently dirty against HEAD, so a clean-only rule would wedge every later update. The accepted cost is that./relayshows as modified ingit statuson the machine running the runner.The restart stops the heartbeat and the log forwarder, calls
release_runner_locks()— required, because RLY-193's flocks belong to open file descriptions that surviveexecvwhile the re-exec'd image opens new descriptors and would die "already running" — thenos.execvs withRELAY_UPDATED_FROM/RELAY_UPDATE_ATTEMPTSset. On the way back upcheck_update_handshakecompares: newer, and it logs the transition and clears the failure count (the update took, so nothing before it was thrash); same or older, and auto-update is disabled for that process, leaving RLY-184's fail-stop as the behaviour. Three failed attempts disable it the same way. Opt out with"auto_update": false.
Usage limits (RE320 → RE436)
relay start stops claiming work for one harness once that harness's usage passes the PAUSE
AT its board rule sets, and resumes by itself. RE320 did this for Claude Code alone, from
hard-coded event shapes and per-runner .relay/runner.json limits; RE436 made it generic.
Reading usage, and deciding when to pause, is now each harness's usage rule (Settings → Agents
→ Harnesses, served by GET /api/harnesses — see "Harness routing" above). ./relay names no
CLI, event or window in executable code (bin/test_relay.py RunnerHarnessGuardTest).
-
Config. PAUSE AT lives only on the harness (
windows[].pause_at). There is no per-runner override..relay/runner.jsonlimitsis retired:load_runner_configdrops it and logsLIMITS_IGNORED_LINEonce; it never refuses to start. The starter config no longer writes it. -
The book.
cmd_startkeeps aHarnessUsageBook(the globalUSAGE): oneUsageLimiterper harness key.RunnerHeartbeat.refresh_harnesseshands each successful fetch's raw definitions toon_definitions, which callsHarnessUsageBook.rebuild. A limiter whoseusagerule is unchanged is kept, so its readings survive a digest move caused by an unrelated edit; a changed rule starts afresh; a harness gone from the board is dropped. -
Usage sources. A limiter is fed three ways, all through
usage_reading(the rule evaluator):output— the job's own stream.run_node_jobpasses the claimed job's harness limiter (job["harness"]["key"]) to_stream_agent_job(usage=…), which feeds every parsed event to it. There are no extra calls in the normal path. Claude Code's seed reads therate_limit_eventitsstream-jsonemits on every call. Talk turns feed no limiter: they are out of RE436's scope, and the old shared limiter they fed is gone.command—UsagePoller, a daemon thread ticking everyUSAGE_SLICE_S(5s), runs the rule'scommandthroughsh -cfrom the system temp dir everyinterval_min(run_usage_command). Each stdout line that parses as JSON is observed; the last readable one wins. Codex's seed reads thetoken_countline of its newest rollout file.- re-check — while a harness is paused and has no job in flight (the loop's
in_flightentries record each job's harness key), the poller runsrecheck_harnesswhenrecheck.interval_minhas passed since its last reading or re-check. That expands the harness's owncommandtemplate with promptok, the rule'srecheck.modeland the temp dir as{worktree}, runs it from the temp dir, and reads its stdout (output) or then runs the usage command (command). It returns a reason rather than raising (a template needing{ref}, a timeout, a CLI that refuses a non-git cwd), never un-pauses anything by itself, and a failure is logged once per pause per harness. Every usage command and re-check times out atUSAGE_COMMAND_TIMEOUT_S(60s).
-
Pause. A window trips at
used_pct >= pause_at. A line meeting the rule'spause_now.whenis a refusal and trips regardless, until its reset (reason: "rejected"); a later readable line that is not a refusal ends it early. With several trips,UsageLimiter.pausedreports the latest reset (a refusal wins a tie). Each reading and refusal is discarded at its reset. Before each claim the loop takesinstalled(the inventory's installed keys, None before the first fetch) andpaused = USAGE.paused_keys(now):- with every installed harness paused, it claims nothing — node, gate or talk — and sleeps
in
USAGE_SLICE_Sslices; jobs already running finish (RE320's contract, now for the all-paused case only); - otherwise it claims with
harnesses = sorted(installed - paused)(omitted before the first fetch). The server ignores the claim'sharnessesuntil it filters by it.
It logs
⏸ <name> paused: <label> <used>% ≥ <pause_at>% · resumes HH:MM · re-checking every <N>m(orrefused a call) once when a harness enters a pause and▶ <name> usage back under its pause-at — claiming its nodes againonce when it leaves. - with every installed harness paused, it claims nothing — node, gate or talk — and sleeps
in
-
RE267 resume-at. When
_stream_agent_job's limiter reads a refusal with a reset from the job's own stream, it setsreport.rejected/report.rejected_resets_at;blocked_resume_atsends that as the outcome'sresume_at.ClaudeRunReportno longer parses any usage event itself. -
Heartbeat. Each
harnesses[]entry carriesusage:UsageLimiter.wire(now)—{source, read_at, windows: [{label, used_pct, resets_at}], paused: {label, used_pct, pause_at, resets_at, reason} | null}withreasoninSchemas.Runner.rate_limit_reasons/0(limit|rejected, mirrored asUSAGE_PAUSE_REASONS) — or{source: "none", read_at: null, windows: [], paused: null}for a harness the book does not know. The top-levelrate_limitis no longer sent. Beats continue while paused, so the runner staysfresh. The server stores each reading on itsrunners.harnessesentry viaSchemas.Runner.normalize_harnesses/1. That normalizer is total: an unknownsourcedrops the wholeusageand keeps the entry; a malformed window is dropped, and at most 20 are kept; a malformedpaused(unknownreason, non-positiveresets_at) becomes nil. The next beat'spaused: nullclears a pause; a beat withoutharnessesleaves the stored readings alone. The roster derives "rate limited" from these readings (see the Rate-limited roster bullet).
Talk turns (RE268, ADR 0009)
The claim loop's worker() branches on job.get("kind"): a "talk" job runs execute_talk
instead of execute_one, and reject() reports a rejected talk job through
report_talk_outcome (POST /api/talk/turns/:id/outcome) rather than the node-job outcome
route — a talk job has no run_id for that route to accept. No second thread: it is the same
claim/execute/report loop, one more branch.
-
Worktree. A talk turn always runs in the card's own exclusive per-card worktree
<ns>-<ref>— never the shared<ns>-cleantree, which other cards' jobs are using.RunnerPool.assign_talk/1attaches to a live worktree a node job already holds (dirty reads are the point — the tree may be mid-edit, and that is often exactly what is being asked about), reuses a retained failed one as-is (post-mortem is what people ask about), or creates one on demand for a card the flow engine would never itself dispatch.A node job and a talk turn can occupy the SAME worktree record at once, so occupancy is tracked per occupant:
livefor the node job, and atalk_userscount for talk turns — a count, not a flag, because two turns can legitimately overlap on one tree: Stop finalises a turn server-side at once, but the runner only learns of the revoke on its next heartbeat (15s), so a person who hits Stop and immediately retypes has turn 1 still running when turn 2 is claimed into the same tree. A flag did not count the second occupant, and turn 1's release then stashed and force-removed the tree turn 2 was answering in.RunnerPool.assign_talk/1andrelease_talk/1only ever touchtalk_users, neverlive— an earlier version shared oneliveflag between both occupants, which let either tear the tree down (or believe it idle) out from under the other: a node job'srelease()finishing while a talk turn was still streaming would rungit worktree remove --forcemid-answer and stash the person's uncommitted edits; a talk turn ending would clearliveunder a still- running node job, defeating the release path's "never touch a worktree with a live job" guarantee.release()/release_held/2(which replaced the run-id-keyedrelease_run/2in RE311, keeping both of its guards verbatim) DEFER a terminal disposition (pending_finish) while any other occupant (live, ortalk_users > 0) is still present;release_talk/1finishes it once the last one leaves — last occupant out tears down. The deferred disposition is read only after those early returns, so a non-final release cannot discard it.A talk-created worktree carries
run_id: None, but so does a worktreerecover()re-derives after a restart — the exact caserelease_held/2's ref keying exists to reach (RE311) — sorun_idalone can no longer gate the skip.release_held/2instead checks thetalkmarker: a talk-created record is skipped outright, because its tree spans runs and must outlive them, while a recovered one is not, so arelease_heldentry can still tear down a tree bound to nothing that was never talk-only. Once such a tree goes fully idle (its talk turn ends and no node job ever adopted it),release_talk/1retires it — releasing its partition and marking itretained, evictable oldest-first exactly like a failed run's leftover — so a card that was only ever talked to, never run, does not permanently consume an exclusive slot (and keep the poller from ever idling). A retained tree a talk turn has reattached to is excluded from that eviction while it is live.Retirement is gated on the record's own
talkmarker, not onrun_id is Nonealone (RE268):recover()rebuilds every restart-adopted per-card worktree as{"run_id": None, "state": "active", ...}too — the identical shape, with notalkmarker. Retiring onrun_idalone mistook a recovered worktree for talk-only and retired (stashed + hard-reset) it out from under its resuming job.RunnerPool.assign/1(the node-job path) likewise refuses to reclaim a retained tree a talk turn has reattached to (talk_users > 0) rather than re-baselining it out from under that turn's still-reading claude process — the same "refuse rather than steal a live worktree" precedent it already applies to a tree bound to a different live run.Restart durability of the
talkmarker (accepted gap, RE311). That marker is in-memory only:recover()rebuilds every adopted tree with notalkkey, so after a restart a talk-created tree on a card that HAS runs (all terminal) is named by the server and torn down, which ADR 0009 §2 otherwise forbids. The exposure is narrow — an idle talk-only tree has already beenretainedby_retire_talk_only_locked, andretainedIS disk-marked and survives — so it takes a talk turn that was live at crash time, whoseclaudeprocess died with the runner anyway; teardown stashes any dirty edits (_teardownsalvages viagit stash push -u) rather than discarding them. Markingtalkon disk the wayRETAINED_MARKERis would close it, but the marker would have to be written after the tree is created (in the talk worker, notassign_talk/1) and CLEARED the moment a run adopts the tree — a stale marker would makerelease_held/2skip that tree forever, which is the very slot leak this card removed. Left as a documented gap rather than a half-durable marker.Adopting the card's own tree (RE311).
assign/1reuses anactiverecord when it is bound to this run, has an unknown binding (run_id: None, therecover()shape), OR is idle and bound to an earlier run of the same card — the last case is cancel-then-retry: a revoke deliberately leaves the treeactive, idle and bound to the cancelled run, and the release channel only frees it on a later beat (~15s). Refusing there placed nothing for the bounded retry window and then failed the retry with a misleading "no free 'exclusive' slot advertised", so a card's retry could fail for holding its own worktree. The refusal is now exactly what its comment always claimed: a genuinely LIVE run, or a deferredpending_finish.Bounded, and retried: while a node job's terminal disposition sits deferred (
pending_finish), the record staysstate="active"bound to the OLD run_id. If a NEW run for the same card is dispatched before the talk turn ends,assign/1refuses it — a deferred disposition still owns the tree, and adopting it would letrelease_talk/1tear the tree down under the new run. (This is now the ONLY reasonassign/1refuses an idle tree: since RE311 an idleactiverecord bound to an EARLIER run of the same card is adopted, because the tree's identity is the card's branch. See "Adopting the card's own tree" below.) That refusal used to be reported as a failed job, so someone asking "why did this fail?" during a retry failed the retry. The claim loop now retries placement for a bounded window (PLACEMENT_ATTEMPTS×PLACEMENT_RETRY_S, ~10s,./relay) before rejecting, which covers one talk turn handing the tree back; only a persistent miss is treated as genuine capacity exhaustion. See failures.md E2t.A talk turn that creates its own worktree fresh (the card's run is done/cancelled and was torn down, or the card was never run) lands on the card's own branch when one exists, via
checkout_talk_branch/2right aftercreate_or_rebaseline— not left detached atorigin/main. Falls back to the existing detached-at-base state when there is no branch yet or it has not reached the remote. It prefers the LOCAL branch when one already exists and never resets it —checkout -B <branch> origin/<branch>force-moves the branch ref to the remote tip, which would silently discard commits a run made and never got to push — and falls back to creating the local branch from the remote only when there is no local ref to lose. A failed checkout (e.g. the branch is already checked out in another worktree) is logged rather than swallowed, so it doesn't silently fall back to a detached tree describing none of the card's work. -
Prompt.
talk_prompt(job)is built in product code, not a.claude/agents/*.mddefinition — a recorded exception to ADR 0006 (ADR 0009 §5): Talk is a property of Relay itself and must behave identically on every connected repo. It is the preamble (names the pane, the read-only rule, and./relay why/runs/card) + the card's seed fields + the human's text, spliced in verbatim and last — never passed throughrender(), so a person's{ref}-shaped typing can never reach into the var namespace. -
Event mapping.
_stream_agent_job(then_stream_claude_job) gained anon_event=Nonecallback, invoked with each parsed stream-json event as it arrives — this is what lets a turn's transcript stream while the turn is still working, not all at once at the end.talk_events_from(ev)maps one event to the transcript lines it produces: assistant text → an:outline, atool_useblock → one dim:toolline naming its target, an erroredresult→ an:errorline. Anything else (including an unrecognised event type) produces nothing — a mangled or unfamiliar event must never cost the turn. The tool-use "brief" (command/file_path/path/pattern/description, truncated to 140 chars) is one shared_tool_briefhelper, used by both this and the console/board-log renderer (_print_claude_event)._stream_agent_jobalso gainedmirror=True; a talk turn passesmirror=Falseso its events print locally but are never also forwarded to the board-log mirror. Two things make that necessary: the tagrun_talk_jobbuilds is ref-clean (f"[{ref}] (talk) ", "talk" outside the brackets — an earlierf"[{ref} talk] "made_ref_from_tagextract a ref that does not exist, e.g. "DE3 talk", spammingPOST /api/board/logsunder a phantom card), and a talk job's worker deliberately registers nonode_job_attribution. Board-log attribution is per worker thread (RE437, replacing the ref-keyedNODE_JOB_IDS/RUN_IDSmaps):forward()stamps a line with therun_id/node_job_idof the node job the calling thread runs for that ref, so a talk turn's lines carry neither, and concurrent fan-out branches on one card never clobber — or, finishing, erase — each other's ids. -
Delivery.
TalkEventSenderbatches and POSTs lines to/api/talk/turns/:id/eventsat-least-once — deliberately unlikeLogForwarder, which drops by construction (enqueueon a full queue, swallowed_senderrors). Each line'sclient_seqis per-turn and monotonic from 1, which is what makes a retried batch idempotent server-side (Relay.Talk.append_events/2dedupes on it). A failed POST retries with backoff up tomax_attempts(no sleep after the last attempt — the decision to give up is already made by then); exhausting the budget setsfailedandrun_talk_jobcallssend_best_effortto re-arm and try once more to deliver a single visible:errorline saying the transcript could not be fully delivered, then reports the turnfailedrather than silently reporting adoneturn with missing output.batch=40is a size cap, not a delay: a typical short turn (a couple of tool lines, a couple of text blocks) never reaches it, so without a second trigger every turn's transcript arrived in one POST at the very end — defeating the point of theon_eventstreaming hook (RE268).enqueuealso flushes whatever is already pending once it has waitedflush_interval(default 1s), checked before the new line joins it, so a slow trickle of events posts as it arrives instead of all landing in one batch together. -
Stop. Arrives as an ordinary revoke on the existing heartbeat —
RunnerHeartbeatalready terminates a revoked job's subprocess via itsJobControl;run_talk_jobcheckscontrol.cancelled()after the process exits and reportsstopped(notfailed) with whatever partial output was already delivered. No talk-specific channel was added.
RUNNER_VERSION 33 → 39 for this change (34 shipped the initial worker, 35 the occupancy/
retirement/attribution hardening above, 36 the recovery/retained-tree/streaming/branch fixes
from a follow-up review, 37 the branch-checkout non-destructiveness fix above, 38 the
failure-line-into-the-transcript fix from the whole-branch review, 39 the talk_users
occupancy count, the bounded placement retry, and the rejected-turn transcript line).
One version floor (talk included). Talk originally shipped with its own, higher floor
(38, then 39): Relay.Runs.min_runner_version/0 was 21 at the time and was not raised for
Talk, because a runner without Talk is not worse than a stopped one for the flow work it still
does correctly. RE311 then raised the base floor past it (it is 63 today), and RE414 removed the
separate talk floor. Today talk_capable?/1 inside claim_next_job/1 gates talk claims on
min_runner_version/0, narrowing the claim to NodeJob.flow_kinds() for anything below it
(over HTTP such a runner is refused 409 runner_outdated first, so the gate is defense-in-depth
for in-process callers). Without it a pre-Talk runner would happily claim the first
(deliberately unpinned, capacity-exempt) turn on a card, KeyError on the isolation key a
talk payload does not carry, and reject it to the flow-only outcome route — a 404 that leaves
the turn claimed forever and wedges Talk for the whole board (failures.md D4t).
Declaring an outcome
An agent node must declare its verdict before it exits, by running:
relay outcome <outcome> [--detail TEXT|@file]
./relay writes the JSON to $RELAY_NODE_OUTCOME (set per node; the verb refuses to run
outside a node), so a detail containing quotes or newlines cannot corrupt the file. detail
becomes the context handed to the next node. Which outcomes exist and what each one does to the
run and the card is state.md's "Node outcomes" table — the schema owns
that set, not this page.
Five rules sit on top of it:
- Silence is failure — but silence is not nothing. A node that exits without declaring is
reported
failedwhatever its exit code: a node that did nothing is indistinguishable from one that exited early, so it must never route past its own gate. The detail, however, names any uncommitted work the node left in its worktree (RE298), because "did not declare" and "did nothing" are different failures and the retry receives this detail as its findings. Without it the work is invisible — the retry redoes it from scratch and the next job on that slot sweeps it into a stash. - A success claim must be backed by a commit — or by proof the commit already exists. On a
node marked
expects_commits, asucceededthat left HEAD unmoved is rewritten tofailedbefore finalize (failures.md A6). The one way out isrelay outcome succeeded --no-changes(RE310): an ASSERTION that the work was already committed, which the engine honours only when it can show from this run's own history that this node already produced a commit for the thing it is bound to — the same node and, under aforeach, the same sub-task. On a node's first visit for that binding the two baselines coincide, so the claim is always rejected and default-deny survives. The claim is recorded on the execution (no_changes) whether accepted or rejected, and a rejected one fails with(no_changes_unproven: <node>). - A declared card write must actually land. On a node declaring
writes, asucceededthat left one of those card fields blank is rewritten tofailedbefore finalize (failures.md A10).readsis never checked at run time — it is advisory. - Asking a human wins. If the node moved the card to
needs_input, that is the outcome even if the node also declared something else. - An agent that could not run is
blocked, notfailed(RE308). Whenclaude -pexits non-zero and its own stream shows the environment refused it — the assistant event'serrortag (authentication_failed/billing_error→ auth,rate_limit→ usage limit), the result'sapi_error_status(401/403, 429), arate_limit_eventwithstatus: "rejected"seen in this job's stream, or else a phrase fromCLAUDE_AUTH_FAILURE_SIGNATURES/CLAUDE_USAGE_LIMIT_SIGNATURESin its last words —determine_agent_outcomereportsblockedwithagent could not run: <reason>(a usage limit adds· resets HH:MM). The engine parks without spending retries (failures.md A11). The check runs after "asking a human wins" and after a declared outcome file, so a declared verdict is never reclassified.blockedis not one ofrelay outcome's choices, and an outcome file declaring it is reportedfailed, so an agent cannot park itself around its own retry budget. An unclassified non-zero exit staysfailed, its detail nowagent exited non-zero: <the stream's last words>. When the classification is a usage limit AND this job's stream carried a rejectedrate_limit_eventwithresetsAt,blocked_resume_atreturns that epoch,run_node_jobreturns it as its 6th element, andoutcome_bodysends it asresume_at(the key is omitted otherwise, so every other body is unchanged). The server then requeues the node rather than parking it — up to 3 consecutive waits — while the runner's own RE320 pause holds its claiming until the reset (RE267). Auth,billing_errorand a phrase-only usage limit send noresume_atand park as before.
The reminder is appended to every agent node's prompt automatically, so the requirement travels
with every invocation. shell and gate nodes are exempt — their exit status is already an
unambiguous verdict.
What an agent node's prompt is made of
./relay composes an agent node's prompt from up to four parts, in this order
(compose_node_prompt):
- the node's own
run, rendered —{ref},{branch},{relay}and the rest substituted from the job'svars; - the findings block, appended only when the job carries a non-blank
vars["findings"](RE251); - the guidance block, appended only when the job carries a non-blank
vars["guidance"](RE358); - the outcome contract, rendered, appended to every agent node.
shell and gate nodes get part 1 only: their run is a command line, not a prompt.
The findings block (RE251)
On a loop-back the engine sets findings on the next job's payload — the reviewer's detail on a
transition, and the ORIGINATING findings plus this attempt's failure detail on a retry
(RunServer.apply_decision/4). Substitution alone was never enough to deliver it: it only fires
for placeholders a template contains, and no shipped flow node contains {findings}, so every fix
node in the system was told to fix findings it was never handed. Appending the block in the
runner makes it universal — every agent node, every flow, every repo, and nothing for a flow
author to remember.
The block states that this is a loop-back and the findings are the subject of the run, carries the
findings, requires the node to account for every one of them in its outcome detail (fixed, or
rebutted with a reason), and makes "no finding needs a change" an escalation (needs-input)
rather than a succeeded. It points at the outcome contract below it for the exact commands, so
there is one rendered copy of each command per prompt.
The findings text is spliced verbatim and never rendered. Findings are free-form reviewer
prose: a reviewer who writes {ref} must see those characters reach the model, and reviewer text
must never be able to reach into the var namespace. The static wrapper carries no placeholders.
The guidance block (RE358)
Human comments and rejection notes written while a card is being worked reach every agent node.
The server derives them live, never snapshotting them into run.context:
Relay.Runs.guidance_for/2 selects them and Relay.Runs.render_guidance/2 renders them into
vars["guidance"] (a string, or nil when there are none).
- The window opens at the earlier of the card's open rejection's
rejected_atand the run'sinserted_at(just the run's when no rejection is open), inclusive — timestamps are second-precision and the rejection's own note lands in the same second. So a note a human wrote between a reject and the hand-back still counts on the run the hand-back starts. - The entries are user comments whose kind is in
Schemas.Comment.guidance_kinds/0(:comment,:changes_requested,:steer) plus any:changes_requestednote whatever its actor. Agent:comments and every:questionare excluded. The rejection note is already a:changes_requestedcomment, so it is never appended a second time fromcard.rejection. Entries are ordered oldest → newest (inserted_at, thenid). - The rendering is one entry per comment, separated by a blank line: a header
<author> · <YYYY-MM-DD HH:MM> UTC(tagged· CHANGES REQUESTEDor· STEERby kind), the body verbatim, and — when the note carries images —(N images attached — see ./relay card <ref>), since images don't travel in a prompt. - Derived at insert, refreshed at claim.
build_payload/4computes it for every job it builds (agent, shell and gate alike). In the pull model the claim is the real dispatch — a job is inserted when its run starts or advances but may be claimed much later — soclaim_next_job/4recomputes it on the job it hands out and stores it back on the row (refresh_job_guidance!/1, inside the claim transaction; talk jobs are untouched). A note added after a reject but before the runner picks the job up still reaches it.
In the runner, guidance_block/1 wraps the text in GUIDANCE_CONTRACT_HEAD /
GUIDANCE_CONTRACT_TAIL. The head frames the notes as instructions for this run and says later
entries supersede earlier ones — including the CHANGES REQUESTED note that sent the card back;
the tail asks for one short line per entry in the outcome detail (followed, superseded, or not
applicable) and makes an unresolvable conflict a needs-input escalation. Like the findings
block, the guidance text is spliced verbatim and never rendered — a human who types {ref}
gets those characters, and human prose can never reach the var namespace — and the static wrapper
carries no placeholders. An older runner ignores the var, which is the pre-RE358 behaviour, so
@min_runner_version was not raised.
Steer now (RE358)
The drawer's Steer now is Relay.Runs.steer/4: a human's note that stops the agent node
working the card and restarts that same node with the note in its guidance.
- The comment comes first, always. The steer is saved as a
:steercomment (Relay.Activity.add_comment/2) in its own transaction before anything else happens, so the words survive whatever follows. A blank steer with no images is a changeset error and nothing else happens. - Steered re-entry. When the card's run is
:running, the call goes through the run'sRunServer({:steer, actor, body, image_ids}, serialized with outcome reports). If the run's current node is an agent node with an active job whose attempt has no outcome yet — claimed or still queued, so a steer is never lost to a payload built before it — then in one transaction the server stamps that attemptoutcome: :steered(no detail), revokes its job (revoke_active_jobs/1; the runner kills the process when its heartbeat'srevokedlist names it) and enters the same node, same visit, attempt + 1, samesub_task_id. The new job's payload carriesresume_session= the stopped attempt'ssession_id(soclaude --resumekeeps the conversation when one was captured — see "The session survives the kill" below) plus the stopped job'sfindingsandprior_detail. After commit:{:node_finished, run, steered},{:node_started, run, new}, dispatch. The run's status is never written — it stays:running. :steeredis server-only. It is inSchemas.NodeExecution.server_only_outcomes/0, so the outcome API refuses it like any unknown outcome (outcomes/0, the wire contract pinned by the runner contract fixture, is unchanged) and no edge may route on it. The engine reads a steered row as itself, never as a degraded:failed, so it spends nomax_loopsbudget; and since any outcome ends RE410's outcome-less streak, steering repeatedly never trips the RE410 cap (the steered re-entry doesn't consult it anyway). A later re-entry still carries the last real failure as findings —last_failure_detail/1skips server-only rows as it skips:blocked. The Run tab shows a steered attempt as↪with a "steered" chip; it is not a spent attempt, and its restart reads "session resumed" when a session was kept.- Degrade to a note. No running run (none, parked, or one that stopped mid-call), a shell or
gate node, or no outcome-less active attempt: nothing is revoked and
steer/4returns{:ok, comment, :noted}. The steer still reaches the next agent node through its guidance. - Receipts land at claim.
refresh_job_guidance!/1(the claim-time guidance refresh) stampscomments.delivered_execution_id = job.node_execution_idon every:steerit just rendered that has no receipt yet — for agent jobs only (a shell or gate job never delivers a steer), and first delivery wins.steer_receipts/1reads them back for the drawer (node,session_kept?, and the attempt's outcome detail as its reply once it reports);steer_state/1says whether the card's AI-held run is on an agent node and whether a restart is still waiting to be claimed. A flow-job claim emits{:run_changed, card_id}so the drawer refetches. - The session survives the kill. A revoked job reports no outcome, so its session id would be
lost. Instead the runner reports every running agent job's captured session on each heartbeat
(
sessions: {job-id: session-id}), andRelay.Runs.record_job_sessions/2writes it onto that job'sNodeExecution.session_id— which is exactly what the steer reads asresume_session. So the hand-off is beat →NodeExecution.session_id→resume_session: a steer after at least one beat has carried the session resumes it; a steer before the first such beat restarts on a fresh session (the guidance still carries the note). - The restart never shares a worktree with the process it replaced. When the runner claims the
restarted job, it first supersedes the run's still-running stale job at that node — cancels it and waits for
it to exit (
supersede_stale_jobs, see "Runner mode" above) — before placing the new one. The revoke alone was not enough: it only arrives on the next heartbeat, andRunnerPool.assignreuses an exclusive worktree bound to the same run even while it is live.
Agent node → .claude/agents definition
A flow node of type agent may name an agent (e.g. plan-implementer). The server
carries it in the job payload (Relay.Runs.build_payload/4 → the claim response's
agent), and ./relay fills the harness template's {subagent} placeholder with it (RE433).
The Claude Code harness carries it as an optional [--agent {subagent}] segment, so a node with
an agent runs claude -p … --agent <name> and a node without one drops the segment: the agent
file supplies the system prompt, the node's run string stays the user prompt. A harness whose
template has no {subagent} simply ignores the node's agent (.claude/agents on non-Claude
harnesses is out of RE433's scope). An unknown name makes the CLI fail loudly rather than
silently fall back to the default agent (verified against CLI 2.1.214), which is the
property that makes this safe to depend on. A node with no agent invokes exactly as
it did before RLY-139.
Fallback if --agent ever regresses: delegate by name from the node's run prompt
— "Use the spec-reviewer subagent to review …". It works today with zero new plumbing
and needs no schema change.
Escalating a plan-mandated finding (RLY-190)
A reviewer that finds a defect the code implements faithfully because plan.md mandates it
does not return failed into the fix loop. The implementer is instructed to follow the plan, so
neither side can yield: the loop burns its budget until loop_budget_exhausted or the circuit
breaker ends the run with a message describing the symptom and not the cause. Instead the
reviewer raises needs-input and stops without declaring an outcome, which parks the run for
a human. spec-reviewer, quality-reviewer and final-reviewer each carry this as a third
verdict beside Approve/Pass and Fix; plan-implementer uses the same route when the plan tells
it to build something it can see is wrong.
This needed no engine change. needs_input is decided before any edge is consulted
(Relay.Runs.Engine), so it consumes no max_loops budget, does not increment the visit count,
and is not degraded to :failed. The command itself reaches every agent node for free —
./relay appends its outcome contract, already rendered, to every agent prompt (see
What an agent node's prompt is made of).
The human's answer, not the plan, is authoritative for the remainder of the run. This is a deliberate decision, and the tempting alternative — have the human edit the card's plan and treat the edited plan as authoritative — does not work:
- the plan is materialized once, to the per-ref
$RELAY_PLANpath, by the Code flow'sbranchnode (Relay.Flows.DefaultLibrary), and no later node ever re-writes it. sub_tasksare seeded fromcard.planat run start only, and are deliberately never re-materialized on re-entry so that done-state isn't wiped (Relay.Runs).- Re-entry replays the same node, same visit, fresh attempt with the agent's claude session
resumed (
RunServer.enter_same_node!), and the prompt is byte-identical to the parked attempt's —build_payloadexposes noanswervariable.
So editing the plan mid-run changes nothing any node reads. The answer arrives as a card
comment (Cards.answer_input/3) and the resumed reviewer reads it with relay card <ref>.
plan.md and card.plan stay stale by design; any lasting plan correction is a follow-up
card, not a mid-run mutation.
Escalation is deliberately rare: Fix remains the default, and a reviewer may escalate only when
it can quote the plan text that mandates the defect — the test being "can the implementer act on
this without contradicting the plan?". A reviewer that escalated because a finding was merely
hard would convert a self-healing loop into a human queue. On resume the reviewer resolves rather
than re-parking: "fix it anyway" returns Fix with the authorization quoted verbatim (which the
implementer is instructed to treat as outranking plan.md for that task), "waive it" returns
Approve with the waiver and follow-up recorded.
Where the contract's text lives — inline vs. .claude/agents/references/ (RE423). An agent
definition IS its system prompt: the .md file is loaded whole at invocation (--agent <name>)
and has no include mechanism, so nothing in a sibling file is ever passively in context. A
references/ file reaches the model only when the agent reads it at runtime, with a Read or
Bash tool call, from the card checkout it runs in. That fixes the rule for what goes where:
Inline what the agent needs in order to decide; put in
references/what it needs only after it has decided. An agent file is always in context. A reference file reaches the model only when the agent reads it at runtime, so its content must be something the agent only needs once it already knows it needs it.
- Escalation. The decision rule (
### Escalate sparingly, the quote-the-plan test) and the**Escalate**verdict in## Decidestay inline in each of the three reviewers — a reviewer that never reads the rule can never decide to escalate. The mechanics of how to escalate (what the question must say, the two options, what to do on resume) live once, in.claude/agents/references/escalating.md, which each reviewer points at.plan-implementercarries its own, role-specific escalation text and does not point atreferences/escalating.md. - Re-review. The trigger (a fix commit on top, or a findings block in the prompt), each
reviewer's own blocking rule and its own Pass/Approve line stay inline, because they decide the
verdict. How to conduct a re-review once the agent knows it is in one — re-check only the
findings, don't re-run the checklist, never re-raise a rebutted finding — lives once, in
.claude/agents/references/re-review.md.
Inline by design — these paragraphs appear word for word in more than one agent file on
purpose, and are the only such reviewer prose: (1) the re-review trigger paragraph, in all three
reviewers; (2) the blocking bullets and the "→ Approve" line shared by quality-reviewer and
final-reviewer; (3) the ### Escalate sparingly decision rule shared by quality-reviewer and
final-reviewer; (4) the one-line pointers to the two reference files.
Probe evidence (2026-10-08, claude --version → 2.1.294 (Claude Code)). Each of
spec-reviewer, quality-reviewer and final-reviewer was run headless with the runner's exact
flags (claude -p … --permission-mode auto --verbose --output-format stream-json --agent <name>,
cwd = the card checkout) and asked — without naming the path — to open the escalation reference,
then the re-review reference, its definition points to and reply with the first line. All six
runs read the right file and replied with its heading:
| Agent | escalation reference | re-review reference |
|---|---|---|
spec-reviewer |
Read .claude/agents/references/escalating.md |
Read .claude/agents/references/re-review.md |
quality-reviewer |
Bash head -n 1 .claude/agents/references/escalating.md |
Bash head -n 1 .claude/agents/references/re-review.md |
final-reviewer |
Bash head -n 1 .claude/agents/references/escalating.md |
Bash head -n 1 .claude/agents/references/re-review.md |
Replies: # Escalating a plan-mandated finding and # Re-reviewing after a fix pass. An earlier
version of this page said a references/ file "would never reach the model"; that is wrong for
any agent that reads it with a tool, and the probe is what settled it.
Shipping caveat for the relay-* skills. The three relay-* skills still cannot rely on a
sibling file — not because of how they load, but because of how they ship: Relay.Scaffold.items/0
names SKILL.md only, so a reference file next to one would silently vanish from every scaffolded
repo (which is why test/relay/onboard_skill_test.exs pins the relay-onboard directory to
SKILL.md alone). Agents are unaffected: Relay.Scaffold
ships no agents at all — every agent is the repo's own, per ADR 0010.
test/relay/agents/escalation_contract_test.exs pins the markers so an edit can't silently drop
the contract, and fails on any references/ path an agent file names that does not exist (a
dangling pointer).
Operating invariants
If you build your own runner or your own node behavior, honor these — break one and cards corrupt each other's work:
- One agent per working directory at a time. A
git checkout(or branch/file edit) is global to the directory — two agents on two branches in one directory overwrite each other. Serialize (one card at a time), or give each agent its own clone orgit worktree. Don't run the runner and an interactive session in the same working tree at once. - State lives on the board, never in the working tree. Many cards are in flight, moving back and forth between stages; a card may be specced now and planned days later while others pass through. Nothing durable may depend on what's currently checked out or on a shared repo-root scratch file.
- Each card owns its branch — check it out at the start of a step, commit at the end. Every
step must be self-contained: begin by checking out the card's branch (from its
branchfield), end by committing (never leave uncommitted changes for the next card to inherit). - Work travels with the card. The spec is the card's
spec; the acceptance criteria itsacceptance_criteria; the plan itsplanfield. Materialize these into the branch just-in-time (at the per-card$RELAY_PLANpath), never via a shared file another card would clobber.
Readiness, ordering, WIP and failure routing are not on this list: they are decided server-side by the scheduler and the engine (see "Dispatch is server-side" above, state.md and failures.md). A runner that tries to decide them locally will disagree with the server.
Sources of truth: ./relay, .relay/runner.json, bin/test_relay.py,
lib/relay_web/controllers/api/node_job_controller.ex, lib/relay/runs.ex,
lib/relay_web/controllers/api/board_controller.ex,
lib/relay_web/controllers/api/run_controller.ex,
lib/relay_web/controllers/api/runner_controller.ex,
lib/relay_web/controllers/api/flow_metrics_controller.ex,
lib/relay_web/controllers/api/flow_controller.ex,
lib/relay_web/controllers/api/talk_controller.ex, lib/relay/talk.ex,
lib/relay/scaffold.ex, lib/relay_web/controllers/api/scaffold_controller.ex.