magus v0.4.3 is out. See what's new
¶ View markdown source · ✎ Suggest an edit
7 min read

How orchestration works

When one agent session hands work to several, magus splits the work into three roles. Each role has a boundary, and the guard enforces every one of them from the same record: the job row. This page covers who does what and what stops a role from doing someone else's work. The loop itself (checkpoint, partition, fork, release, verify) is in Jobs and leases, which has its own figure.

{"figure":"agent-orchestration","stamp":"sha256:4629f60ec014e9a9fe5de54d4d8b1cf45a44c1349d505807dfe2179e3f2d4f8e"}Three roles, one store, and a guard at every boundaryThe orchestrator forks job rows into one JSON Lines store and waits on them; it never does groundwork. Each worker is bound to one row: its write paths, its one check, its own checkout. The integrator merges the finished branches with git merge --no-ff, regenerates once and runs the full checks. The guard reads the row on every write and command, the sandbox narrows a run's writes to the same row, and the broker holds the one build budget every checkout on the machine shares.ORCHESTRATIONThree roles, one store, and a guard at every boundaryTHE ROLESWHAT MAGUS HOLDS AND ENFORCESFORK, WAIT./MAGUS, EXECEACH CALLMERGE, BUILDLEASESEATSOrchestratorthe root sessionWorktree bootstrapplaces ./magusWorkerone row, one checkIntegrator<parent>/integrateJob storejobs.jsonl, one per repoGuardevery write and commandSandboxwrite grant from the rowBrokerone machine budgetvcscheckpoint, mergeLEGENDTHE ONE ENFORCERTHE DECLARED ROWSAGENT SESSIONSA SCRIPT
role who it is what it may do
orchestrator the root session fork rows, spawn, magus job wait, the final gate, push, open the PR
worker a subagent bound to one row write its row's write paths, run its row's check, exit its row
integrator the subagent spawned <parent>/integrate <job> merge finished branches, build ./magus, regenerate, run the full checks

Above all three is the human in the loop. What lands, what is published and what a round may spend are that person's calls.

The orchestrator plans and never does groundwork

The orchestrator is the root session. It owns the goal, the partition and the final verification, and it hands out everything else:

  1. Record the working state with magus vcs checkpoint -o name.
  2. Partition the work by write set from graph evidence (magus refs, magus explain, magus affected ci --plan --stdin), as the magus-multi-agent skill lays out.
  3. Fork one row per unit with magus job fork: criteria, write paths, one check, a model tier, the checkpoint.
  4. Spawn each worker with the description <parent>/<role> <job>, which binds the worker to that row.
  5. Collect with magus job wait <job>, which grades the filed result against the diff since the checkpoint and against a recorded passing run of the job's own check.
  6. Run magus affected ci once, in its own tree, after every unit lands, then push and open the pull request.

The boundary is a design principle "the orchestrator plans; workers do the work", and the guard's orchestrator-groundwork rule enforces it. While a row the session handed out is declared, running or exited, the guard refuses that session's recursive searches, source reads, full diffs, target runs, edits and integration. It lets the job verbs, graph reads, branch work, the final gate, git push and gh pr through. The role comes from the live lease, never from the session's history or its context size, and a person at a terminal is never judged by it.

The root may take one row itself with magus job exec <row>. It spawns nothing while it holds that row, and magus job exit <row> returns it to orchestrating. The one build it runs itself is the fix for a missing or stale ./magus: ./magus run go-build . or the bootstrap, alone on its line.

A worker holds one row

A worker is bound to exactly one job row by its lease. The row is everything it is handed, and magus describe job <job> renders it the same way every time:

  • Write paths. Exact files, or one declaration of a file (run.go#executeStages), so two workers can share a file without sharing a declaration. A write outside them, or into a path another live job holds, is refused and names the owner.
  • One check. The narrowest target that covers the edits, never the gate. The guard's worker-check-only rule refuses any other target, except the generator of an output the worker's own write paths make stale.
  • Its own checkout and branch. A worker names its checkout on every magus line, and the guard refuses a magus call that resolves outside it.
  • Its result. The last act is magus job exit <job> --stdin with the changed paths, the check's command and output ref, and the risks it could not settle. There is no passed field: magus job wait reads the outcome from the recorded run.

Isolation levels

magus job fork chooses where each job runs, cheapest first, and records the reason:

level where the job runs chosen when
1 the forker's checkout a read-only job, or a disjoint writer at a high risk_tolerance
2 one scratch checkout its wave shares, with one ./magus a light writer, or one touching a file the workspace loads beside a peer
3 its own checkout, with the forker's ./magus placed in it a writer by default, a generator's output, or an overlap with no claim
4 its own checkout, where the holder builds ./magus a write path is a build input of ./cmd/magus

A write path covering a magusfile, magus.yaml or a spell source a magusfile imports never shares a checkout with another writer: half-saved, it stops the workspace loading for everyone in that checkout.

When a worker retries

A worker reruns only its own check. When the machine's build budget refuses that check (MGS3009), the worker may wait for a seat with magus status --wait | magus run <its check>, which keeps the seat the wait sized for the run it pipes into. Any other wait, or a bare rerun within the minute, is refused.

Retrying a whole job is not the worker's. magus job fork <same id> on an ended job starts its next attempt and folds the one that ended into the row's prior, which the next holder reads in magus describe job. magus never retries on its own, and an attempt it marks not retryable names what must change first.

The integrator merges, regenerates and gates

Each checkout that lands work has one integrator: the subagent spawned <parent>/integrate <job>, unisolated, in that checkout. It runs everything the workers and the root are refused there: the build of ./magus, the generate targets, and the full checks, one at a time.

An integration job is a row whose integrates names the finished branches it merges:

  • magus job fork computes its write paths from the files those branches changed since its checkpoint, and notes a live holder of one of them instead of refusing, since the merge settles it.
  • The guard lets its holder run git merge --no-ff <branch> for each branch the row names, and no other merge.
  • It regenerates declared outputs once, after the merges and before the full test run, so a drift test does not fail for nothing.

magus counts the finished branches waiting off the base. From jobs.integrate_advise_at other forks are advised to integrate first, and from jobs.integrate_refuse_at they are refused.

The guard enforces every boundary

The guard is the one reader that turns a row into a verdict. It runs on every file write and every command the host reports, reads the job store for the lease the caller acts under, and denies what crosses that lease's boundary with the command to run instead. The full table of what it refuses under a lease is in Jobs and leases.

The sandbox reads the same row for a run: when the sandbox is on, every target run and magus buzz script in a worker's checkout gets a write grant of exactly its write paths, the cache and a private temp directory. A boundary written twice would disagree with itself, so both read one declaration.

The job store is one jobs.jsonl per repository

Every row lives in <XDG state>/magus/jobs/<repo>/jobs.jsonl, one line per version of a row, the last line per id winning. Every worktree and clone of one repository reads the same set, which is how an orchestrator forks in its own checkout and a worker takes the lease from another.

The store enforces one thing itself: one author per job. A holder may record its base, shrink its own write paths, fork a child inside them and end its own job. Widening a boundary and verifying a result are the forking session's.

The broker holds one budget per machine

Editing costs nothing; validation contends. Every magus run on the machine, in every worktree, takes its slots and memory from one budget the broker holds, so N workers each running a check cannot start N budgets of work. A step the budget cannot seat is refused at once with MGS3009, naming who holds it, rather than queued. magus status shows what is held and by whom. See Concurrency.

Each checkout gets its own ./magus

A worktree starts with no ./magus, and the guard's stale-binary rule refuses calls that change state until a magus that loads the tree is in place. A leased worker is never told to build: there is one binary per base. The worktree bootstrap, hack/dev/bootstrap-worktree.buzz, places ./magus, installs the harness skills, runs magus job exec, checks the fit and builds the graph, in that order, so an edit lands only after the base is recorded.

magus job fork places the forker's ./magus in a new checkout whenever no build input of ./cmd/magus differs from its commit. A job that changes one runs at isolation level 4, where the holder builds its own.

agentsorchestrationorchestratorworkerintegratorjobsleasesguardbrokerbootstrap
Last updated (2df714e8)
Earlier changes on this page (1)

Full history ↗ · Blame source ↗

Glossary

Workspace

The magus root directory that owns a set of projects and shared config; the unit magus operates over. Workspace in full.

Magusfile

The magusfile.buzz that declares a project's targets (as export funs) and binds its spells. Magusfile in full.

Target

A named operation (build, test, ...) you invoke with magus run <target>; it may compose a spell's tool-native operations and depend on other targets. Target in full.

Op

A single tool-native command a target composes (long form: operation); the middle of the work hierarchy (Spell to Op to Target). Op in full.

Spell

A language/runtime adapter (e.g. go, md) that maps generic targets onto a toolchain's real commands. Spell in full.

Buzz

The language magusfiles are written in (the .buzz engine). Buzz in full.

Cache

The content-addressed store magus consults before running a target, so unchanged work is skipped. Cache in full.

Affected

The set of projects touched by a change; magus affected <target> runs a target only over them. Affected in full.

Sandbox

The restricted filesystem and environment a target runs in, so builds stay reproducible and side-effect-free. Sandbox in full.

Broker

The per-user background process that holds this host's capacity: the machine budget every run claims slots from, and the shared services runs keep warm. A run starts it on demand; broker: off in magus.yaml runs without one. Broker in full.

CI

An ordinary target you compose yourself with ctx.needs; magus does not hardcode its stages. It is the anchor magus affected ci keys off, and ci strips the rw charm. CI in full.

Slot

One unit of the pool's capacity. A target acquires the slots it needs to run (most take one) and releases them when it finishes; the pool tracks capacity (total slots), running (acquired), and queued (blocked). Slot in full.

Concurrency

How many targets run at once. It is bounded by the pool's capacity and set with --concurrency, MAGUS_CONCURRENCY, or the concurrency config key. Concurrency in full.

Queued

A target that wants a slot while the pool is full; it blocks first-in-first-out until a slot frees. The dashboard colors a sample with queued > 0 accordingly. Queued in full.

Agent

A model working through an agent host such as Claude Code, Codex, Cursor or OpenCode. magus treats an agent as one more caller of its CLI, MCP tools and console, held to the same checks as a person. Agent in full.

Harness

The deterministic scaffolding around the work on a change, for a person or an agent: compiler errors, formatters, linters, checks, gates, leases and hooks, placed at points a person chose. Around an agent's loop, magus supplies one and keeps it outside. Harness in full.

Skill

A packaged set of instructions an agent host loads when a task matches its description: a directory holding a SKILL.md. magus ships its magus-* skills in the binary and writes them with magus agent install. Skill in full.

Session

An agent host's conversation, by the id the host delivers to its hooks. magus never mints one: a record with no session is unattributed, and the OS user it carries says whose account ran it. Session in full.

Job

The unit of delegated work, and one row of the job store: what an orchestrating agent handed out, with its goal, the checkpoint it was cut against, the paths it may write or must not touch, and the one check it runs. A job's holder is either a session, for work an orchestrator handed out, or the daemon, for its own maintenance. The store records; the agent guard is what reads those facts back when grading a write. Job in full.

Run

One target executing under one magus invocation, such as magus run test web or magus affected ci. A run keeps its captured output behind an output reference. Every magus run is a run whether or not any job asked for it; see Job for how the two relate. Run in full.

Lease

The grant a holder takes on a job: the write and read paths that job declared, enforced in the checkout that took it with magus job exec. A job is the piece of work; a lease is permission over it. Lease in full.

Conventions

Placeholders

Angle brackets mark a value you replace with your own; never type the brackets:

magus run <target>
magus completion <shell>    # e.g. bash, zsh, fish

<target>, <path>, <shell>, <name> and the like are stand-ins, not literal text.