MGS3009: machine budget exhausted

A step was not started. What it needs does not fit in this machine's build budget alongside the claims other live magus invocations are already holding.

magus never queues behind another magus invocation, so a budget they fill is refused immediately, naming who holds it:

[MGS3009] not starting (root) ci: this machine's build budget is full; it runs test,
  which declares 10.0 GiB and takes 6 slots, and 2.0 GiB of 48.0 GiB is left and 2 of
  8 slots are free; held by pid 41221 (root) test (10.0 GiB), in
  /Users/x/repos/magus-worktrees/polish.

A step kept out only by claims of its own process or its own run (the siblings of one fan-out) waits for them instead, and never sees this code.

The code fires in three cases, none of which a retry-without-delay fixes.

The machine is full right now. The step above; the same command succeeds once the holder finishes.

The declaration cannot fit at all. No position in any queue seats a step whose declaration exceeds the whole budget, so magus says so immediately rather than waiting forever:

[MGS3009] refusing to start (root) ci: it runs test, which declares 64.0 GiB, and takes
  8 slots, which does not fit in this machine's whole build budget of 48.0 GiB across
  8 slots. Waiting would not help; that budget is 75% of the memory available here; the
  rest runs the OS and everything else. Run `magus doctor` and read MGS1030, which
  compares this declaration to the peak memory magus measured: correct the declaration
  if it has drifted; if it is honest, get a bigger machine.

A nested magus cannot tell who its parent is. magus passes MAGUS_INVOCATION_ANCESTORS to every magus it starts, and admission uses it to excuse a run from the claim its own parent is holding. A magusfile that clears the environment breaks that, and the affected run would be refused by a budget its own parent filled - a permanent deadlock, since the parent cannot finish until this run does:

[MGS3009] not starting docs ci: this magus runs underneath another one but was started
  without MAGUS_INVOCATION_ANCESTORS, so it cannot tell its own parent's claim from a
  stranger's and will not queue behind a run that is waiting for it; ... Pass that
  variable through to nested magus invocations, or let magus set it by not clearing
  the environment.

None of them exits 1, and they do not all exit the same way. A busy machine exits 75, EX_TEMPFAIL from sysexits.h, meaning "failed, try again later": the machine is full now and the same command runs once it drains. The other two exit 78, EX_CONFIG, because no amount of retrying changes their answer. A declaration larger than the whole budget and a nested magus with no ancestry both refuse identically on every retry, so a wrapper that retries on 75 stops on those instead of looping. A run where real targets also failed exits 1 as usual: a broken build does not become a scheduling problem because a peer happened to be busy too.

Why

The concurrency limiter is per-process. Every magus invocation admits work up to its own capacity, and nothing sums those capacities, so N invocations across N worktrees admit N budgets' worth of work against one machine. That is not a theoretical concern on a developer machine running several agents at once: it is how a workstation ends up tens of gigabytes deep in swap with every individual run comfortably inside its own budget.

memory_mb is the declaration that makes this arbitrable, and the broker is what arbitrates it. There is one broker per user, so there is one budget per machine: every magus asks the same process, and the answer accounts for the worktrees this run knows nothing about. A run starts the broker if none is up (and it exits by itself ten minutes after it last held anything), because admission has no other arbiter; see the broker and the server.

It refuses rather than queuing behind another invocation. magus never waits on another magus invocation, so a step that does not fit right now is told so immediately, naming who holds the budget, rather than parked until room frees. What it does wait for is its own run: a step kept out by claims its own process or its own root invocation holds waits for them, since refusing would fail a run over work it is already doing.

The budget is derived from the memory the broker may commit, not from a reading of what is free right now. That keeps the verdict deterministic: the same command on the same machine reaches the same decision regardless of what a browser is doing, and the message names figures a reader can check afterwards. Observed pressure has a separate, advisory job - the memory-headroom warning a run emits while it works.

Targets that declare nothing claim no memory and are never held for that reason. An absent memory_mb means magus does not know what the target needs, and inventing a figure would gate work on a guess. Every step still claims its concurrency slots, which is the half every step spends.

A composed target inherits its chain's largest declaration. Only the target you name is scheduled as a step. So ci, which composes test through ctx.needs, would otherwise run the heaviest target in the workspace as if it had declared nothing, and magus affected ci is exactly the invocation that fills a machine. The maximum rather than the sum, because a chain runs its steps in order.

A declaration is a ceiling, and measured runs size the claim under it. A step claims 1.25 times the highest peak magus recorded across successful runs with the same charms and forwarded args, or across every run of the target when that shape has fewer than three, and never more than the declaration. With fewer than three successful measured runs of any shape it claims the declaration. So a -- -run TestOne run claims what narrowed runs reach rather than what the whole suite does, and a target declaring nothing still claims only its slots.

A run that loses the broker finishes, unless you said otherwise. Under the default broker: best-effort, a broker that dies mid-run takes its claims with it, the run says once that it is no longer arbitrated, and it completes; its running steps re-assert their claims on the next broker to come up. Under broker: required a step that finds no broker is refused instead (MGS3022). Every claim rides its run's connection to the broker, so a run that dies, even to SIGKILL, releases its claims at once and nothing has to release cleanly.

Resolution

1. Retry once a peer finishes. The message names the pid, project, target and directory holding each claim, and magus status shows the whole budget across worktrees. Exit 75 is the signal a script or an agent should branch on to retry rather than investigate.

2. Check that the declaration is honest. A memory_mb well above what the target actually peaks at holds up work that would have fit. magus doctor reports a declaration that disagrees with the peak resident memory magus has measured across recent runs (MGS1030), and that figure is the one to declare.

3. Reduce what the target needs. A suite that declares ten gigabytes because every concurrent test binary carries the race detector's shadow memory can often be sharded, or run with lower in-tool parallelism, and then declare less.

4. Run on a bigger machine. The budget is a fraction of what the host has, so a target that cannot fit on an idle machine is a target this machine cannot run.

See also

• Concurrency: the three scopes magus coordinates, and where this one sits.
• The broker and the server: the process that owns the budget, and why a run starts one.
• MGS3022: the refusal when no broker answers under broker: required.
• memory_mb: declaring what a target needs, and how magus converts it into concurrency slots within one process.
• MGS1030: the declaration that has drifted from what the target measurably uses.
• MGS3007: the other code about a run the process context makes impossible.