MGS3009: machine budget exhausted

A step was not started. What it needs does not fit in this machine's build budget alongside the claims every other live magus process is already holding.

Ordinarily this is not an error at all. The daemon that owns the budget queues the step and starts it the moment room frees, naming who holds the budget while it waits:

magus: (root) ci is queued for this machine's build budget, 1 ahead of it; held by
  pid 41221 (root) test (10.0 GiB), in /Users/x/repos/magus-worktrees/polish. This run
  starts automatically once room frees; set MAGUS_NO_WAIT=1 to fail fast instead.

The code fires in the three cases a wait cannot fix.

The caller asked not to wait. MAGUS_NO_WAIT=1 turns a queue into an answer:

[MGS3009] not starting (root) ci: this machine's build budget is full and
  MAGUS_NO_WAIT is set; it runs test, which declares 10.0 GiB and takes 6 slots, and
  2.0 GiB of 48.0 GiB is left and 2 of 8 slots are free; held by pid 41221 (root) test
  (10.0 GiB), in /Users/x/repos/magus-worktrees/polish.

The declaration cannot fit at all. No position in any queue seats a step whose declaration exceeds the whole budget, so magus says so immediately rather than waiting forever:

[MGS3009] refusing to start (root) ci: it runs test, which declares 64.0 GiB and takes
  8 slots, which does not fit in this machine's whole build budget of 48.0 GiB across
  8 slots. Waiting would not help; correct the declaration if it is wrong, or run
  this on a bigger machine.

A nested magus cannot tell who its parent is. magus passes MAGUS_INVOCATION_ANCESTORS to every magus it starts, and admission uses it to excuse a run from the claim its own parent is holding while it waits for that run to finish. A magusfile that clears the environment breaks that, and the affected run would queue behind a step that cannot move until the queued run completes - a deadlock the heartbeat would report as "not hung" indefinitely. So it refuses rather than queues:

[MGS3009] not starting docs ci: this magus runs underneath another one but was started
  without MAGUS_INVOCATION_ANCESTORS, so it cannot tell its own parent's claim from a
  stranger's and will not queue behind a run that is waiting for it; ... Pass that
  variable through to nested magus invocations, or let magus set it by not clearing
  the environment.

All three exit 75, not 1. That is EX_TEMPFAIL from sysexits.h, meaning "failed, try again later", and it is what separates a full machine from a broken build. A caller that cannot tell the two apart retries nothing and debugs a target that never ran. A run where real targets also failed exits 1 as usual: a broken build does not become a scheduling problem because a peer happened to be busy too.

Why

The concurrency limiter is per-process. Every magus invocation admits work up to its own capacity, and nothing sums those capacities, so N invocations across N worktrees admit N budgets' worth of work against one machine. That is not a theoretical concern on a developer machine running several agents at once: it is how a workstation ends up tens of gigabytes deep in swap with every individual run comfortably inside its own budget.

memory_mb is the declaration that makes this arbitrable, and the daemon is what arbitrates it. There is one daemon per user, so there is one budget per machine: every magus asks the same process, and the answer accounts for the worktrees this run knows nothing about. A run starts the daemon if none is up (and one started that way exits by itself once nothing has needed it for ten minutes), because admission has no other arbiter; see the daemon guide.

It queues rather than refusing. A queue only works when something can tell a waiter that its turn came, and a process can. The budget reserves room for the oldest waiter, so a stream of small runs cannot starve the ten-gigabyte gate they are queued in front of.

The budget is derived from the memory the daemon may commit, not from a reading of what is free right now. That keeps the verdict deterministic: the same command on the same machine reaches the same decision regardless of what a browser is doing, and the message names figures a reader can check afterwards. Observed pressure has a separate, advisory job - the memory-headroom warning a run emits while it works.

Targets that declare nothing claim no memory and are never held for that reason. An absent memory_mb means magus does not know what the target needs, and inventing a figure would gate work on a guess. Every step still claims its concurrency slots, which is the half every step spends.

A composed target inherits its chain's largest declaration. Only the target you name is scheduled as a step. So ci, which composes test through ctx.needs, would otherwise run the heaviest target in the workspace as if it had declared nothing, and magus affected ci is exactly the invocation that fills a machine. The maximum rather than the sum, because a chain runs its steps in order.

A run that loses the daemon finishes. Admission fails open: a daemon that dies mid-run leaves its claims behind with it, the run says once that it is no longer arbitrated, and it completes. Losing the arbiter must never fail a build that was going to succeed. A claim whose process died is retired by the next request; the daemon reaps by process liveness, so nothing has to release cleanly.

Resolution

1. Let a peer finish. The message names the pid, project, target and directory holding each claim, and magus status shows the whole budget across worktrees. Without MAGUS_NO_WAIT this happens on its own. Exit 75 is the signal a script or an agent should branch on to retry rather than investigate.

2. Check that the declaration is honest. A memory_mb well above what the target actually peaks at holds up work that would have fit. magus doctor reports a declaration that disagrees with the peak resident memory magus has measured across recent runs (MGS1030), and that figure is the one to declare.

3. Reduce what the target needs. A suite that declares ten gigabytes because every concurrent test binary carries the race detector's shadow memory can often be sharded, or run with lower in-tool parallelism, and then declare less.

4. Run on a bigger machine. The budget is a fraction of what the host has, so a target that cannot fit on an idle machine is a target this machine cannot run.

See also

• Concurrency: the three scopes magus coordinates, and where this one sits.
• Daemon and concurrency: the process that owns the budget, and why a run starts one.
• memory_mb: declaring what a target needs, and how magus converts it into concurrency slots within one process.
• MGS1030: the declaration that has drifted from what the target measurably uses.
• MGS3007: the other code about a run the process context makes impossible.