MGS3013: build slots deadlocked

Every slot in this run's concurrency pool is held by a step that is itself waiting for something, and other steps are queued for a slot. No slot can free, so nothing queued can start, so nothing can free a slot. magus refuses the wait instead of spending it:

[MGS3013] refusing to keep waiting for a build slot: all 8 of this run's slots are held
  by steps that are themselves waiting, so no slot can free and nothing queued can start.
  Holding: . build holds 1 and is waiting on 1 build slot(s); . ci holds 1 and is waiting
  on the cache lock for 4f1ac2be, held by . generate; ... Queued: . mocks-generate needs 1;
  . types-generate needs 1. The usual cause is a target reading what a target beside it
  writes with no ctx.needs between them (see ...); order the reader after the writer.
  Raising concurrency widens the window rather than closing it.

The message is the whole investigation: every holder, how many slots it has, and what each one is waiting for.

The rule this protects

magus bounds concurrency the way make's jobserver does: a slot is held while a recipe runs, and a step that stops running to wait for something hands its slots back first. A composed target yields its slots across its ctx.needs fan-out for exactly this reason, so the targets it composes can be admitted.

Two waits keep the slots: a step waiting for the per-key cache lock, and a step waiting for another slot while already holding one. Either is fine on its own. Both at once, across every holder, is a wait that nothing in the process can end.

What it usually means

An unordered reader inside one composer's chain (MGS4008). A reader that shares a step with the writer of the files it reads has no way to wait for it except to occupy its seat while the writer queues for one that will never come. MGS4008 refuses that plan before the run starts; this code catches the same shape when it arrives some other way, and names it in seconds instead of leaving the run to the stall watchdog (MGS3012).

Resolve it

1. Read the "Holding" list. Each entry says what that step is waiting for. One of them is the wait to remove.

2. Order the reader after the writer. When a holder is waiting for a slot while a generator is queued, they are almost certainly members of one chain: ctx.needs(<writer>) in the reader is the fix, and MGS4008 has the worked example.

3. Check for a target that dispatches magus recursively. A target running another magus command from its body waits on a child that has to be admitted by the same pool. Compose it with ctx.needs instead, which yields.

4. Do not reach for higher concurrency. It makes the wedge less likely on this machine and no less real: the same run on a smaller machine, or with one more project selected, deadlocks again.

What it does not fire on

A busy pool. Queuing behind steps that are working is ordinary, and holders that are running answer "not deadlocked" however long they take.

A momentary hand-off. A step releases its slots in a defer while its peer is already queued, so a sample taken in that instant looks like the wedge. The verdict requires the shape to hold continuously for a short grace before it is believed.

Slots reserved outside a step. A spell that reserves slots for a tool's own workers holds them without a step record, which leaves the accounted total short of capacity, and a wait that might still end is never refused.

See also

• MGS4008: the plan-time refusal for the authoring mistake that causes this.
• MGS3012: the stall watchdog, which bounds a run making no progress for any reason at all.
• MGS3009: the machine budget, the other reason a step does not start.
• Concurrency: slots, yielding, and the jobserver rule.