MGS3012: invocation stalled

A run stopped making progress and magus aborted it. No target started, no target finished, and nothing wrote a line of output for the whole stall window, while every selected project's advisory lock stayed held:

[MGS3012] aborting a stalled run: nothing has started, finished or printed a line for
  15m0s, and every selected project's lock is still held.
  last step: docs:graph-generate (executing)
  stall window: 15m0s
  captured log: .magus/logs/9f2c11ab4e01.log
  override: set stall_timeout (MAGUS_STALL_TIMEOUT, --stall-timeout) higher, or negative
    to turn the watchdog off

The three facts in the message are the whole investigation: the target that ran last, how long ago that was, and the log holding whatever it managed to print before it went quiet.

What counts as progress

magus keeps one heartbeat per invocation, beaten at every point work is accounted:

• a target taking its limiter slot, machine claim and inflight record;
• the same target handing them back;
• every line a target's subprocesses write.

The last one is what separates a slow target from a wedged one. A target that runs for an hour while printing keeps beating the heartbeat and never approaches the window, so the watchdog does not bound how long work may take. It bounds how long a run may do nothing at all.

Two waits are supervised rather than stalled, and both beat while they wait:

• a step queued for the machine budget, which already prints "this run is NOT hung" on the same cadence;
• fs\watch, parked waiting for a file to change.

Any other way of blocking forever is already rejected: a command op runs to completion, and a never-exiting one is MGS5003.

Why it is not a target timeout

target_timeout and a target's own declared timeout (MGS3011) bound work that runs long, and only a target whose author declared a bound. This is the opposite check: an invocation making no progress, in work no target declared. The case it was built for is a post-batch pass, belonging to no target and so outside every ceiling, blocked on a network read while magus status reported an idle machine and the run journal's last event was an hour stale. Every observer was telling the truth; none of them was watching that gap.

The watchdog runs entirely inside the invocation. The daemon is an accelerant and never a capability gate, and a run stalling with nothing else up is exactly the case where nobody else is watching, so the net cannot depend on one.

Configuring the window

stall_timeout in magus.yaml, MAGUS_STALL_TIMEOUT, or --stall-timeout. It takes a Go duration:

stall_timeout: 30m

Unset uses the built-in 15 minutes, which is set against the longest legitimately silent stretch a run produces rather than its longest target: a cold module download, a link step, a container image pull, each single-digit minutes. A negative value turns the watchdog off.

Note the default differs from target_timeout, which is off unless you set it. A ceiling near a legitimate target's runtime fails builds that were fine, so it has to be opted into; a stall window is only reached by a run that has stopped working, so it is on.

Resolution

1. Read the captured log. The message names it. Whatever the last target printed before it went quiet is the closest thing to a cause.

2. Find what it was waiting on. A stalled step is almost always blocked on something outside magus: a network read with no timeout, a lock held by another process, a prompt on a subprocess that has no terminal. magus query output <ref> reads the same log by reference.

3. Give the target a deadline of its own. A target that can legitimately block should say so: "<target>": {"timeout": "15m"} in its magus.project policy. A ceiling fails faster than the watchdog does and names the target rather than the invocation.

4. Raise the window. If your workspace genuinely has a quiet stretch longer than the default, stall_timeout takes a bigger one. Prefer that over turning the watchdog off; the case it catches is the one nobody is watching.

See also

• MGS3009: the machine budget, and the other reason a step does not start.
• MGS3010: a gate deferred rather than stalled.
• MGS3011: a target that ran long rather than an invocation that stopped.
• Concurrency: the locks a stalled run holds.
• config: stall_timeout and target_timeout in the key inventory.