---
title: "MGS3012: invocation stalled"
description: Fires when a run holding every selected project's lock has gone a whole stall window without a target starting, finishing, or printing a line, so magus aborts it instead of letting it hold the workspace indefinitely.
tags: [MGS3012, stall, watchdog, hang, deadlock, locks, stall_timeout, target_timeout, progress]
---

# MGS3012: invocation stalled

A run stopped making progress and magus aborted it. No target started, no target
finished, and nothing wrote a line of output for the whole stall window, while every
selected project's advisory lock stayed held:

```text
[MGS3012] aborting a stalled run: nothing has started, finished or printed a line for
  15m0s, and every selected project's lock is still held.
  last step: docs:graph-generate (executing)
  stall window: 15m0s
  captured log: .magus/logs/9f2c11ab4e01.log
  override: set stall_timeout (MAGUS_STALL_TIMEOUT, --stall-timeout) higher, or negative
    to turn the watchdog off
```

The three facts in the message are the whole investigation: the target that ran last,
how long ago that was, and the log holding whatever it managed to print before it went
quiet.

## What counts as progress

magus keeps one heartbeat per invocation, beaten at every point work is accounted:

- a target taking its limiter slot, machine claim and inflight record;
- the same target handing them back;
- every line a target's subprocesses write.

The last one is what separates a slow target from a wedged one. A target that runs for
an hour while printing keeps beating the heartbeat and never approaches the window, so
the watchdog does not bound how long work may take. It bounds how long a run may do
nothing at all.

Two waits are supervised rather than stalled, and both beat while they wait:

- a step queued for the [machine budget](MGS3009.md), which already prints "this run is
  NOT hung" on the same cadence;
- `fs\watch`, parked waiting for a file to change.

Any other way of blocking forever is already rejected: a command op runs to completion,
and a never-exiting one is [MGS5003](../services/MGS5003.md).

## Why it is not a target timeout

`target_timeout` and a target's own declared `timeout` ([MGS3011](MGS3011.md)) bound
work that runs **long**, and only a target whose author declared a bound. This is the
opposite check: an invocation making **no** progress, in work no target declared. The
case it was built for is a post-batch pass, belonging to no target and so outside every
ceiling, blocked on a network read while `magus status` reported an idle machine and the
run journal's last event was an hour stale. Every observer was telling the truth; none
of them was watching that gap.

|                                 | Bounds                                | Covers                                          | Default |
| ------------------------------- | ------------------------------------- | ----------------------------------------------- | ------- |
| Ceiling ([MGS3011](MGS3011.md)) | one target running **long**           | only a target whose author declared a bound     | off     |
| Watchdog (this code)            | one invocation making **no** progress | everything the invocation does, declared or not | 15m     |

The watchdog runs entirely inside the invocation. The daemon is an accelerant and never
a capability gate, and a run stalling with nothing else up is exactly the case where
nobody else is watching, so the net cannot depend on one.

## Configuring the window

`stall_timeout` in `magus.yaml`, `MAGUS_STALL_TIMEOUT`, or `--stall-timeout`. It takes a
Go duration:

```yaml
stall_timeout: 30m
```

Unset uses the built-in **15 minutes**, which is set against the longest legitimately
silent stretch a run produces rather than its longest target: a cold module download, a
link step, a container image pull, each single-digit minutes. A negative value turns the
watchdog off.

Note the default differs from `target_timeout`, which is off unless you set it. A
ceiling near a legitimate target's runtime fails builds that were fine, so it has to be
opted into; a stall window is only reached by a run that has stopped working, so it is
on.

## Resolution

1. **Read the captured log.** The message names it. Whatever the last target printed
   before it went quiet is the closest thing to a cause.

2. **Find what it was waiting on.** A stalled step is almost always blocked on something
   outside magus: a network read with no timeout, a lock held by another process, a
   prompt on a subprocess that has no terminal. `magus query output <ref>` reads the
   same log by reference.

3. **Give the target a deadline of its own.** A target that can legitimately block
   should say so: `"<target>": {"timeout": "15m"}` in its `magus.project` policy. A
   ceiling fails faster than the watchdog does and names the target rather than the
   invocation.

4. **Raise the window.** If your workspace genuinely has a quiet stretch longer than the
   default, `stall_timeout` takes a bigger one. Prefer that over turning the watchdog
   off; the case it catches is the one nobody is watching.

## See also

- [MGS3009](MGS3009.md): the machine budget, and the other reason a step does not start.
- [MGS3010](MGS3010.md): a gate deferred rather than stalled.
- [MGS3011](MGS3011.md): a target that ran long rather than an invocation that stopped.
- [Concurrency](../../../concepts/concurrency.md): the locks a stalled run holds.
- [config.md](../../config.md): `stall_timeout` and `target_timeout` in the key inventory.