magus v0.4.2 is out. See what's new
¶ View markdown source · ✎ Suggest an edit
3 min read

MGS3011: target exceeded its declared timeout

A target ran longer than the ceiling its magusfile declared. magus cancelled it, killed its process tree, and failed that target. The message carries every input to the decision, so a reader can reconstruct it without opening any code:

[fail] magus security (ran, 15m0s)
  cause: [MGS3011] target "security" exceeded its declared timeout of 15m after 15m0s;
    its process tree was killed; captured output:
    /repo/.magus/logs/9c4f...log

A timeout is a failure. Not a skip, not a warning, and never a pass: a target that was told it may take fifteen minutes and took more has not done its job, and a build tool that reports that as success is worse than one with no ceiling at all.

Why

The incident this exists for: a gate held three project locks for over an hour with zero concurrency slots in use and nothing running. No target failed, so nothing named a culprit, and the invocation had to be killed by hand. The likely cause was a network-bound scanner whose far end stopped answering - a shape with no natural failure mode, because waiting forever is what a socket does.

Nothing else in magus reclaims that. The cache waits on the target, the limiter waits on the cache, and the machine-wide budget (MGS3009) hands out claims that are only released when a run exits. One target that never returns holds all of it.

What was actually killed

The ceiling bounds the whole target, subprocesses included. magus starts every subprocess in its own process group for exactly this reason, so expiry sends SIGTERM to the group and then SIGKILL to whatever ignored it - grandchildren the target's own script forked included. There is no orphaned scanner still holding the network socket after this fires.

Because the deadline rides the context, a ceiling on a target also bounds every target it composes with ctx.needs. A composed target that declares its own, tighter ceiling is bounded by that one first, and the message then names the composed target rather than the gate that ran it - which is the difference between "ci hung" and "security hung".

Resolution

  1. Read the captured log the message names. The ceiling says when magus gave up, not what the target was doing. The log holds the last thing it printed, which is usually the answer.

  2. Fix the target, if the log shows it genuinely stopped making progress: an unreachable registry, a scanner database mirror that stalled, a prompt nobody is going to answer.

  3. Raise the ceiling, if the target legitimately got slower. Declare a multiple of the worst run on record, never a figure near its typical one:

    magus\project({
        "targets": {
            "security": {"timeout": "30m"},
        },
    });
    

    MGS1032 is the check that tells you when a ceiling has drifted into either mistake, measured against the durations magus recorded.

  4. Remove the declaration, if the target should not be bounded at all. An undeclared target is unbounded, which is the default and stays the default; nothing infers a ceiling for you.

See also

  • MGS1032: a declared ceiling that no longer describes the target.
  • MGS3009: the machine-wide budget a hung target holds while it hangs.
  • MGS3012: the other half - an invocation making no progress, in work no target declared a ceiling for.
  • Spells: what magus bounds, and target_timeout, the workspace-wide runaway guard this is the per-target form of.
MGS3011timeouttarget_timeoutrunawayhanglocksprocess groupdeclarations
Last updated (71633af0)
Earlier changes on this page (1)

Full history ↗ · Blame source ↗

Glossary

Workspace

The magus root directory that owns a set of projects and shared config; the unit magus operates over. See workspace.

Project

A directory magus recognizes as a unit of work (it has a magusfile); the unit of caching, scheduling, and dependency tracking. See workspace.

Magusfile

The magusfile.buzz that declares a project's targets (as export funs) and binds its spells. See targets.

Target

A named operation (build, test, ...) you invoke with magus run <target>; it may compose a spell's tool-native operations and depend on other targets. See targets.

Op

A single tool-native command a target composes (long form: operation); the middle of the work hierarchy (Spell to Op to Target). See operations.

Spell

A language/runtime adapter (e.g. go, md) that maps generic targets onto a toolchain's real commands. See spells.

Buzz

The language magusfiles are written in (the .buzz engine). See engines.

Cache

The content-addressed store magus consults before running a target, so unchanged work is skipped. See cache.

CI

An ordinary magusfile-defined target you compose yourself with magus\needs - magus does not hardcode its stages. Magus.RunCI treats it specially only in that it strips the rw charm, it is the anchor magus affected ci keys off, and a selected scope with no project declaring it is a load error rather than a silent no-op. See targets.

Slot

One unit of the pool's capacity. A target acquires the slots it needs to run (most take one) and releases them when it finishes; the pool tracks capacity (total slots), running (acquired), and queued (blocked). See daemon.

Concurrency

How many targets run at once. It is bounded by the pool's capacity and set with --concurrency, MAGUS_CONCURRENCY, or the concurrency config key. See daemon.

Lease

One row of the lease ledger: a piece of work an orchestrating agent handed out, with its goal, the checkpoint it was cut against, and the paths it owns or must not touch. The ledger records; the agent guard is what reads those facts back when grading a write. See doctrine.

Conventions

This page uses none of the site's convention markers. The full set is on the conventions page.