magus v0.4.3 is out. See what's new
¶ View markdown source · ✎ Suggest an edit
3 min read

MGS3011: target exceeded its declared timeout

A target ran longer than the ceiling its magusfile declared. magus cancelled it, killed its process tree, and failed that target. The message carries every input to the decision, so a reader can reconstruct it without opening any code:

[fail] magus security (ran, 15m0s)
  cause: [MGS3011] target "security" exceeded its declared timeout of 15m after 15m0s;
    its process tree was killed; captured output:
    /repo/.magus/logs/9c4f...log

A timeout is a failure. Not a skip, not a warning, and never a pass: a target that was told it may take fifteen minutes and took more has not done its job, and a build tool that reports that as success is worse than one with no ceiling at all.

Why

The incident this exists for: a gate held three project locks for over an hour with zero concurrency slots in use and nothing running. No target failed, so nothing named a culprit, and the invocation had to be killed by hand. The likely cause was a network-bound scanner whose far end stopped answering - a shape with no natural failure mode, because waiting forever is what a socket does.

Nothing else in magus reclaims that. The cache waits on the target, the limiter waits on the cache, and the machine-wide budget (MGS3009) hands out claims that are only released when a run exits. One target that never returns holds all of it.

What was actually killed

The ceiling bounds the whole target, subprocesses included. magus starts every subprocess in its own process group for exactly this reason, so expiry sends SIGTERM to the group and then SIGKILL to whatever ignored it - grandchildren the target's own script forked included. There is no orphaned scanner still holding the network socket after this fires.

Because the deadline rides the context, a ceiling on the target you SCHEDULED bounds every target it composes with ctx.needs. A composed target that declares its own, tighter ceiling is bounded by that one first, and the message then names the composed target rather than the gate that ran it, which is the difference between "ci hung" and "security hung".

That tighter ceiling stops where the composed target's own dependencies begin. A dependency is dispatched once and awaited by everyone that needs it, so a ceiling declared halfway down the chain would govern work its siblings depend on too: security declaring fifteen minutes must not decide how long generate may take, because lint, build and test all wait on the same run of it. A dependency reached this way stays bounded by the scheduled unit's ceiling, and by the stall watchdog (MGS3012).

Resolution

  1. Read the captured log the message names. The ceiling says when magus gave up, not what the target was doing. The log holds the last thing it printed, which is usually the answer.

  2. Fix the target, if the log shows it genuinely stopped making progress: an unreachable registry, a scanner database mirror that stalled, a prompt nobody is going to answer.

  3. Raise the ceiling, if the target legitimately got slower. Declare a multiple of the worst run on record, never a figure near its typical one:

    magus\project({
        "targets": {
            "security": {"timeout": "30m"},
        },
    });
    

    MGS1032 is the check that tells you when a ceiling has drifted into either mistake, measured against the durations magus recorded.

  4. Remove the declaration, if the target should not be bounded at all. An undeclared target is unbounded, which is the default and stays the default; nothing infers a ceiling for you.

See also

  • MGS1032: a declared ceiling that no longer describes the target.
  • MGS3009: the machine-wide budget a hung target holds while it hangs.
  • MGS3012: the other half - an invocation making no progress, in work no target declared a ceiling for.
  • Spells: what magus bounds, and target_timeout, the workspace-wide runaway guard this is the per-target form of.
MGS3011timeouttarget_timeoutrunawayhanglocksprocess groupdeclarations
Last updated (0c2c811f)
Earlier changes on this page (2)

Full history ↗ · Blame source ↗

Glossary

Workspace

The magus root directory that owns a set of projects and shared config; the unit magus operates over. See workspace.

Project

A directory magus recognizes as a unit of work (it has a magusfile); the unit of caching, scheduling, and dependency tracking. See workspace.

Magusfile

The magusfile.buzz that declares a project's targets (as export funs) and binds its spells. See targets.

Target

A named operation (build, test, ...) you invoke with magus run <target>; it may compose a spell's tool-native operations and depend on other targets. See targets.

Op

A single tool-native command a target composes (long form: operation); the middle of the work hierarchy (Spell to Op to Target). See operations.

Spell

A language/runtime adapter (e.g. go, md) that maps generic targets onto a toolchain's real commands. See spells.

Buzz

The language magusfiles are written in (the .buzz engine). See engines.

Cache

The content-addressed store magus consults before running a target, so unchanged work is skipped. See cache.

CI

An ordinary magusfile-defined target you compose yourself with magus\needs - magus does not hardcode its stages. Magus.RunCI treats it specially only in that it strips the rw charm, it is the anchor magus affected ci keys off, and a selected scope with no project declaring it is a load error rather than a silent no-op. See targets.

Slot

One unit of the pool's capacity. A target acquires the slots it needs to run (most take one) and releases them when it finishes; the pool tracks capacity (total slots), running (acquired), and queued (blocked). See server.

Concurrency

How many targets run at once. It is bounded by the pool's capacity and set with --concurrency, MAGUS_CONCURRENCY, or the concurrency config key. See server.

Invocation

One magus process's recorded facts - the targets it finished, their outcomes, the lease it acted as, and the session it ran in when a host delivered one - kept in a repo-scoped store every worktree shares. magus session lists them; the store prunes itself by last-fact age.

Job

The unit of delegated work, and one row of the job store: what an orchestrating agent handed out, with its goal, the checkpoint it was cut against, the paths it may write or must not touch, and the one check it runs. A job's holder is either a session, for work an orchestrator handed out, or the server, for its own maintenance. The store records; the agent guard is what reads those facts back when grading a write. See doctrine.

A job is not a run. magus run build web is a run, and no job exists for it. A job causes runs: its check executes as one, and a server job records the invocation of its last one. Jobs are listed with magus ls jobs and in the console's Jobs view; runs are listed in the Runs view.

Run

One target executing under one magus invocation, such as magus run test web or magus affected ci. A run keeps its captured output behind an output reference. Every magus run is a run whether or not any job asked for it; see Job for how the two relate.

Lease

The grant a holder takes on a job: the write and read paths that job declared, enforced in the checkout that took it with magus job exec. A job is the piece of work; a lease is permission over it.

Conventions

This page uses none of the site's convention markers. The full set is on the conventions page.