magus v0.4.2 is out. See what's new
¶ View markdown source · ✎ Suggest an edit
3 min read

MGS1030: memory declaration disagrees with what was measured

magus doctor compared each target's declared memory_mb against the peak resident memory magus recorded the last several times it ran, and they disagree.

advice  memory declarations
        2 target(s) declare memory that disagrees with what magus measured;
        admission is only as good as the declarations it arbitrates
        - console test declares no memory_mb and reached at least 5312MB; an undeclared
          target claims no memory, so every other magus on the machine is blind to it
        - docs site-generate declares 2048MB and reached at least 6100MB; admission
          seats it against a figure it exceeds

Why

memory_mb is not decoration, and it does two jobs. Inside a run it converts to concurrency slots on the limiter, so a target declaring a large figure holds several slots and throttles the peers magus would otherwise start alongside it. A 16GB runner and a 64GB workstation get different parallelism from the same declaration, without the magusfile naming either machine. Across runs it is the figure the daemon arbitrates machine-wide, so a target that says what it needs queues behind the peers in other worktrees instead of racing them (MGS3009).

That makes both exactly as good as the declarations, and declarations rot quietly. A suite written at 2GB grows over a year into one that reaches 9GB, and nothing says so: the target keeps passing, it keeps taking one slot's worth of room, and the protection is gone long before anyone notices.

magus already measures the resident memory of every process a target runs, for every run, so the disagreement is a fact it holds rather than a question the author has to think to ask.

Two shapes are reported:

  • Under-declared: the measurement is more than a quarter above the declaration, by at least 512MB. This is the dangerous direction.
  • Undeclared: no memory_mb at all, and a measurement of at least 2GB. An undeclared target claims no memory, so magus schedules its peers alongside it however heavy it turns out to be, and every other magus on the machine is blind to it.

Below those thresholds nothing is reported: the run-to-run variation in measuring a small target is larger than the disagreement would be.

What the measurement is, and is not

The recorded figure is a floor, not a true peak. magus folds a target's processes as a maximum, and the kernel does the same upstream: wait4 propagates a subtree's peak by taking the largest process, never the sum. A driver that forks four concurrent 800MB children reports 800MB. So a suite that fans out, which go test does by default, records roughly its largest single binary rather than what its tree held together.

Both findings above survive that, because a floor only argues in the safe direction: a target that reached at least 2GB still wants a declaration, and a declaration a floor already exceeds is exceeded by more.

The symmetric finding does not survive it, which is why magus does not report an over-declared target. "You declared 10GB and never used more than 3.5GB" needs a real peak to be true, and a figure that undercounts a parallel suite would argue for lowering exactly the declarations written from a machine that actually died. If a declaration looks generous, measure the target's whole process tree before trusting the gap.

Resolution

  1. Declare what magus measured, rounded up to leave headroom for a heavier input than the runs on record:

    magus\project({
        "target_policies": {
            "test": {"memory_mb": 10240},
        },
    });
    
  2. Leave it alone deliberately. A ceiling above the measurement is a legitimate thing to declare when a target's memory varies with its input, or when the figure came from watching a real machine die rather than from a recorded run. Nothing will nag you about the gap: this check reports only under-declaration and non-declaration, and it is advice rather than a failure so the decision stays yours.

  3. Reduce what the target needs, if the measurement is the surprise rather than the declaration. A suite whose peak is dominated by the race detector's shadow memory across concurrent test binaries can often be sharded or run with lower in-tool parallelism.

See also

  • memory_mb: declaring what a target needs.
  • MGS3009: the machine-wide budget these declarations are arbitrated against.
MGS1030memory_mbdoctormemorydeclarationsadmission
Last updated (cc3d21b1)
Earlier changes on this page (1)

Full history ↗ · Blame source ↗

Glossary

Project

A directory magus recognizes as a unit of work (it has a magusfile); the unit of caching, scheduling, and dependency tracking. See workspace.

Magusfile

The magusfile.buzz that declares a project's targets (as export funs) and binds its spells. See targets.

Target

A named operation (build, test, ...) you invoke with magus run <target>; it may compose a spell's tool-native operations and depend on other targets. See targets.

Op

A single tool-native command a target composes (long form: operation); the middle of the work hierarchy (Spell to Op to Target). See operations.

Buzz

The language magusfiles are written in (the .buzz engine). See engines.

Sandbox

The restricted filesystem and environment a target runs in, so builds stay reproducible and side-effect-free. See sandbox.

Daemon

The background magus host that owns shared state such as services and the warm knowledge graph. See daemon.

CI

An ordinary magusfile-defined target you compose yourself with magus\needs - magus does not hardcode its stages. Magus.RunCI treats it specially only in that it strips the rw charm, it is the anchor magus affected ci keys off, and a selected scope with no project declaring it is a load error rather than a silent no-op. See targets.

Slot

One unit of the pool's capacity. A target acquires the slots it needs to run (most take one) and releases them when it finishes; the pool tracks capacity (total slots), running (acquired), and queued (blocked). See daemon.

Concurrency

How many targets run at once. It is bounded by the pool's capacity and set with --concurrency, MAGUS_CONCURRENCY, or the concurrency config key. See daemon.

Conventions

This page uses none of the site's convention markers. The full set is on the conventions page.