magus v0.3.0 is out. See what's new
¶ View markdown source · ✎ Suggest an edit
30 min read

Cache model

magus's build cache is content-addressed: a target's outputs are keyed by the SHA-256 of its inputs, so an unchanged target replays its previous outputs instead of rerunning. This page is the local model: what magus hashes, what invalidates a key, what "replay" restores, and where it all lives on disk. The remote cache shares these same artifacts across machines and layers a signed trust model on top; this page is the substrate it references, so we describe it once here and link there for the distributed story.

Design intent

  • Correctness is a declaration contract. magus caches what a target declares, not what it touches. A target's needs, provides, and claims (see below) define its whole cache footprint. Under-declare an input and a stale hit slips through; over-declare an output and every replay snapshots more than necessary. The cache is only as correct as those declarations, which is why the vocabulary is explicit rather than inferred from a traced filesystem.
  • Identical inputs replay. The key is a pure function of the inputs. Two runs with byte-identical sources, tool versions, charms, and dependency keys produce the same key and so the same hit. Nothing about wall-clock time, machine, or run order enters the key.
  • A hit never runs the body. On a hit magus restores the recorded outputs and emits the result event; the target's export fun never executes. The saved work is the point.
  • It is just files. The store is a directory of blobs, JSON manifests, and captured logs under .magus/. There is no database and no daemon in the read path. You can ls it, cat a manifest, and reason about a hit or miss with ordinary tools.

needs, provides, claims: a target's cache footprint

A bound spell contributes three glob sets to its project. Only operations are runnable; these three are metadata that make caching and the affected set correct (see What a spell provides). Binding a spell contributes its needs/provides/claims to a project's cache key and affected set even before you wire a target.

Declaration What it is Role in the cache
needs input globs (the sources) hashed into the cache key; also seed the affected set
provides output globs snapshotted into the cache on a miss and replayed on a hit
claims files the spell owns affected-set attribution only; not hashed, not snapshotted

Internally these map to a Step the cache hashes and replays: needs become Step.Sources, provides become Step.Outputs. claims do not appear in the Step at all: they attribute changed files to a project for affected-set computation and never touch the cache key or the snapshot. Two rules follow directly:

  • Declare every input in needs. A source file that isn't matched by a needs glob doesn't enter the key, so editing it produces no miss and you replay a stale build.
  • Keep provides tight and complete. Under-declare and the cache can't replay an output it never recorded; over-declare and every hit restores files that were never outputs.

The output tree is never treated as an input: source expansion excludes the provides globs and prunes their static directory prefixes, so a generated file can't feed back into its own key.

Per-target inputs and outputs

A spell contributes its globs to every target on the project. To attach a glob to one target, declare it in that target's body with ctx.readsFiles(...) / ctx.writesFiles(...):

export fun build(ctx: magus\Context, args: [str]) > void {
    ctx.readsFiles("schema/**", "codegen.config.json");
    ctx.writesFiles("dist/**");
    go["go-build"](ctx);
}

An explicit ctx.readsFiles(...) declaration defines that target's source footprint; an explicit ctx.writesFiles(...) declaration defines its snapshot/replay footprint. Magus retains the magusfiles and any spell sources specific to that target, but it does not inherit the broad project baseline. This is what lets one target be precise without making its siblings under-declared (see Granularity).

Files a target edits rather than produces

The declaration names encode ownership, not merely direction:

Declaration File relationship Cache and clean behavior
ctx.readsFiles(...) the target reads the named files hashes their current bytes into the cache key
ctx.writesFiles(...) the target creates or replaces complete generated files snapshots and replays them; magus clean may remove them
ctx.modifiesExistingFiles(...) the files already exist and the target changes only part of each one hashes their current bytes, but never snapshots, replays, or removes them

That last case is for a hand-written page with a generated region between markers, or a manifest a tool rewrites in place. It is deliberately not an output Magus owns.

export fun content_generate(ctx: magus\Context, args: [str]) > void {
    ctx.writesFiles("reference/buzz/*.md");          // created and fully owned
    ctx.modifiesExistingFiles("concepts/spells.md"); // existing page; only the table changes
}

ctx.modifiesExistingFiles is never deleted by magus clean and never replayed from a snapshot, because the bytes magus produced are only part of the file. It still folds into the target's cache key exactly as an input does - so editing the prose around a generated region invalidates the target that maintains that region, which declaring the file as an output could not do (an output is excluded from its own source hash).

Unlike reads and writes, a modification infers no ordering edge in either direction: "I edit one region of a file someone else authored" says nothing about build order. Declare ctx.needs if you need it.

The globs are read statically, before the target runs - a cache hit skips the body, so the run can't be the source of truth. magus recovers them from the source: it walks each target body and the helpers it calls by name, collecting the string-literal globs. Two disciplines follow, both enforced:

  • A non-literal argument (ctx.readsFiles(someVar)) is a magusfile load error - a computed glob is invisible to the static read, and silently dropping it would risk a stale hit.
  • A call the walk can't reach (in an unreferenced helper, or the identifier used as a value) never enters a key; magus doctor flags it as MGS1004.

This shares the literal-first discipline of magus\needs: declare the footprint at the target, in literals magus can see.

The cache key

The key is the hex SHA-256 of a deterministic, newline-delimited serialization of the Step. magus writes these lines, in this order, into one hash:

  • keyVersion - an internal schema version. Bumping it (when the set of hashed fields changes) forces a global rebuild.
  • projectPath and target - so the same sources under different targets key separately.
  • charm: lines - the active charms, sorted by name. A charm-variant run (lint:rw) hashes differently from the bare run, because the charm changes behavior. Empty charms add nothing, so charm-less runs are unaffected.
  • arg: lines - one per argument after -- (magus run test -- -run TestFoo), in the order given. Unlike charms and env these are never sorted, since -run X is not X -run; a run with different trailing args must not replay another run's result. Empty when no args are forwarded, so an ordinary run hashes unaffected.
  • src: lines - for every file matched by needs, its workspace-relative path, its content SHA-256, and its executable bit. Files are discovered by a single walk, sorted by path, and hashed in parallel. Only the executable bit of the mode is folded in (not the full permissions, which would differ across machines with different umasks), so chmod +x on a script - which changes no content - still invalidates the key.
  • env: lines - each allow-listed environment variable name and its value, sorted, distinguishing unset from set-to-empty. A variable's value contributes to the key only if the spell opted it in.
  • exec: lines - per-op ctx.withEnv/ctx.withCwd execution overrides, sorted. Unlike env: lines, which read a variable's live process value at hash time, an override's value is fixed in the magusfile source itself, so it hashes directly - two runs differing only by a derived override must not share an entry.
  • dep: lines - the resolved cache keys of upstream dependencies, sorted. This is how a change ripples: a dependency's new key becomes an input line here, so a dependent misses transitively.
  • spellDefVersion - a binary fingerprint of the spell definition, so a magus upgrade that changes a spell forces a miss.
  • tool: lines - spell:version strings, sorted, so a toolchain upgrade (a new go or prettier) invalidates the key even when no source changed.

Because the serialization is stable and sorted, the key is reproducible: identical inputs anywhere yield the identical key. A src file's content hash uses an mtime + size fast path (a per-file memo persisted under the cache dir), so an unchanged tree re-keys without re-reading every byte; the memo is a performance cache for the hash, never a substitute for it.

Invalidation: what busts a key

A miss is "no manifest stored under this key." Anything that changes a hashed line above yields a new key, and thus a new (empty) slot:

  • editing, adding, or removing a file matched by needs;
  • toggling the executable bit on a needed file;
  • changing the value of an allow-listed env var (or setting/unsetting it);
  • an upstream dependency's key changing (transitive invalidation);
  • a spell definition change (spellDefVersion) or a tool-version bump;
  • applying or dropping a charm;
  • renaming the project or target.

What does not invalidate: a file's mtime alone (content is what's hashed), a claims-only file, or anything outside the declared needs.

Old keys are never mutated - a miss writes a new entry beside the old one - so invalidation is additive. Reverting a change restores the earlier key and replays its still-present entry. Disk is reclaimed separately by eviction and pruning (see On disk).

Anti-pattern: a shared manifest as an input

The commonest way to wreck a cache is to reach for the file that pins your tools - mise.toml, package.json, go.mod, a lockfile - and declare it an input, usually project-wide.

The intent is right: a tool version really is part of what produced the output, and a bump really should invalidate. The result is not. A manifest pins many tools, and it moves for reasons unrelated to most of them. Wire it into every project and one linter bump rebuilds the entire graph - in CI, the difference between an affected run and a from-scratch build, for a change that could not have altered almost any of it. Do that a few times and people stop trusting the affected set, which is the actual loss: a cache nobody believes is worse than no cache.

Declare what changed, not what contains it. The blast radius should match the tools a project genuinely uses:

  • a tool with a version probe (mgs_getVersionProbe, or mgs_getVersionProbes for a spell driving several binaries) contributes spell:tool:version to the key of every project binding that spell, and nothing to any other. Bumping hadolint moves projects using the docker spell; a Go project's key never notices. This is almost always the right answer for an external binary.
  • a manifest that is genuinely a source of one project - go.mod for a Go project whose build reads it - belongs in that project's sources, where it already is. That is not this anti-pattern: the file really does feed those targets.
  • a pin that reaches one target only wants a per-target declaration, not a project-wide one. magus\inputs in that target's body keeps a sibling target's key still.

If you find yourself adding a manifest to sources to fix a staleness bug, the question to ask first is which tool went stale, and whether it can be probed instead. A probe invalidates the projects that use the tool. A manifest invalidates everyone who happens to live near it.

The opposite failure: tools outside the key entirely

Over-invalidating is loud and annoying. Under-invalidating is quiet and much worse, and it is the more common default: most build caches key on file contents and nothing else, so the tools themselves are invisible.

The failure does not look like a cache bug. A linter upgrades, and suddenly code that passed yesterday fails - or worse, code that should fail passes, because the verdict was replayed from an entry the old linter wrote. A formatter upgrades and a "clean" tree starts failing a drift gate on a file nobody touched. A codegen plugin upgrades and the committed output no longer matches what the generator would emit, but the generate step is a cache hit, so it never runs to notice.

What makes it expensive is that every one of those looks like a bug in your change. You bisect, you re-run, you blame the flaky test, you diff the branch - and the answer was never in the repository at all. Someone's toolchain moved.

magus keys on the tool versions for this reason. Each spell declares how to ask:

export fun mgs_getVersionProbe() > [str] { return ["go", "version"]; }

// A spell driving more than one binary declares each, so all of them move the key.
export fun mgs_getVersionProbes() > {str: [str]} {
    return {"golangci-lint": ["golangci-lint", "--version"]};
}

Their output lands in the key as spell:version and spell:tool:version, so a tool that upgrades invalidates exactly the projects that bind that spell - the precise middle between the two failures. magus describe spells reports which spells probe.

A tool pinned by a manifest the project already reads needs no probe: go.mod is a source of the go spell, so bumping a go tool pin invalidates on its own. Probes are for binaries that live outside the project's declared inputs - a linter from PATH, a formatter from a version manager - which is precisely the set nothing else would catch.

Where the declaration lives is the whole point. Most build systems can express this - Nx, for instance, makes tool-version tracking technically possible through executors, but shifts the burden entirely to developers to implement it consistently. Correctness then depends on each of them remembering, in every project, in every repository. One person skips it and that project silently caches across toolchains, and the failure surfaces somewhere else entirely.

magus does not infer this. Nothing sniffs your PATH or guesses which binaries a target touched - the probe is a declaration someone wrote by hand, and mgs_getVersionProbe is as explicit as it looks. What differs is its location: it sits on the SPELL, the adapter that already knows it drives golangci-lint, rather than being restated by every project that uses one. A project binding spells: [go] inherits that declaration the same way it inherits the spell's sources and ops.

So the trade is not magic against discipline. It is declaring a fact once, where it is true, instead of once per consumer - the same reason a spell declares its needs globs rather than each project re-listing **/*.go.

Set MAGUS_CACHE_TOOL_VERSION=off to drop probes from keys, or =workspace to probe once per workspace instead of per project.

Opting out and busting

Four controls, at four different scopes:

Control Scope Semantics
skip_cache target policy one target, every run Always runs; never replays or snapshots (a long-running fs\watch loop, a service op).
magus run <target> --no-cache one target, one invocation Skips replay for this run only, but still snapshots on success - the entry is refreshed, not left stale, unlike skip_cache.
magus\bust_cache(path?) runtime, one magusfile call Clears manifests (one project, or the whole cache if path is omitted) from inside a target body. An escape hatch that logs a warning every time - the fix is usually to model the missing input as a declared needs source instead.
magus clean --cache CLI, whole cache Wipes the on-disk store from outside any run.
cache.write.enabled (MAGUS_CACHE_WRITE_ENABLED) whole cache, whole run When false, replays hits, but a miss runs the target and does not write a new manifest - locally or to a remote. Restoring still populates the local cache.

skip_cache states that replaying this target would be wrong: it signs a fresh artifact, records a screen capture, mutates go.mod, rewrites a badge, or never returns at all. It is a claim about the target's nature, which is why it lives in the magusfile rather than in the operator's fingers. --no-cache says something entirely different and far weaker: I do not trust the cache for this one run. That is a session-level judgment, so it belongs on the command line.

The two are not interchangeable, and collapsing them breaks in both directions. Move a skip_cache target to --no-cache and correctness now depends on everyone remembering a flag, so a forgotten one replays a cached signature into a release. Reach for skip_cache when you merely wanted a fresh run and the target stops caching forever, for everyone.

skip_cache is not how you handle a target that produces no files. A pure orchestration target - a ci that only composes lint, build, and test - caches correctly with no policy at all: it snapshots an empty manifest and replays as a hit, while its stages keep their own entries. Output globs inherited from the project or a bound spell are allowed to match nothing, and only a glob the target declared itself via ctx.writesFiles must produce a file. If a no-output target ever fails at snapshot time, that is a bug to report, not a reason to opt out of the cache. Opting out instead costs the replay AND is indistinguishable from a real never-replays defect, which is what MGS1009 exists to catch.

Both skip_cache and --no-cache force a genuine re-execution; the mechanical difference is what happens to the cache entry afterward (never snapshot vs. snapshot-and-refresh). bust_cache and clean --cache both delete entries, at different granularities and from different sides of a run. cache.immutable is the odd one out: it does not force anything to re-run, it just stops the cache from ever writing - the common case is a read-only CI runner or a shared cache mirror that must not accumulate local entries.

Granularity: project-wide vs per-target

Without a target declaration, baseStep seeds the cache key with the project sources, every bound spell's claims, and the magusfile. That conservative default keeps an undeclared target safe, but it can make unrelated work invalidate together.

An explicit ctx.readsFiles(...) call changes that contract. It is the target's exact source footprint: magus keeps the magusfiles and that target's spell inputs, then hashes only the declared inputs. A ctx.readsFiles("src/**") build therefore does not re-run for a sibling Dockerfile. Use it when a target has a genuinely narrower domain, and name every source that domain reads.

That gives a clean rule for where to declare a glob:

  • affects every target (a shared schema, a project-wide config) -> project-wide magus\project({sources = [...]}), declared once;
  • affects one target -> ctx.readsFiles(...), ctx.writesFiles(...), or ctx.modifiesExistingFiles(...) in that target's body, according to the file relationship above.

Outputs are almost always target-specific (build -> dist/, test -> coverage/), so a project-wide outputs - which makes every target snapshot it - is usually the wrong tool; prefer ctx.writesFiles(...).

Replay: a hit restores outputs, not execution

On a run, magus computes the key, then looks for a manifest stored under it:

  1. Hit. The manifest is read and its outputs are restored into the workspace. The target's body does not run - the export fun never executes on a hit. Each output is materialized from the content-addressed store by reflink (a copy-on-write clone) where the filesystem supports it, falling back to a byte copy. (Hard-linking is deliberately avoided: it would alias the shared blob and a later in-place rewrite would silently poison the cache.) Symlink outputs are restored as symlinks. Any captured build log recorded on the original run is replayed to stdout, so a cached pass looks like the real one.
  2. Miss. The body runs. On success, magus snapshots the provides outputs: each file's content is hashed, its bytes are stored once in the content-addressed store (deduplicated by hash), and a manifest is written atomically recording every output's path, content hash, mode, and size. A subsequent identical run hits.

This is why the target result is emitted, not returned. A return value can't exist on a hit, since the body never ran - but a hit is exactly what you most want to report. So the dispatcher emits a target.result event ({project, target, status, cache_hit, duration_ms}) for both the ran and the cached case, sourced from the cache's per-run callback. See Results for how the event fits the run hierarchy.

A run that "wins the race" against a cancellation is neither snapshotted nor published: its outputs may be incomplete, so magus surfaces the cancellation instead of recording a poisoned entry.

One owner per generated file

A generated file has exactly one owning target: the one that declares it. Two targets writing the same bytes is the most common way to get a build that never settles, and it is worth being blunt about why, because the instinct it provokes is wrong.

Say generate writes gen/** and a formatter rewrites the same tree. The instinct is to call this a race and fix it with a dependency edge. It is not a race, and ordering cannot fix it:

  • Generate, then format: the formatter's bytes land. The next generate regenerates unformatted output, sees it differ from what is committed, and fails its drift gate.
  • Format, then generate: the formatting is immediately undone, and the formatter's own check fails instead.

Whichever runs last wins, and the loser's gate fails on the next run, at every possible ordering. A dependency edge resolves a producer and a consumer. Two producers of one file is an ownership violation, and there is no order that makes both correct.

If generated output needs formatting, the generator formats it, as the last thing it does. It still owns the final bytes, so its drift gate compares formatted output against formatted output and settles. Generated Go needs no special handling here for exactly this reason: mockery and protoc emit gofmt-clean output already.

Excluding generated trees from a formatter is the weaker fallback, and it is correct when nothing else needs to read that output. Every formatter in this workspace does it, and .markdownlintignore states the reasoning: a lint rule "fixed" in generated output is a fix in the wrong place, because the generator overwrites the edit on its next run. The fix belongs in the generator.

Declaring the same output glob from two targets is MGS1020; the cross-project shape, where two projects claim one glob under the same target, is MGS4002. Neither can see an undeclared write, which is why formatters are excluded by configuration as well as caught by a diagnostic. When a target genuinely needs to amend part of a file it does not own, that is ctx.modifiesExistingFiles, not a second output declaration.

The two roles of an output (maintainer note)

An output glob answers two different questions, and magus keeps them on two different code paths. Confusing them is the easiest way to introduce a stale-hit or a broken magus clean, so the model is worth stating once.

Role Question it answers Scope Where it lives
Cache footprint "what does this target snapshot and replay?" one target cache.Step.Outputs, assembled per-target in buildStep: project-wide Outputs when no target output is declared, otherwise that target's magus\outputs.
Generated-files manifest "what files does this project generate?" whole project types.Project.AllOutputs(): the project-wide Outputs unioned with every target's magus\outputs.

The cache role is per-target on purpose. A miss snapshots exactly the outputs in that target's Step, and a hit replays exactly those - so an output must be declared on the target that produces it. This is the producer-ownership rule, and violating it is a real bug, not a style nit: a glob declared project-wide is in every cacheable target's Step.Outputs, including targets that never write it. When one of those unrelated targets gets a cache hit, its replay restores the file to whatever it was when that target last ran - so a go-build hit can silently revert a freshly regenerated MAGUS.md. Scoping the output to its producer with magus\outputs means only the producer's hit replays it. Project-wide outputs is correct only when every target genuinely produces the glob, which is rare - most outputs belong to one generator.

The generated-files role is the union, because "clean everything this project generates" and "which project owns this path?" don't care which target produced what. A consumer that asks a generated-files question goes through AllOutputs(), never raw p.Outputs, or it silently misses per-target declarations. Today that means magus clean --outputs (CleanOutputs), output-ownership (FindOutputOwner), and the git merge driver (workspaceOutputGlobs). The cache path is the one place that stays per-target.

Inputs have just the one role (the cache key), so there is no AllInputs: a source glob that isn't in a given target's Step.Sources simply doesn't key that target, which is a footprint question, never a "what does the project consume" one.

Should generated output be committed?

Two ecosystems answer this in opposite directions, and each answer follows from its own build model. Go projects commit generated code - *.pb.go, stringer output, mocks - and a clean clone then builds with only the Go toolchain. TypeScript projects regenerate at build time and gitignore the result, since you cannot build without node_modules in the first place, so the committed copy would duplicate something the build already produces.

Both follow the same rule applied to different starting conditions: is the generator already required to build? Ask that first.

flowchart TD
    S[a target generates a file] --> P{pure function of<br/>its committed sources?}
    P -- "no: records the commit,<br/>the clock, or the network" --> R[do not commit it]
    P -- yes --> B{is the generator already<br/>required to build?}
    B -- yes --> R
    B -- no --> C{does anything read it<br/>without running the build?<br/>module zip, IDE, code browser}
    C -- yes --> K[commit it]
    C -- no --> Z{large, or does it churn<br/>on every commit?}
    Z -- yes --> R
    Z -- no --> K
    K --> KG[gate: plain `magus run generate`<br/>fails when the tree changes]
    R --> RG[gate: CI builds it on the<br/>path that publishes it]
Diagram source - renders with JavaScript enabled.

The first question is the one that decides it outright. A file recording its own commit cannot be committed and stay correct, whatever the other answers are - that is the next section. The rest trade cost against reach:

Question Commit it Regenerate it
Is the generator already required to build? No - committing removes a dependency Yes - committing adds churn, removes nothing
Does anything read it without running the build? Yes - IDEs, pkg.go.dev, a downstream module No
Is it a pure function of committed sources? Yes No - see the next section
Is it small and slow-churning? Yes No - large or per-commit churn

Commit it when a consumer cannot regenerate it. A Go module's generated code ships in the module zip. Leave it out and everyone importing your package needs protoc and your buf.gen.yaml to build. Committing moves that dependency from every consumer onto you, which is what the Go convention buys.

Regenerate it when the build already needs the generator. A generated TypeScript client is the usual case: the package manager and the bundler are prerequisites either way, so the committed copy duplicates them. It also churns, since bundled output shifts on a dependency bump.

Regenerate it when it is large or churns per commit. Every clone pays for committed output, CI included, and it pays forever. Untracking the rendered docs site and the console here removed 27% of this repository's blob history. Untracking does not shrink what earlier commits already hold, which is a separate problem that a blobless clone solves.

What each choice costs, and where magus sits

Not committing turns an artifact into a build-order dependency. Something has to regenerate the client before the code importing it compiles, and a repository without a build graph records that ordering in a README or a script. A committed file carries no such edge: it is present before anything runs.

magus lets you declare the edge instead. A generator states what it writes (magus\outputs), a consumer states what it depends on, and the run order comes from those declarations - magus affected reruns codegen when a .proto changes, FindOutputOwner resolves which project owns a generated path, and MGS4004 reports a project reading a path another project wrote without declaring the dependency.

That is a deliberate bias: magus prefers a coupling you write down to one you remember, so it invests in making the "regenerate it" option checkable. Where the ordering is declared, the main argument for committing falls away, and the reasons that remain are about consumers outside the workspace and readers who never run a build.

Either way, gate it. Committed output needs a plain magus run generate that fails when the tree changes, so review catches a forgotten regeneration. Untracked output needs CI to build it on the path that publishes it, or a broken generator ships the last good copy without anyone noticing.

The self-staling output: generated files that record VCS state

There is one combination of ordinary decisions that produces a build which can never be clean. Each half of it reads as sound practice on its own:

  1. A generator records VCS state in its output - a "Last updated" line, the commit that produced a page, a build stamp.
  2. That output is committed, because generated files are usually committed so a reader can see them and CI can drift-gate them.

Each is defensible. Together they cannot converge. Committing the source changes the commit, the commit is an input to the output, so the output you just committed is now stale. Regenerate and commit that, and the new commit stales the output again. Amending does not escape it either: a new hash restales the footer that recorded the previous one.

The only stable resting point is a second commit containing nothing but regenerated output, because a commit that does not touch a page's source does not change the commit that page records. That is why repositories in this state grow a trail of "refresh generated metadata" commits after every real one. Those commits are not sloppiness; they are the fixed point of the loop.

How to recognize it

The tell is a drift gate that passes before you commit and fails immediately after, with a diff containing only timestamps, hashes, or "last updated" lines. If magus run generate is clean, you commit, and magus run generate is suddenly dirty again, you are in this loop.

magus does not diagnose this today. It is genuinely hard to detect without a false positive: after the fix below, the same generator still writes the same commit hash into the same files, and the only thing that changed is whether those files are tracked. Distinguishing the broken state from the fixed one needs a "is this path tracked?" primitive that types.VCSDriver does not currently expose. Until it does, this section is the diagnostic.

The fix: stop committing the output, or stop recording the state

Two ways out, and they are not equally good.

Untrack the output and render at publish time. The generator keeps its provenance line, and the deploy renders from source with the final commit already known, so there is nothing to restale. This is what this repository does: the rendered docs site is generated into docs/gen/ and never committed (.github/workflows/cd.yaml renders it on every push to main). Cost: the output is no longer reviewable in a diff, and a broken generator now blocks a deploy that a file copy could never fail.

Or drop the VCS state from the output. If the provenance line is not worth the cycle, remove it and the output becomes a pure function of its sources, which is what a drift gate wants anyway.

What does not work is keeping both and being disciplined about it. The loop is structural, so "remember to regenerate and commit again" is a rule that has to hold forever, and the failure mode when it lapses is a silent one: the committed output simply describes a commit that is no longer the one it sits in.

The narrower rule this is an instance of

A committed generated file must be a pure function of its committed sources. Anything else in its inputs - the clock, the machine, the branch, the commit - turns "regenerate and diff" from a correctness check into noise. magus's drift gate assumes that purity, which is why the tapes target here is deliberately kept out of the generate umbrella: it screen-records the CLI, so its bytes are never the same twice and a drift gate over it would fail every run by construction.

On disk: just files

The cache lives at .magus/ in the workspace root (override with MAGUS_CACHE_DIR, or cache.dir in magus.yaml). Its layout is three directories plus a hash memo:

.magus/
├── cas/         content-addressed blobs, sharded by the first two hex chars
│   └── ab/ab34...f0      one file per unique output content (deduplicated)
├── manifests/   one JSON manifest per cache entry
│   └── api/<key>.json     project path flattened; file named by cache key
├── logs/        captured build output, replayed on a hit
│   └── api/<key>.log
└── mtimes/      the per-file hash fast-path memo

A manifest is plain JSON you can read directly. It records the project path, the cache key, the target, and one record per output - path, content-address (blob), mode, size, and (for symlinks) the link target:

{
  "projectPath": "api",
  "hash": "ab34...f0",
  "target": "build",
  "outputs": [
    {
      "path": "api/dist/server.js",
      "blob": "9c1f...",
      "mode": 420,
      "size": 20481
    }
  ],
  "createdAt": "2026-07-07T12:00:00Z"
}

That transparency is deliberate: a hit or miss is answerable with ls and cat, and a build log for any entry is a file you can open. magus never mutates an existing manifest, and a manifest read back under the wrong key or project (copied or renamed onto the wrong slot) is treated as a miss rather than trusted.

Space is bounded two ways. An optional size cap (cache.size_mb / MAGUS_CACHE_SIZE_MB) drives LRU eviction of the oldest manifests after a build, and orphaned blobs are garbage-collected once no surviving manifest references them (blobs are shared, so a blob's bytes are only reclaimed when its last referencing manifest is evicted). Out of band, magus config cache prune evicts entries older than a cutoff. To force a clean rebuild of specific projects, magus clean --cache <project> drops their entries. The whole store is portable: magus config cache export / import move it as a gzip-tar.

Connecting to the remote cache

Everything above is local to one machine. A remote cache shares these exact artifacts across CI runners: on a local miss magus asks the remote backend for the artifact keyed by the same (projectPath, hash), and if found imports it into the local store so the ordinary hit path replays it - no rebuild. After a genuine build, magus uploads the artifact so the next machine hits.

The artifact is the same content: the manifest, its blobs, and the build log, packed as a gzip-tar. The key computation, the replay path, and the manifest format are identical - the remote layer only moves those bytes between machines. On top of that it adds a signed trust model: because a replayed artifact injects files into a consumer's build, every remote artifact is verified against an Ed25519 trust set before it is allowed to replay, and an unsigned or untrusted one falls back to a local build. That trust boundary, the backend contract, and CI wiring are covered in full in remote-cache; this page's model is what it builds on.

Glossary

Term Definition
needs A spell's declared input globs. Hashed into the cache key (Step.Sources); also seed the affected set.
provides A spell's declared output globs. Snapshotted on a miss and replayed on a hit (Step.Outputs).
claims Files a spell owns, for affected-set attribution only. Never hashed and never snapshotted.
Cache key The hex SHA-256 of the serialized Step: sources, env, deps, tool versions, spell version, charms, project, and target.
Content-addressed Stored by content hash: identical output bytes are stored once, and a blob's name is its own SHA-256.
Manifest The JSON record of one cache entry: project, key, target, and one record (path, blob, mode, size, symlink) per output.
Blob One unique output content, stored once under cas/, sharded by the first two hex chars of its hash.
Replay Restoring a manifest's outputs on a hit (reflink then copy) without running the target body.
Snapshot Recording a miss's outputs into the store and writing its manifest.
target.result The emitted report event for one target run ({project, target, status, cache_hit, duration_ms}); fires on both hits and misses.
.magus/ The on-disk cache in the workspace root: cas/ + manifests/ + logs/ + the mtime memo.

See also

  • spells: where needs/provides/claims are declared, and what a bound spell contributes.
  • dependencies: how depends_on's dep: propagation and a magus\needs call each interact with this cache key.
  • operations: the run hierarchy and the target.result event that fires on a hit.
  • targets: what a Target is - the unit a cache key is computed and replayed for.
  • charms: the execution modifiers that key into the cache as charm: lines.
  • cache/output-refs: how the key's hex digest becomes a portable reference id, what is deliberately excluded from it, and a known leak that puts a tool's database timestamp in the key.
  • remote-cache: sharing these artifacts across machines under a signed trust model.
cacheneedsprovidesclaimscache-keyinvalidationreplaycontent-addressed
Last updated (a103255f)
Earlier changes on this page (7)

Full history ↗ · Blame source ↗

Glossary

Workspace

The magus root directory that owns a set of projects and shared config; the unit magus operates over. See workspace.

Project

A directory magus recognizes as a unit of work (it has a magusfile); the unit of caching, scheduling, and dependency tracking. See workspace.

Magusfile

The magusfile.buzz that declares a project's targets (as export funs) and binds its spells. See targets.

Target

A named operation (build, test, ...) you invoke with magus run <target>; it may compose a spell's tool-native operations and depend on other targets. See targets.

Operation

A single tool-native command a target composes; the middle of the work hierarchy (Spell to Operation to Target). See operations.

Spell

A language/runtime adapter (e.g. go, md) that maps generic targets onto a toolchain's real commands. See spells.

Charm

An execution modifier attached with : (lint:rw) that changes how a target runs, not which one; the built-in rw flips a check-only target to mutate in place, and ci always strips it. See charms.

Ward

A coded diagnostic that inspects a resolved op and nudges or blocks an anti-pattern before it runs. See wards.

Module

A magus stdlib namespace a magusfile imports for host capabilities: filesystem, exec, vcs, and more. See the module reference.

Buzz

The language magusfiles are written in (the .buzz engine). See engines.

Cache

The content-addressed store magus consults before running a target, so unchanged work is skipped. See cache.

Affected

The set of projects touched by a change; magus affected <target> runs a target only over them. See affected.

Service

A long-running or shared process magus manages across runs, distinct from a one-shot target. See services.

Daemon

The background magus host that owns shared state such as services and the warm knowledge graph. See daemon.

CI

An ordinary magusfile-defined target you compose yourself with magus\needs - magus does not hardcode its stages. Magus.RunCI treats it specially only in that it strips the rw charm, it is the anchor magus affected ci keys off, and a selected scope with no project declaring it is a load error rather than a silent no-op. See targets.

Trace

OpenTelemetry's name for one whole magus invocation; every target it runs is a span beneath it. See telemetry.

Slot

One unit of the pool's capacity. A target acquires the slots it needs to run (most take one) and releases them when it finishes; the pool tracks capacity (total slots), running (acquired), and queued (blocked). See daemon.

Remote cache

A CI-only backend that shares content-addressed artifacts across runners: a cold machine replays a build another runner already did instead of rebuilding. Every remote artifact must be signed by a trusted key. See remote-cache.

Snapshot

A point-in-time view of live state - the pool's occupancy or a tick of exported metrics - as opposed to accumulated history. See daemon.

MAGUS.md

The committed routing index at a workspace root, regenerated from the knowledge graph: it lists every node and points at the exact query for a given question, so it is the entry point an agent reads first. See knowledge.

Ownership

An insight lens: author concentration - the primary author and their share, the distinct-author count (the bus factor), and abandonment. See insight.

Conventions

Placeholders

Angle brackets mark a value you replace with your own - never type the brackets:

magus run <target>
magus completion <shell>    # e.g. bash, zsh, fish

<target>, <path>, <shell>, <name> and the like are stand-ins, not literal text.