magus v0.4.3 is out. See what's new
¶ View markdown source · ✎ Suggest an edit
20 min read

Git integration

magus writes three things into a repository, and no more:

  • a managed section in .gitattributes, marking every declared output as generated and routing it, and every file auto-resolution may settle, to magus's merge driver;
  • a merge.magus.driver registration in the clone's own git config, because a driver cannot be committed;
  • the refresh, drift-notice and owed-regeneration hooks below, when the server starts.

All three are managed sections or single config keys. Nothing rewrites your history, your branches, or a hook body you wrote yourself. Hooks your workspace writes in Buzz are a fourth thing, installed only when you ask; see Your own hooks, in Buzz.

Hooks hand off work, never do work

This is the rule, and it is not a performance target to aim at later. It decides whether a hook may exist at all.

A git hook runs on the critical path of a command the user did not ask magus to be part of. Someone typed git checkout. They are waiting on git, not on a build tool, and every millisecond a hook spends is stolen from an operation that had nothing to do with us. So the bar is not "fast enough". The bar is:

A magus hook may look up something already computed, or hand the work to something else and return. It may not do the work.

That is checkable by reading the hook, not by timing it. A hook that computes is disqualified even when it happens to be quick today, because the thing it calls will get slower and nobody will notice until it does.

Automatically disqualified, whatever the measured time:

  • starting a language runtime to answer the question (a cold node, python, or deno process has lost before it has parsed anything);
  • loading the workspace, evaluating magusfiles, or opening the knowledge graph;
  • touching the network, including a cache the hook thinks is nearby;
  • running a build, a test, a linter, a formatter, or a generator;
  • enforcing policy locally - blocking a commit, rewriting a file, staging something on your behalf.

That last one is worth stating plainly, because it is the one every other tool reaches for. magus does not enforce in a hook. Enforcement belongs where it can be read, re-run, and argued with: a target, a gate, or a CI check. A hook that blocks is a hook people disable with --no-verify, and a disabled hook enforces nothing at all.

Every magus hook also fails open. The installed body ends in || true and sends its output to /dev/null, so a hook can never fail your git command, and a broken or half-installed magus is invisible to it. A hook that is not certain it is correct does nothing and says nothing.

The hooks magus installs

magus server start installs them, best-effort: a repository with no server never gets them, and a failure to write one is a warning, never a reason the server does not start.

Hook Fires on
post-checkout a branch switch (guarded so a file checkout does not trigger it)
post-merge a merge or pull
post-rewrite a rebase or amend

Each is a history change that can stale the knowledge graph and the symbol index. Mercurial gets the same treatment through its update hook; jj has no hook support and gets none.

The body is one line:

magus server job sync-graph >/dev/null 2>&1 || true

That command is the whole point of the rule above. It does not reconcile anything. It enqueues a job on the already-running server and returns, so the reconciliation happens in the background, after your git checkout has finished, on a process that was going to be running anyway. The hook's own cost is one short-lived client that posts a message.

They are written into a managed section:

# BEGIN magus-refresh - do not edit this section manually
magus server job sync-graph >/dev/null 2>&1 || true
# END magus-refresh

Your own hook body, above or below that section, is preserved across every reinstall. Delete the section to remove the integration; the next magus server start puts it back.

The drift notice

post-commit and pre-push carry a second managed section, magus-drift, in the same shape: post a check-drift job, return. The server compares the commit's changed sources against the outputs they produce and runs gofmt -l over its changed, format-governed files, then prints the fix and the command that folds it into the offending commit (MGS4006 stale output, MGS4009 stale formatting).

It never blocks and never writes to your tree. CI is the check; this is the earlier warning. A hook only runs where someone installed it and did not pass --no-verify, and jj has no hooks at all, so hooks catch three backends on a good day and CI catches everything. See the gate is a commit hook, so it is not a gate.

The difference is the feedback loop: CI rejects the same commit ten minutes later on a pull request you then have to push again; the notice catches it while it is still the commit in front of you.

Your own hooks, in Buzz

The rule above binds the hooks magus installs for itself. A hook your workspace writes is your policy, and whether it blocks is your call. What magus adds is a way to keep it under version control, written in the same language as your magusfile, instead of a script in .git/hooks that nobody reviews and no clone receives.

spells/git/hooks.buzz in the magus repository is a workspace-local Buzz module; copy it into your workspace's spells/git/. It takes a directory of <hook>.buzz files, one per git hook, and installs a short sh shim for each into the directory git runs hooks from. That is core.hooksPath when it is set, and otherwise the common directory's hooks/, which linked worktrees share. The shim runs the checkout's own copy with magus buzz -s <file> -- <git's arguments>, so each worktree runs its own version of the hook, and a non-zero exit from main blocks the git operation.

Wire it to two targets:

import "spells/git/hooks" as githooks;

export fun git_hooks_install(ctx: magus\Context, args: [str]) > void !> any {
    githooks\install(ctx, dir: "tools/git-hooks");
}

export fun git_hooks_remove(ctx: magus\Context, args: [str]) > void !> any {
    githooks\remove(ctx, dir: "tools/git-hooks");
}

Then write the hook, a Buzz file whose main takes git's arguments:

import "fs";
import "os";

fun main(args: [str]) > void !> any {
    if (fs\readFile(args[0]).trim() == "") { os\exit(1); }
}

Save it as tools/git-hooks/commit-msg.buzz and run magus run git-hooks-install:rw .. Adding another hook later is the same two steps: the file, then the target.

Both targets follow the rw charm: without it they report what they would change and fail, and with it they change it. --dry-run writes nothing. The rules they enforce:

  • A file named for no git hook is an error, not a hook git would silently never run.
  • An existing hook install did not write is an error. Delete it, or call your Buzz file from it.
  • remove deletes only shims install wrote, found by the marker on their second line, including a shim whose Buzz file you have since deleted.
  • The shim exits 0 with a notice when magus is not on PATH, and silently when the checked-out branch has no copy of the Buzz file.
  • The shim sets BUZZ_INCLUDE_PATH to the project directory, so a hook imports workspace Buzz by its project-relative path.

The magus repository's own commit-msg.buzz is the worked example: it refuses a commit made directly on the base branch whose subject is not a conventional commit, by the rule CI applies to pull request titles.

Two limits worth knowing. A hook runs only where someone installed it and did not pass --no-verify, so anything that must hold belongs in CI as well. And magus server start writes managed sections into post-checkout, post-merge, post-rewrite, post-commit and pre-push; install refuses those files as hooks it did not write, so a Buzz hook for one of those names needs the server's section removed first.

The merge driver, and what it cannot do

Generated files conflict constantly and merge meaninglessly: the correct merge of build output is whatever regenerating produces, not a reconciliation of two byte streams. So each declared output glob gets a .gitattributes entry:

MAGUS.md merge=magus linguist-generated

merge=magus routes conflicts to magus vcs merge-driver, which git invokes per file. You never run that command yourself. linguist-generated is the other half, and it is the half that always works: it collapses the file in GitHub's diff view and keeps it out of language statistics.

Diff drivers

The same managed block opens with a diff driver for each source language magus reads:

*.go diff=golang
*.ts diff=typescript
*.buzz diff=buzz

A diff driver's hunk-header pattern is how git names the declaration a change lands in, the text after the second @@ of each hunk. A job's footprint reads those names to say which functions, types and tests a change touched, and git diff prints them for people too. golang, python, rust and markdown ship with git. typescript and buzz do not, so magus registers their patterns as diff.typescript.xfuncname and diff.buzz.xfuncname in git config, beside merge.magus.driver and in the same scope. Each attribute resolves to the last line that sets it, and a driver line sets only diff, so a generated .go file keeps merge=magus and gains diff=golang. A workspace load restores a missing line or registration, and magus doctor fails its merge-driver-loads-workspace check when one is still missing.

Auto-resolving source files

A source file conflicts for real more often than not, but some conflict in ways nobody has to think about: two branches each appending an entry to CHANGELOG.md. The driver settles such a file when two things hold, the same two the merge queue checks:

  • the merge settles. Each side's edits are hunks placed by the file's diff driver, and a region both sides changed settles only when both made the same change, one side's change holds the other's, or both only added lines where the base had none (ours first, then theirs).
  • the change is low risk by magus's one change classifier, the ci gate's: generated, prose (gate_low_risk, markdown by default) or comment-only. Code qualifies only where its project lists it in merge_low_risk.

Anything else leaves the whole file conflicted, with conflict markers the driver writes itself (git keeps whatever a failed driver left in the file, so without them the other side's change would vanish), and a message naming each region that did not settle as path#declaration, or the classifier's line. A file both sides added, a binary file and a symlink are never settled.

Every declared gate_low_risk and merge_low_risk glob gets its own line in the managed section, merge=magus without linguist-generated, since the file is source and review has to show it. A glob with no slash is written anchored, because git would otherwise match it at any depth. The built-in markdown defaults are not written, so a workspace that declares neither key keeps the .gitattributes it had: declare "gate_low_risk": ["**/*.md"] to route markdown to the driver. A comment-only edit has no glob either, so git and hg reach one only through the queue or magus vcs resolve.

Each VCS reaches the driver its own way, and the decision is the driver's in all of them:

VCS Registration When it runs
git .gitattributes and merge.magus.driver during git merge, rebase and cherry-pick
Mercurial, Sapling [merge-patterns] and [merge-tools] in the config during hg merge and sl merges
Jujutsu merge-tools.magus in the repository's config magus vcs resolve, or jj resolve --tool magus <files> by hand

jj records a conflict in the commit and never runs a tool on its own, so magus vcs resolve runs jj resolve --tool magus over each conflicted source file before it settles the rest, and the driver decides each, comment-only code included. The registration (jj config set --repo) leaves ui.merge-editor alone, and tells jj that exit status 1 means the file still conflicts, so jj keeps its own markers for a file the driver does not settle.

Regeneration after the merge

The driver keeps the current version of the file and does not regenerate it. git calls it once per conflicted file while the merge is still writing the tree, so a generator started there would read a half-merged checkout. Instead the driver records what it owes: the project, the target that rebuilds the file, and the file itself, in magus-owed-regeneration.json in the worktree's git directory. Fifty kept files of one target are one entry.

post-merge, post-rewrite and post-commit carry a third managed section, magus-regenerate-owed, in the same shape as the others: post a regenerate-owed job, return. post-commit is there because a merge git stopped on is concluded by git commit, which never fires post-merge. On the server the job:

  1. returns at once when nothing is owed, which is every ordinary commit;
  2. waits up to 30 seconds for the merge or rebase to let go of the tree, because post-rewrite fires before a rebase removes its state and post-commit fires on every pick, and otherwise leaves the record for the hook that fires when it finishes;
  3. runs each owed target once with :rw, deepest projects first, so docs regenerates before the root that indexes its pages;
  4. stages the declared outputs of the rebuilt projects that changed, and clears the record.

It prints one line naming the runs, the file count and how to finish. It never amends: the job runs after git has returned, while you may be typing the next command, so it stages the result and prints git commit --amend --no-edit when HEAD is an unpushed commit, or asks for a new commit when HEAD may already be published.

A failed regeneration leaves the record in place. magus doctor reports a non-empty record under owed-regeneration with the commands that settle it; run magus job run regenerate-owed to retry, or magus server regenerate-owed in a clone with no server, where no hook is installed. Only git records owed regenerations. Under Mercurial and Sapling the driver logs the command to run, as before.

A forge never runs a custom merge driver. merge=magus needs merge.magus.driver in a git config, which is per-clone and cannot be committed, so github.com computes mergeability with the plain three-way merge and reports conflicts your local git would have settled silently. The same is true of GitLab, Gerrit, and every merge queue. This is architectural, not a configuration gap, and no .gitattributes change will fix the conflict banner on a pull request.

A driver has a second limit: no VCS invokes a content merge driver when one side deleted the file, so a modify/delete conflict reaches you untouched no matter how it is configured.

Both are why the real answer is a bulk command rather than a per-file callback:

git merge origin/main
magus vcs resolve

magus vcs resolve classifies every conflicted path at once, regenerates once instead of once per file, settles the deletions a driver is never called for, and stages everything the regeneration touched so git rebase --continue does not refuse on a dirty tree. Conflicts in files magus does not generate are reported and left for you, after the driver has had its turn at them.

To settle a branch against its base without merging first, hand it the ref:

magus vcs resolve --against origin/main

That merges, resolves, and leaves the merge in progress for you to commit. It needs a clean tree, so backing the merge out cannot lose uncommitted work. Add --dry-run to see the classification and have the merge backed out again.

magus vcs resolve works on git, Mercurial and Jujutsu - all three implement conflict reporting. Only --against is git-only, because only git has a merge-starting implementation; on the others, merge the base in yourself and then run magus vcs resolve.

Merge, do not rebase

Reach for git merge origin/main, not git rebase origin/main. This is not a style preference. The rebase costs you three things the merge does not:

  • A force-push. A rebase rewrites your commits, so the branch can only move with --force-with-lease. A merge only adds a commit, so a plain git push works.
  • Everything stacked on you. A force-push moves the commits every dependent branch was built on, so each branch above yours needs the same treatment, in order, every time the base moves. A merge leaves them all valid.
  • The conflict, repeatedly. A rebase replays each of your commits against the new base, so a generated file can conflict once per commit. A merge settles it once.

The usual reason to prefer a rebase is a linear history on the trunk, and a squash merge already gives you that: it collapses the branch to a single commit, so merge commits inside your branch never reach the trunk at all. If your project squash-merges, merging the base into your own branch is free.

Stacked pull requests

A stack is a chain of branches, each based on the one below it. Generated files are the usual reason one falls apart, and the mechanism is worth knowing.

When the bottom pull request is squash-merged, the trunk gains a new commit that is not an ancestor of the branch above, while that branch still carries its own copy of the commits that were just squashed. For hand-written files git copes: both sides made the same change, and a three-way merge recognizes that. For generated output it cannot. The trunk's output was regenerated from the bottom branch's sources alone, and the branch above holds output regenerated from both, so both sides changed the same region differently. That is a real conflict, in a file neither author edited.

Settle a stack from the bottom up, one branch at a time, each against its own base:

git switch feature-b          # the branch directly above what just merged
git merge origin/main
magus vcs resolve
git push

git switch feature-c          # then the next one up, against ITS base
git merge origin/feature-b
magus vcs resolve
git push

Every step is a merge, so nothing is force-pushed and settling one layer does not invalidate the layers above it. The same walk with rebases means redoing every branch above whichever one you touch.

Whose conflict is it

magus vcs resolve reports the paths it deliberately left alone. To answer the same question before you start a merge, ask the workspace - magus describe file reads the declarations rather than guessing from a path convention:

magus describe file MAGUS.md internal/describe/extract.go

role: output is generated: regenerate it, never hand-merge it, and do not bother reading its diff. role: source is yours, and an ordinary conflict.

When the driver is doing nothing

The driver's failure mode is quiet, and it looks like the opposite of a failure - git reports a conflict in a generated file it should have settled by itself. A file that both sides changed to the same bytes is the clearest tell, because that is the easiest merge there is.

The registration is a command line, held in one clone's config:

git config --get merge.magus.driver

It names a magus binary and a subcommand. If that command cannot run - the binary moved, or it predates the subcommand the registration names - the driver exits non-zero, and git reads a non-zero driver as conflict. Every file routed to it is then reported as conflicted whether it is or not, which inflates the count rather than raising an error.

magus rewrites the registration whenever it opens a workspace, so running a command that loads one repairs it; magus ls is enough. Two things stop that happening on their own: a magus that cannot load this workspace never reaches the refresh, and the vcs verbs skip it on purpose, because the refresh writes the tracked .gitattributes and those verbs run while that file may itself be unmerged.

magus vcs resolve does not depend on the driver - it reads the conflicted paths from git and regenerates - so it settles the inflated list too. A broken driver makes a conflict look worse than it is; it does not stop you fixing it.

Staging

magus vcs add stages what the workspace declares: sources, and the generated outputs a source change in the same commit accounts for. Anything undeclared is reported rather than swept in, which is the difference between it and git add -A.

It also skips generated output that nothing in the change accounts for - output that moved with no declared input behind it, which means either a different magus build produced it or a generator is not deterministic. Name such a path explicitly to stage it anyway.

Reading a change through git

magus diff reads a changeset the way this repository's conventions rank it: declared outputs folded away, the rest ordered by what they can break. Nothing about that requires leaving git. Reviewing your changes covers what it reports; this section is only how to reach it without typing a magus command.

Wire it as git's pager for diff, and plain git diff renders through magus:

git config pager.diff 'magus diff -'

That is the whole integration. git diff, git diff <ref>, and git diff --staged all work, because git hands its pager the entire patch on stdin and magus diff - reads a patch on stdin. Nothing is intercepted that you cannot get back: git --no-pager diff prints the raw patch, and git -c pager.diff=cat diff does the same for one invocation.

Prefer to opt in per command rather than always? An alias costs one line and leaves git diff alone:

git config alias.reading '!f(){ git diff "$@" | magus diff -; }; f'

Then git reading and git reading main read through magus, and git diff does not.

Why not an external diff or a difftool

GIT_EXTERNAL_DIFF and diff.external are for a program that renders ONE file's diff: git calls them once per file, with seven arguments. Almost everything magus has to say is a property of the whole changeset - which projects rebuild, who owns them, what it costs, what to read first - so per-file invocation would mean printing the report once per file, or not at all. Wire magus there and it refuses, naming the pager setting above; it does not half-answer.

git difftool --dir-diff has the opposite problem. It copies the changeset into two temporary directories and runs the tool against those. magus refuses to run inside a temporary copy of a tree on purpose: the verdict would describe a workspace nobody ships, and anything regenerated would land in the copy. Per-file difftool is the external-diff mismatch again, with a prompt between each file.

The pager is the one git integration point whose contract already matches: the whole diff, once, on stdin.

The same shape in the other backends

This is not a git quirk. Every backend magus supports offers a diff-tool slot and a pager, and in each of them the tool slot is the wrong shape for the same reason - measured against the installed versions:

backend diff-tool slot hands the tool pager hands the tool
git seven arguments, once per file (GIT_EXTERNAL_DIFF), or two temp directories (difftool --dir-diff) the whole unified diff on stdin
Mercurial two directories - a temp snapshot and the working dir (extdiff) the whole unified diff on stdin
Sapling two file paths, once per file (extdiff) the whole unified diff on stdin
Jujutsu two directories, $left and $right (ui.diff-formatter, or file-by-file with diff-invocation-mode) whatever ui.diff-formatter produced

So the pager is the portable answer, and two settings make it work everywhere:

hg config --edit    # [pager] pager = magus diff -   and   [color] mode = off
sl config --user pager.pager 'magus diff -'
jj config set --repo ui.diff-formatter ':git'
jj config set --repo ui.pager '["magus", "diff", "-"]'

Jujutsu needs the extra line because its default diff is a side-by-side rendering rather than a patch; :git makes jj diff emit the unified form its pager then hands over. With that set, jj behaves like the rest.

Turn color off for the diff being handed over. A VCS colorizes when it believes it is writing to a terminal, and paging is exactly that case. A colorized patch has escape sequences in front of every header, so the headers no longer begin a line and nothing parses. magus refuses such a patch and names this as the cause rather than reporting an empty changeset, but the fix is upstream: --color=never, hg --config color.mode=off, or jj --config ui.color=never. git does not colorize into a pager by default and needs nothing.

Every temp-directory variant above is refused for one reason: magus declines to run against a copy of a tree, because the verdict would describe a workspace nobody ships and anything regenerated would land in the copy.

Reading a patch you did not produce

The same input works from anywhere a patch comes from - a colleague, a mail attachment, a stash, a code-review tool:

gh pr diff 123 | magus diff -

Both patch dialects parse: git's diff --git a/x b/x headers, and the bare --- a/x / +++ b/x pair that GNU diff -u and patch speak. A patch magus cannot read is refused rather than reported as an empty changeset.

gitvcshooksgitattributesmerge-drivergenerated-filesconflicts
Last updated (a9ff8609)
Earlier changes on this page (7)

Full history ↗ · Blame source ↗

Glossary

Workspace

The magus root directory that owns a set of projects and shared config; the unit magus operates over. See workspace.

Project

A directory magus recognizes as a unit of work (it has a magusfile); the unit of caching, scheduling, and dependency tracking. See workspace.

Magusfile

The magusfile.buzz that declares a project's targets (as export funs) and binds its spells. See targets.

Target

A named operation (build, test, ...) you invoke with magus run <target>; it may compose a spell's tool-native operations and depend on other targets. See targets.

Op

A single tool-native command a target composes (long form: operation); the middle of the work hierarchy (Spell to Op to Target). See operations.

Spell

A language/runtime adapter (e.g. go, md) that maps generic targets onto a toolchain's real commands. See spells.

Charm

An execution modifier attached with : (lint:rw) that changes how a target runs, not which one; the built-in rw flips a check-only target to mutate in place, and ci always strips it. See charms.

Module

A magus stdlib namespace a magusfile imports for host capabilities: filesystem, exec, vcs, and more. See the module reference.

Buzz

The language magusfiles are written in (the .buzz engine). See engines.

Cache

The content-addressed store magus consults before running a target, so unchanged work is skipped. See cache.

Server

The background process a person starts with magus server start. It serves MCP, the console, background jobs and the warm knowledge graph, and adopts nested magus calls into one pool. See server.

CI

An ordinary magusfile-defined target you compose yourself with magus\needs - magus does not hardcode its stages. Magus.RunCI treats it specially only in that it strips the rw charm, it is the anchor magus affected ci keys off, and a selected scope with no project declaring it is a load error rather than a silent no-op. See targets.

Slot

One unit of the pool's capacity. A target acquires the slots it needs to run (most take one) and releases them when it finishes; the pool tracks capacity (total slots), running (acquired), and queued (blocked). See server.

Snapshot

A point-in-time view of live state - the pool's occupancy or a tick of exported metrics - as opposed to accumulated history. See server.

Knowledge graph

The queryable graph of a workspace's spells, targets, docs, and code relationships; query it with magus query/explain/path. See knowledge.

MAGUS.md

The committed routing index at a workspace root, regenerated from the knowledge graph: it lists every node and points at the exact query for a given question, so it is the entry point an agent reads first. See knowledge.

Invocation

One magus process's recorded facts - the targets it finished, their outcomes, the lease it acted as, and the session it ran in when a host delivered one - kept in a repo-scoped store every worktree shares. magus session lists them; the store prunes itself by last-fact age.

Job

The unit of delegated work, and one row of the job store: what an orchestrating agent handed out, with its goal, the checkpoint it was cut against, the paths it may write or must not touch, and the one check it runs. A job's holder is either a session, for work an orchestrator handed out, or the server, for its own maintenance. The store records; the agent guard is what reads those facts back when grading a write. See doctrine.

A job is not a run. magus run build web is a run, and no job exists for it. A job causes runs: its check executes as one, and a server job records the invocation of its last one. Jobs are listed with magus ls jobs and in the console's Jobs view; runs are listed in the Runs view.

Run

One target executing under one magus invocation, such as magus run test web or magus affected ci. A run keeps its captured output behind an output reference. Every magus run is a run whether or not any job asked for it; see Job for how the two relate.

Lease

The grant a holder takes on a job: the write and read paths that job declared, enforced in the checkout that took it with magus job exec. A job is the piece of work; a lease is permission over it.

Conventions

Placeholders

Angle brackets mark a value you replace with your own - never type the brackets:

magus run <target>
magus completion <shell>    # e.g. bash, zsh, fish

<target>, <path>, <shell>, <name> and the like are stand-ins, not literal text.