Event stream design notes

Why the event stream is shaped the way it is, and what building the first clients changed about it. Recorded 2026-08-27.

The request

Provide a mechanism third parties can build integrations against - an Emacs package, a Vim plugin, a status bar, a notifier - without magus shipping one integration per host.

What the research found

magus does not lack an integration mechanism. It has five partial ones that do not compose, and no integrator can be pointed at a single thing.

An editor plugin can already reach targets (-o json), Buzz completion (magus buzz lsp), file changes (magus watch), and per-target results carrying a fetchable ref. It cannot reach, without speaking protobuf and holding a bearer token: live output from a run it did not spawn, workspace-wide change notifications, or diagnostics.

So the work is not "add hooks". It is "pick one envelope and give it a subscribe verb". Everything underneath is built.

Naming

hook is taken. magus session hook is the agent-host guard adapter: stdin payload, verdict out. The outbound stream is named events, matching the plain register magus already uses for reads (refs, status, watch, query) rather than the thematic register reserved for engine concepts (spell, charm, ward).

The seam test this design has to pass

docs/scope.md seals the engine, the cache, the graph schema, and the guard's evaluation, and states the test for any new extension seam:

it may change what magus does, never what a verdict means.

An outbound stream passes cleanly: it reads a model magus already built, and no subscriber can alter an outcome. Inbound lifecycle callbacks - commands magus invokes mid-run - do not pass, because a post-target callback that touches outputs breaks the cache-replay contract. They are out of scope, deliberately, and this section is the record of that decision rather than a gap to fill later.

Direction: outbound only

magus emits; integrations consume and react. Extending build BEHAVIOR stays where it already is - spells, charms, the magusfile - which are declarations the sealed engine evaluates.

Relationship to magus session hook

They are duals, not the same mechanism, and fusing them would open the seam above.

The unification that is free and correct is at the VOCABULARY, not the mechanism: the guard already writes KindAgentCommand / KindAgentSpawn into the trail, and those become guard.verdict events on the stream. session notify becomes attention.raised. The inbound mechanism keeps its sealed reply channel and additionally broadcasts what it decided, so a status bar can show a denial live without anything being able to influence one.

Transport: one contract, two transports, one front door

                  types.StreamEvent  (one envelope, one taxonomy)
                           |
                     stdout JSONL
                  magus events --follow
                           |
              <cacheDir>/runs/*.jsonl  (the bus)

The transport question answered itself. Every magus run already appends to <cacheDir>/runs/<inv>.jsonl, so a follower polling that DIRECTORY sees runs from any terminal with no daemon, no socket, and no token. The directory IS the bus.

internal/proc was the candidate before that: a JSONL-framed Unix socket in a 0700 directory where the filesystem permissions ARE the authentication, and proc.DiscoverSocket finds a live daemon with no env var. It lacks a subscribe frame - every call in internal/proc/client.go is request/reply - and adding one turns a control plane into a bus. That work was not done and is not needed: it would buy latency over a 250ms poll, and it would make the stream depend on a daemon the design deliberately does not require.

The CLI stays the front door because the daemon is optional by design (docs/scope.md names the daemon dependency as a standing strain) and because a subprocess pipe is available in every editor, while a Unix socket is not. There is no socket frame for this stream and no plan to add one until something needs the latency: an earlier draft of this section promised integrators "a documented frame", which was never built.

The taxonomy

SHIPPED. All four come from the run journal, which is what let the adapter stay a leaf with no dependency on the engine.

target.output is the one high-volume type and is off unless asked for. An editor that subscribes to everything by default drowns.

NOT SHIPPED, and deliberately absent from StreamEventTypes() rather than present and silent. Each has a store; none has an adapter. The cost column is what the review of this branch measured, not an estimate made while designing.

An earlier draft of this table listed a target.started type sourced from KindExec. It was dropped - KindExec is per subprocess, not per target - and the row outlived the decision by several commits. It is recorded here because a taxonomy table that describes intentions as facts is the specific way this document went wrong.

What a first implementation covers

1. types/streamevent.go - the envelope, the taxonomy, the per-type bodies. This is the contract, and it is the deliverable that has to be right.
2. internal/eventstream - the adapter mapping journal records onto StreamEvent, plus the cross-process follower over the run-log directory. Journal is the ONLY producer adapted; the table above says what the others cost. The existing producers keep their on-disk schemas; nothing is rewritten.
3. magus events - replay plus --follow, --type, --limit.
4. internal/proc - the events.subscribe frame and the daemon-side bus. NOT built: the run-log directory turned out to serve as the bus without it, so this is a latency optimization rather than a requirement.
5. A reference client living beside docs/guides/integrations/ the way the OpenCode plugin does - a template the reader owns and edits, per the "the host wiring is yours" entry in docs/doctrine.md. Shipped as POSIX sh; see Reference clients for why, and for what is still held back.

Open questions

• Retention. magus events with no --follow replays recent events; from where? The journal store is per-invocation and already rotated; the trail caps at 10000 and rotates on a schedule. A replay window that spans producers needs one answer, and inventing a sixth durable log to get it would be the wrong one.
• target.started was dropped: journal emits KindExec per subprocess, not per target, so there was no producer to adapt. A consumer that wants per-target progress infers start from the first target.output. Reopen this only if the engine grows a per-target start event for its own reasons.
• Windows. SockDir has a Windows variant but the socket bus needs checking there; the CLI front door is unaffected.

Decision: the daemon is an accelerant, never a capability gate

Recorded 2026-08-27, after the transport question surfaced a wider one.

The complaint that prompted it: some capabilities are daemon-only and some are not, which is hard to support and impossible to document without a fork in every paragraph. The proposed fix was to auto-start the daemon on any command.

That was rejected, and a narrower rule adopted instead:

No capability is daemon-only. The daemon is only ever an accelerant.

Amended 2026-09-02: machine-wide admission is daemon-owned

One capability is now daemon-only, by an explicit decision rather than by drift: a machine-wide budget for concurrency and declared memory. A run routes its admission through the user's daemon, and starts one if none is up. Everything else about the daemon remains an accelerant, and the rule above still governs it.

The invariant could not survive this feature, and the honest thing is to say so rather than to claim a lease directory or a lock file counts. magus tried the directory (bae836df6, deleted 20 hours later): a passive registry can add up what its peers claim, and it cannot tell a waiter that its turn came, so every contended run had to be REFUSED. That is a capability magus could not offer without a process, which is the definition of daemon-only. What the rule protects against - documentation that forks on whether a daemon is running - is bought back by starting one, so no page says "if the daemon is up, X; otherwise Y".

Two things bound it:

• Only a run pays. dispatchProfile.spawnsWork is true for run and affected and nothing else, so magus ls, describe, query, and every other question still spawns no background process. This is the same reading that lets graph export --follow start a daemon: asking for the machine's budget IS asking for the thing that holds it.
• It fails open, loudly. A daemon that will not start, or that dies mid-run, leaves the run unarbitrated and finishing, having said once that it is. A scheduler that can fail a build it was only supposed to order is worse than no scheduler.

A related correction landed with it. An adopted run executes in the daemon's process, where its console output goes to the daemon's log and the caller's terminal shows nothing - measured, not theoretical. That was survivable while nobody started a daemon for an ordinary build, and it is not now that a run starts one. So a TOP-LEVEL run/affected no longer forwards to the stable daemon: it runs here, prints here, and asks the daemon only for admission. A NESTED call still forwards to the socket its parent exported, which is its parent's own process, and prints where the parent does.

Why not global auto-start

• It trades a legible failure for an illegible one. "Daemon off" is visible today and one command fixes it. A daemon that fails to launch, wedges, or is a stale binary serving a newer CLI turns EVERY command into a hang with no obvious cause - the same failure class MGS1021's stale-binary explainer exists for.
• magus ls inside a docker build layer would leave an orphaned background process. A build tool that silently spawns long-lived processes is a surprise.
• docs/scope.md promises the daemon "carries an asterisk". Global auto-start makes it a de facto runtime requirement, which is a doctrine change rather than a feature, and it should be made deliberately if it is made at all.

Why the invariant fixes the documentation

Documentation forks because capabilities fork, not because the daemon is sometimes down. With the invariant, no page ever says "if the daemon is running, X; otherwise Y". It says X, and the daemon makes X faster.

The residual set, and why it is not a violation

/mcp and the console are network surfaces BY DEFINITION - something connects to them over a socket. The shared concurrency pool, machine-wide admission, and background jobs are cross-process by definition. Nobody is surprised that asking for a server needs a server, so these are not a capability split; they are the daemon's own surface.

For those, a command SHOULD auto-start the daemon, because asking for the console IS asking for the daemon and starting it is doing what was asked rather than a side effect.

Amended 2026-09-02. This paragraph used to end "CI never spawns one under this rule - not by a special case, but because CI never asks for a console. That absence of a conditional is the point." That is no longer true, and the honest correction is to say so rather than to add the special case it was praising. magus run and magus affected now start a daemon for machine-wide admission, and CI runs exactly those. A CI-only exemption would be precisely the conditional this passage argued against, and it would also be wrong on the merits: a self-hosted runner executing several jobs at once is the machine the budget is for.

What bounds it instead is a lifetime, not a caller. A daemon started for admission exits on its own after ten minutes with no claims held, no work in flight, and no client asking (watchAdmissionIdle); a daemon a person started with magus server start has no such bound, because they said what they wanted. That keeps the docker build objection answered - a layer that runs one magus command leaves a process that reaps itself, rather than one that lives as long as the machine.

graph export --open --follow does this, via ensureConsoleDaemon. A first implementation was reverted before merge and rebuilt, and the three failures that review found are the specification for anyone touching it again:

• The child must not inherit MAGUS_DAEMON_SOCKET. spawnDetachedDaemon was safe only because its one caller ran under a dispatch profile that never hosts a per-process proc server. Called from one that does, the child decided it was already adopted, bound no socket, and reported its PARENT's - leaving a daemon magus server stop could not find and only kill could remove. daemonChildEnv scrubs it.
• Every failure path must reap what it spawned. With console.enabled=false the wait times out by construction, so each attempt leaked one more process. reapDaemon kills it on both the timeout and the cancellation path.
• It must say which tree it came up in. One socket per user serves every workspace, so a daemon started here is authoritative for whoever connects next. This is the same hazard servingSuffix exists for on the server start path.

--print is exempt: the scriptable "just give me the URL" form must not leave a background process behind.

Still open: promoting one worktree's binary to a long-lived per-user service is what the checkout guard exists to prevent, by a route the guard cannot see because there is no cd in the command line.

This was already the de facto rule

Two sites decided it independently before it was stated:

• internal/doctor/checks_mcp.go degrades the console check when no daemon is running, reasoning that "a check that is red by default is a check people learn to ignore, taking the real failures with it".
• cmd/magus/graph.go refuses --follow with hint.ServerStart rather than starting one, under a comment reading "magus never auto-starts a daemon".

The second is the site this decision REVERSES: --follow is a plain request for the console, so it starts the daemon rather than refusing. The first stays as it is and becomes the worked example of the rule.

The event stream needs none of this

magus events reads the run-log directory, which every magus process already writes to. It has no daemon dependency and no tier split, and this decision does not change it. A daemon-side push would lower latency below the poll interval and is an optimization, not a second implementation.

How the invariant is enforced

A rule that lives only in prose is a rule with roughly even odds (CLAUDE.md says so, and measured it). It needs a test in the shape of TestNoHostSpecificBehaviorInCode: a gate that fails when a command's failure path reports a capability as unavailable because no daemon is running, rather than degrading or starting one.

What the implementation changed about the design

Five things only showed up once real events flowed. Each is recorded because the design as written would have shipped wrong without it.

The run-log directory is the bus. Every magus RUN already appends to <cacheDir>/runs/<inv>.jsonl, so a follower reading that directory sees runs from any terminal with no daemon, no socket, and no token. Only runs: commands that open no invocation write no log and produce no events. Verified end to end: a follower process picked up runs started by a separate process, live. This demotes the proc events.subscribe frame from REQUIRED to a latency optimization, and it is what lets the stream satisfy the daemon invariant above rather than violating it.

The journal buffered, so a naive tail could not fire. journal.FileHandler wrote into a bufio.Writer flushed only at run end, so a follower would have lagged by up to a page and a short run would have delivered nothing until it finished - shipped, green, and unable to work. It now flushes every kind EXCEPT output: lifecycle and result are one per run and one per target, so the syscall is free at that rate, while output stays buffered as the one high-volume kind. That is a hot path changed for a new feature's benefit and deserves review.

jsonv2 does not omit zero numbers. The repo builds with the jsonv2 codec, where omitempty omits only empty JSON values (null, "", [], {}) and NOT 0. Every duration_ms,omitempty therefore shipped "duration_ms":0 on events where the field does not apply - telling a subscriber a cached replay took no time rather than that it never ran. The numeric fields carry omitzero, and a test pins the distinction. Any new numeric field on this contract has the same trap.

"Is a daemon running" cannot be answered by looking at a socket. A magus invocation hosts its own proc server on the stable socket AND exports MAGUS_DAEMON_SOCKET into its own environment so children inherit adoption. Both of the obvious checks therefore report a process as being served by itself, and the warm-graph hint never fired. The question is whether the PID on the other end is somebody else. Three implementations were wrong before the fourth worked; the comment on servedByAnotherDaemon records why.

A relative --root shipped a relative workspace. The contract promises an absolute root because a subscriber watching two workspaces routes on it, and a relative path resolves against the SUBSCRIBER's cwd rather than the producer's. magus events absolutizes.

--limit 0 means replay NOTHING

Found by running the shell watcher: it announced the previous day's failure the moment it started. --limit originally read 0 as "replay everything", so there was no way to ask for "only what happens next" - Follower.Skip existed in the library and the CLI could not reach it.

The semantics are now the ones a subscriber actually needs:

A notifier wants 0; a statusline wants 1. The library's Replay still reads 0 as "no cap", and the CLI maps - changing the method would have altered what it means for every other caller to fix a flag's ergonomics.

Reference clients: shell first

The first draft led with an editor package. That was the wrong shape, and using the thing is what showed it.

The repo's existing reference integrations (docs/guides/integrations/agents/*.sh) are POSIX sh for a reason doctrine states: the contract should be small enough to hold in your head. An editor package buries the ten lines that teach the contract under a couple of hundred lines of mode, hook, and buffer plumbing. magus-events-watch.sh is the whole contract, and an editor client is visibly that loop plus a way to draw on a screen.

Editor clients for Vim and Emacs are drafted but deliberately not shipped yet: they are the part of this work that most needs a human to read it, and an unreviewed plugin in the docs tree is a promise magus has not checked.

Two findings from writing them are worth keeping even so:

• Vimscript reaches more editors than Lua and is shorter here. Nothing in this contract needs Neovim - it is a subprocess pipe, and Vim 8 has had job_start() and json_decode() for years. The deciding detail runs opposite to the usual assumption: Vim's out_mode: 'nl' delivers exactly one complete line per callback, so the Vim path needs no line buffering, while Neovim's on_stdout hands over a list whose last element may be partial.
• VS Code is the widest audience and the worst reference. Its extension scaffolding (package.json, tsconfig, a bundler) dwarfs the fifteen lines that matter, and readline.createInterface hides the partial-line problem rather than teaching it. docs/scope.md already names the OpenCode plugin's upkeep as more than an example should need.