magus v0.4.3 is out. See what's new
¶ View markdown source · ✎ Suggest an edit
2 min read

How gopherbuzz runs Buzz

Every Buzz program magus runs, a magusfile, a spell or a magus buzz script, goes through gopherbuzz, a pure-Go Buzz implementation. Engines covers how magus hosts it; this page follows one program through it.

Source to chunk

{"figure":"buzz-compile","stamp":"sha256:f7339b57580cf05113286c808cbbccb469664ee25d380b0a581fcd9b3fb4d4c1"}From Buzz source to a chunkSession.Exec compiles source in six steps, all in the gopherbuzz root package except the first and the last. The lexer, token.Tokenize, turns the source into tokens; a session given a parse cache keeps them by source text, so a module many sessions import is lexed once. The parser builds an ast.Program. Before the checker sees it, import resolution runs every imported file through this same pipeline and the VM, so their exports are bound when the checker looks names up. The checker reports types.Type mismatches as BZZ diagnostics, and the compiler emits a vm.Chunk, one per function. In the vm package, FoldConsts folds constants and FusePeephole fuses the common local, constant and branch sequences into superinstructions. A session with a bytecode store saves the chunk with every file it was compiled against and, on a later run where all of them still hold the same bytes, hands the saved chunk to the VM without parsing anything.GOPHERBUZZFrom Buzz source to a chunkON A MISS[]TOKEN*AST.PROGRAMEXPORTS BOUNDNO ERRORSEMITSFRESH HITLexertoken.TokenizeParse cachetokens keyed by sourceParserrecursive descent, *ast.ProgramImport resolutionruns each module firstCheckertypes.Type, BZZ codesCompilerAST to *vm.ChunkBytecode store.bo plus its import closureChunkFoldConsts, FusePeepholeLEGENDWHERE AN IMPORT RUNSOPTIONAL, SET BY THE HOST

Session.Exec drives the front end. Every step except the lexer and the chunk passes lives in the root package, package buzz.

Step Where Hands on
Lexer token.Tokenize []token.Token
Parse cache ParseCache (optional) the same tokens, keyed by source text
Parser parseModedTracked *ast.Program
Import resolution Session.loadFileImports exports bound in the session's vm.Env
Checker checkWithPrelude types.Type errors as BZZ diagnostics
Compiler CompileWith *vm.Chunk, one per function
Chunk passes vm.FoldConsts, FusePeephole the same chunk, with superinstructions
Bytecode store Session.SetBytecodeStore a saved chunk, when its files are unchanged

Import resolution runs before the checker and does more than read files: each imported module goes through this same pipeline and runs on the VM, so its exports exist when the importer is checked. A host module arrives instead as a native vm.Value with declaration source for the checker.

The bytecode store is the one superset of upstream Buzz here: it saves a chunk's .bo encoding with every file the chunk was compiled against, including the files its imports read, and replays it while all of them still hold the same bytes. magus uses it for the guard's policy files, which a hook loads in a fresh process on every command.

Chunk to result

{"figure":"buzz-run","stamp":"sha256:c15355510802d2d3fc749c369c2d05a46a4da1a7e0847650e3682daaac890f8e"}Interpreter first, native code when the chunk allows itSession.runChunk makes a VM and calls VM.Run with the chunk and the session's environment. Run reserves the top-level register window and asks jitRun first. With the JIT on, the baseline JIT judges the chunk once and caches the verdict: depths() admits only the numeric subset, locals, constants, arithmetic, comparisons and jumps, and validates every operand, and compileJIT emits amd64 or arm64 code through golang-asm into executable pages. Only the chunk Run was handed can go native; a function it calls runs on the interpreter. There is one native tier and no hotness counter. A value the native code cannot handle deopts to the interpreter at the instruction it stopped on, with the stack depth checked; a chunk the JIT declines, or one with no slot locals, runs on the interpreter from the start. The interpreter's values are NaN-boxed 64-bit words, and heap objects are handles into one table the session releases when it closes.GOPHERBUZZInterpreter first, native code when the chunk allows itPACKAGE VM*CHUNK, *ENVJIT ONJIT OFFCOMPILEDDECLINEDDEOPT AT IPALLOCATESSessionrunChunkVM.Runregister window, inline cachesBaseline JITdepths, then golang-asmNative codeint and float fast pathsInterpreterVM.Exec, one dispatch switchHeap tableNaN-box handles, freed on CloseLEGENDTHE NATIVE TIER

gopherbuzz has two tiers, an interpreter and a baseline JIT, and the unit both run is the chunk. There are no traces and no hotness counter: VM.Run offers the chunk it was handed to the JIT on every run, and the JIT judges each chunk once.

  • Interpreter. VM.Exec is one dispatch switch over word-coded instructions. A value is a NaN-boxed 64-bit word, so the operand stack is a []uint64 the Go collector never scans; a heap object is a handle into one table, and the session's vm.Owner releases its handles when the session closes. Member and field access go through per-VM inline caches.
  • Baseline JIT. On amd64 and arm64, depths() admits a chunk made only of locals, numeric constants, arithmetic, comparisons and jumps, and validates every operand; compileJIT then assembles it with golang-asm into executable pages cached against the chunk. Only the chunk Run was handed can go native, so a function it calls always runs interpreted. A mixed int and float operand is promoted in native code; a non-number arriving through any, a divide by zero, a NaN result or a float % deopts to the interpreter at the instruction it stopped on. BUZZ_JIT=0 turns the tier off.

The gopherbuzz README has the performance design, the platforms the JIT has run on, and where gopherbuzz differs from upstream Buzz.

ConceptsPages in this section. SpellsA spell is a library of tool-native ops (go-build, cargo-clippy, eslint) plus cache metadata that your magusfile composes into runnable targets. Remote cachingWire an S3 or GitHub Actions remote cache provider as a spell so CI runners replay signed build artifacts across machines with a strict trust model. Knowledge graphThe knowledge graph is a deterministic, cache-backed graph of the magus domain (projects, targets, spells, ops, charms, modules, diagnostics, docs, and buzz sources) that agents and humans query instead of grepping. This page covers the schema, the verbs, the file layout, and how to point external graph tools at an export.
buzzgopherbuzzvmjitcompilerlexerbytecodeinterpreter
Last updated (aba0c4fa)
Glossary

Magusfile

The magusfile.buzz that declares a project's targets (as export funs) and binds its spells. See targets.

Op

A single tool-native command a target composes (long form: operation); the middle of the work hierarchy (Spell to Op to Target). See operations.

Spell

A language/runtime adapter (e.g. go, md) that maps generic targets onto a toolchain's real commands. See spells.

Module

A magus stdlib namespace a magusfile imports for host capabilities: filesystem, exec, vcs, and more. See the module reference.

Buzz

The language magusfiles are written in (the .buzz engine). See engines.

Engine

The interpreter a magusfile runs on; magus embeds the Buzz engine. See engines.

Cache

The content-addressed store magus consults before running a target, so unchanged work is skipped. See cache.

Trace

OpenTelemetry's name for one whole magus invocation; every target it runs is a span beneath it. See telemetry.

Session

An agent host's conversation, by the id the host delivers to its hooks. magus never mints one: a record with no session is unattributed, and the OS user it carries says whose account ran it.

Run

One target executing under one magus invocation, such as magus run test web or magus affected ci. A run keeps its captured output behind an output reference. Every magus run is a run whether or not any job asked for it; see Job for how the two relate.

Lease

The grant a holder takes on a job: the write and read paths that job declared, enforced in the checkout that took it with magus job exec. A job is the piece of work; a lease is permission over it.

Conventions

This page uses none of the site's convention markers. The full set is on the conventions page.