magus v0.4.3 is out. See what's new
¶ View markdown source · ✎ Suggest an edit
15 min read

GitHub Actions

magus publishes three composite actions. Each does one thing, and a workflow reaches for as few of them as it needs.

action what it does
setup-magus installs magus and puts it on PATH
magus runs a magus command, writes the run summary, or merges shard histories
advice leaves pull request advice on what your build graph noticed

Reference them from a tag, never a branch:

- uses: egladman/magus/.github/actions/setup-magus@v0.4.0

The smallest workflow that works

name: CI
on:
  pull_request:
  push:
    branches: [main]

jobs:
  ci:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
        with:
          fetch-depth: 0
          filter: blob:none

      - uses: egladman/magus/.github/actions/setup-magus@v0.4.0

      - run: magus affected ci

fetch-depth: 0 is not optional. affected compares against a merge base, and a shallow clone has none; filter: blob:none keeps the full history affordable by fetching file contents only when something reads them. See CI checkout for why that pairing is the right default and what a shallow clone costs you instead.

What belongs in YAML

That workflow is not a starting point you outgrow. It is the shape, and a repository ten times the size should still look close to it. The reason is the whole point of running a task orchestrator: the build graph already knows what to run, in what order, and what can be skipped. Every one of those decisions re-expressed in YAML is a second copy of something magus computes, and the second copy is the one that goes stale.

So the rule is a short one. A workflow contributes a trigger, a checkout, and credentials. Everything after that is a magus run. When you find yourself adding a job, a matrix, an if:, or a path filter, check whether a target could carry it instead.

One trigger, one promise

Give each workflow exactly one reason to run and one thing it is responsible for producing. The test is mechanical: if a job needs if: github.event_name == ... to work out which situation it is in, that file is two workflows wearing one hat, and every reader after you pays to disentangle them.

magus's own repository settles on four, and the name says which is which:

file runs on ships
ci.yaml pull request, and main push nothing
cd.yaml main push docs site, per-commit container image
release.yaml v* tag binaries and release images
audit.yaml cron, manual nothing

A tag build is a deliberate release, not continuous delivery, so it is not called cd. Name a workflow for what it promises, never for the ceremony around it.

Four ways this goes wrong

These are the specific mistakes, in the order they are usually made.

Splitting a workflow to get a permission boundary. This is the most tempting one, because it feels like security. It is not: permissions: is valid per job, so one file can hold a job with packages: write next to one with only contents: read. Splitting the file buys no isolation and costs you a duplicated checkout, toolchain install, and magus install in every copy. Scope permissions at the job. Split files by trigger.

A path filter instead of affected. A filter is a hand-maintained list of every input to a build, and it fails silently: the day someone adds an input the list does not mention, the job stops running and nothing reports it. magus affected derives that set from declared sources, so it cannot fall behind the tree. Prefer paying a few minutes per push over a filter nobody will remember to update.

Pinning a released magus to build the repository that defines it. A workflow that runs the last published release against this commit's magusfile cannot survive the window between a magusfile change and the release carrying it. magus's own docs deploy sat red on every push to main for two days for exactly this reason: the workspace renamed a built-in spell, and no published release knew the new name, so the deploy could not load the workspace at all. Build from the checkout with source-path: . when the repository is the one that defines magus. Pin a release when you are a consumer, and then only where a version skew is the thing you are deliberately measuring.

A non-blocking check on the pull request path. If a job never blocks a merge and its answer does not change with the diff, running it per pull request pays repeatedly for information that only moves when something outside the branch does. Put it on a schedule, where a failure is a signal instead of a row everyone has learned to scroll past.

Installing magus

setup-magus takes five inputs. queue-app-client-id belongs to the merge queue, restore-history is covered under Run history, and the interesting one is installation-strategy:

strategy what it installs
automatic a verified release, falling back to a source build
prebuilt the release named by git-ref, checksum-verified
source the magus that source-path defines

Reach for source when the workspace under test needs a magus that has not been released yet - a magusfile using a feature from this commit. Reach for prebuilt with an explicit git-ref everywhere else: it is faster, and it pins what ran.

If a source build is in play, note the PATH order: it provisions its own Go and prepends it, so a job that pinned a toolchain has to put its own back in front afterwards.

The checkout stays in your job. A local composite action cannot contain the checkout that makes the action loadable in the first place.

Sharding by affected project

One job that runs everything wastes a matrix. magus computes the affected set once, splits it into shards, and each shard runs only its own projects.

jobs:
  plan:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.plan.outputs.matrix }}
      count: ${{ steps.plan.outputs.count }}
    steps:
      - uses: actions/checkout@v5
        with: { fetch-depth: 0, filter: blob:none }
      - uses: egladman/magus/.github/actions/setup-magus@v0.4.0
        with: { restore-history: 'true' }
      - id: plan
        shell: bash
        run: |
          magus affected ci --plan > "$RUNNER_TEMP/plan.json"
          magus run --stdin --dry-run -o 'template={{range .outputs}}{{.name}}={{.value}}{{"\n"}}{{end}}' < "$RUNNER_TEMP/plan.json" >> "$GITHUB_OUTPUT"
          magus run --stdin --dry-run -o 'template={{.summary}}' < "$RUNNER_TEMP/plan.json" >> "$GITHUB_STEP_SUMMARY"

  ci:
    needs: plan
    if: fromJSON(needs.plan.outputs.count) > 0
    strategy:
      matrix: ${{ fromJSON(needs.plan.outputs.matrix) }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
        with: { fetch-depth: 0, filter: blob:none }
      - uses: egladman/magus/.github/actions/setup-magus@v0.4.0
      - uses: egladman/magus/.github/actions/magus@v0.4.0
        with:
          command: run ci:gha ${{ matrix.projects }}
          shard: ${{ matrix.shard }}
          n-shards: ${{ needs.plan.outputs.count }}

magus affected ci --plan computes the plan once, into a file. magus run --stdin --dry-run runs nothing: it checks the saved plan it reads on stdin and renders it through -o, so the step renders it twice without computing it twice. magus prints; the redirects into $GITHUB_OUTPUT and $GITHUB_STEP_SUMMARY are the workflow's, and magus never learns either file's name.

The plan's outputs array is the full set of job outputs, each a name and a value, and the template writes the array rather than naming its members. Keep that loop rather than listing the outputs you use today: a magus release that adds one reaches your workflow without an edit, and no output can go missing because a translator forgot to mention it. summary is the job summary as markdown, rendered by the plan for the same reason.

count guards the matrix: when nothing is affected, or the plan inherited a green run's verdict, there is no job to run, and a matrix of zero shards is an error rather than a skip.

Each shard job runs its share as positionals: the matrix entry's projects is exactly the list magus run takes. Passing shard and n-shards to the action sets MAGUS_SHARD and MAGUS_N_SHARDS, which label the run's timing events with the shard; they select nothing.

The saved plan also runs outside the matrix. magus run --stdin --shard 2 < plan.json runs exactly shard 2 of it, which reproduces one CI job locally from the plan that job ran, and magus affected ci --plan | magus run --stdin runs every shard in one process. A shard id the plan lacks, a target other than the plan's, or a malformed plan is refused before anything runs (MGS3029).

Remote caching

magus can use the Actions cache service as a shared cache, so a target another shard already built replays instead of running again. Wire the bundled spell into your magusfile:

import "spells/github/actions" as github;

magus\cache.remote(github);

The spell reads ACTIONS_RESULTS_URL and ACTIONS_RUNTIME_TOKEN. The runner injects both into an action's process but not into a plain run: step, so re-export them through $GITHUB_ENV in any job that should share the cache. It has to be a JS action that does it - a composite action's run: steps do not see them either:

- uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
  with:
    script: |
      for (const name of ['ACTIONS_RESULTS_URL', 'ACTIONS_RUNTIME_TOKEN']) {
        const value = process.env[name]
        if (!value) continue
        if (name.endsWith('_TOKEN')) core.setSecret(value)
        core.exportVariable(name, value)
      }

A job that exports neither variable misses every read and stores nothing, and so does a laptop. The spell never checks whether it runs under Actions: the exported credentials are what switch it on (see Told, never guessed).

See Remote cache for what gets stored, how entries are keyed, and the guarantees a shared cache does and does not give you.

Credentials

The same spell carries a secret provider. Select it when the workflow asks for it, since it resolves the environment a workflow injects:

import "spells/github/actions" as github;

if (os\env("SECRET_PROVIDER") == "github-actions") {
    magus\secret.provider(github);
}

and set SECRET_PROVIDER: github-actions in the workflow's env:. Selected, it prints ::add-mask:: for every value it resolves wherever it runs, so do not select it on a laptop.

You do not have to. With no provider selected, magus's built-in one already reads the environment, which is the only way to reach a repository secret - an Actions secret is write-only, and ${{ secrets.NAME }} in a step's env: block is the whole mechanism. Selecting this spell buys two things the built-in cannot do.

Short-lived tokens instead of stored keys. A reference prefixed oidc: is an audience, and magus mints a token from the runner's own endpoint:

final token = magus\secret.read("oidc:sts.amazonaws.com");

That needs a permission which is off by default, and a job without it gets no endpoint at all:

permissions:
  id-token: write

This is how a repository holds no long-lived cloud credential. Nothing is stored, and the token expires on its own.

Better masking and better misses. Every value the provider resolves is registered with the runner via ::add-mask::, so GitHub redacts it across every later step rather than only in the output magus captures. The runner already does that automatically for anything interpolated from secrets.*, but not for a step output injected into env:, and it cannot for a token minted mid-job. And when a variable is missing, the error is the workflow line to add rather than a bare "not set":

github-actions: $DOCKERHUB_TOKEN is not set in this step's environment.
  An Actions secret is only readable through the workflow file - nothing
  running inside the job can fetch one. Add it to the step:
      env:
        DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}

One rule to carry over from the secrets page: a target that reads a credential must declare skip_cache with a reason, or it becomes a replay that reports a successful login without ever authenticating.

Annotations and folded logs

The same spell teaches magus how GitHub renders a job log. Wire it when the workflow asks:

if (os\env("CI_PROVIDER") == "github-actions") {
    magus\ci.provider(github);
}
env:
  CI_PROVIDER: github-actions

Neither magus nor the spell detects Actions. Wired, the spell writes workflow commands to stdout wherever it runs, which is why the wiring waits for the workflow to ask.

Failures become ::error:: annotations that surface inline on the pull request, and each target's output folds into its own group. A declared provider wins over magus's built-ins, so a workspace can swap in its own spell for another CI system.

Reporting at the end of a run

One job, after the shards, for everything that describes the run:

report:
  needs: [plan, ci]
  if: always() && needs.plan.result == 'success'
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v5
      with: { fetch-depth: 0, filter: blob:none }
    - uses: egladman/magus/.github/actions/setup-magus@v0.4.0
      with: { restore-history: 'true' }
    - uses: egladman/magus/.github/actions/ci-outcome@v0.4.0
      with:
        ci-result: ${{ needs.ci.result }}
        shard-count: ${{ needs.plan.outputs.count }}
    - if: always() && github.ref == 'refs/heads/main'
      uses: egladman/magus/.github/actions/magus@v0.4.0
      with:
        merge-history: 'true'
        ci-result: ${{ needs.ci.result }}

ci-outcome writes the run's result and the volatility lens to the step summary. It is its own action because it invokes no magus subcommand - it reads the workspace through the typed magus\insight client - so it has nothing to do with the action that runs one.

merge-history folds each shard's run history into the persisted one, which is what makes volatility and timing data accumulate across runs - on main only, since a pull request's history describes a branch about to disappear. always(), so a red run's timings are kept too.

Run history

restore-history: 'true' restores the newest history the workflow saved, which is what --base last-passed and the shard forecaster read. It is off by default. The restore takes the newest cache entry by prefix, and any run sharing the cache scope can save one, including a run that executes pull-request code on main, such as a merge queue's validation. So turn it on only in a job that holds no secret, no write token and no id-token: the plan and the report above, never a job that signs, publishes or pushes. The same rule covers every other Actions cache: jdx/mise-action restores by default, so pass it cache: false in those jobs too.

Pull request advice

- uses: egladman/magus/.github/actions/advice@v0.4.0
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

One comment describing what your build graph noticed: generated files edited by hand, files no project claims, a change that reaches most of the workspace. The pull request comes from the triggering event, so the step takes no pull request inputs. Every advisor is an input, and every one can be silenced per pull request with a label. See Pull request advice.

Merge queue

magus queue runs as two workflows: queue.yaml validates with a read-only token, and queue-apply.yaml merges with the queue's own GitHub App and runs none of the changes' code. Copy both from this repository. The queue runs on nothing but its app: a run or a merge the job's own token makes starts no workflow, so on it the queue would validate and merge nothing.

Setup is four steps, with no YAML edit:

  1. Commit the two workflows.
  2. Run GITHUB_TOKEN=$(gh auth token) magus queue describe --provider github --base main, which stops with the app's registration link; open it and click "Create GitHub App".
  3. Run describe --app <slug> and run what it prints: allow auto-merge, install the app on this repository alone, and the environment, variable and secret commands. On a phone, paste the key into Settings > Environments > magus-queue.
  4. Apply the ruleset change it prints, requiring merge-queue from the app, with "Require branches to be up to date before merging" off.

The apply job takes the app through setup-magus. The client id is an input, since it is not a secret; the key is never an input, and reaches the action through the calling step's env. The excerpt leaves out the checkout, which persists no credential, and the gh auth setup-git step that lets git push with the apply step's GITHUB_TOKEN:

jobs:
  apply:
    environment: magus-queue
    steps:
      - id: magus
        uses: egladman/magus/.github/actions/setup-magus@<sha>
        with:
          queue-app-client-id: ${{ vars.MAGUS_QUEUE_APP_CLIENT_ID }}
        env:
          MAGUS_QUEUE_APP_PRIVATE_KEY: ${{ secrets.MAGUS_QUEUE_APP_PRIVATE_KEY }}
      - run: magus buzz tools/gha-queue.buzz -- apply --run "$RUN" --base "$MAIN" --app "$APP" --committer "$COMMITTER"
        env:
          GITHUB_TOKEN: ${{ steps.magus.outputs.queue-token }}
          MERGEQUEUE_TOKEN: ${{ steps.magus.outputs.queue-token }}
          RUN: ${{ github.event.workflow_run.id }}
          MAIN: ${{ github.event.repository.default_branch }}
          APP: ${{ steps.magus.outputs.queue-app-slug }}
          COMMITTER: ${{ steps.magus.outputs.queue-committer }}

With neither the variable nor the secret, setup-magus mints nothing and its outputs are empty; this repository's apply job then fails before the queue starts, and gha-queue.buzz refuses an apply without --app and --committer. With one and not the other setup-magus fails the job. The token is an output, never an environment variable, because setup-magus also runs in jobs that execute pull-request code. The app reaches the queue as explicit flags; nothing reads the runner's environment to guess.

Permissions

job needs
running targets contents: read
advice pull-requests: write
queue validation contents: read, pull-requests: read
queue dispatch contents: read, actions: write
queue apply contents, pull-requests, statuses and actions: write

On a pull request from a fork the default token is read-only whatever you declare, so the advice comment fails there. That is the platform's rule, not magus's, and the advisors say so rather than failing silently.

See also

cigithubactionscacheremote-cacheshardingaffectedannotationsworkflow designpermissionspath filters
Last updated (c5971189)
Earlier changes on this page (7)

Full history ↗ · Blame source ↗

Glossary

Workspace

The magus root directory that owns a set of projects and shared config; the unit magus operates over. See workspace.

Project

A directory magus recognizes as a unit of work (it has a magusfile); the unit of caching, scheduling, and dependency tracking. See workspace.

Magusfile

The magusfile.buzz that declares a project's targets (as export funs) and binds its spells. See targets.

Target

A named operation (build, test, ...) you invoke with magus run <target>; it may compose a spell's tool-native operations and depend on other targets. See targets.

Op

A single tool-native command a target composes (long form: operation); the middle of the work hierarchy (Spell to Op to Target). See operations.

Spell

A language/runtime adapter (e.g. go, md) that maps generic targets onto a toolchain's real commands. See spells.

Ward

A coded diagnostic that inspects a resolved op and nudges or blocks an anti-pattern before it runs. See wards.

Buzz

The language magusfiles are written in (the .buzz engine). See engines.

Cache

The content-addressed store magus consults before running a target, so unchanged work is skipped. See cache.

Affected

The set of projects touched by a change; magus affected <target> runs a target only over them. See affected.

Sandbox

The restricted filesystem and environment a target runs in, so builds stay reproducible and side-effect-free. See sandbox.

Service

A long-running or shared process magus manages across runs, distinct from a one-shot target. See services.

CI

An ordinary magusfile-defined target you compose yourself with magus\needs - magus does not hardcode its stages. Magus.RunCI treats it specially only in that it strips the rw charm, it is the anchor magus affected ci keys off, and a selected scope with no project declaring it is a load error rather than a silent no-op. See targets.

Remote cache

A CI-only backend that shares content-addressed artifacts across runners: a cold machine replays a build another runner already did instead of rebuilding. Every remote artifact must be signed by a trusted key. See remote.

Volatility

A target that fails once and passes on rerun is volatile, as opposed to a regression that started failing and stays failing. magus keeps per-target pass/fail history and a Wilson-score volatility rate to tell them apart and auto-retry the noise. See volatility.

Insight

The reports magus derives over the graph and history (hotspots, affinity, ownership, trend, volatility, unreferenced). See insight.

Window

The terminal a command runs in. It keys fire-once notices for a caller no host gave a session, and is never recorded as a session.

Job

The unit of delegated work, and one row of the job store: what an orchestrating agent handed out, with its goal, the checkpoint it was cut against, the paths it may write or must not touch, and the one check it runs. A job's holder is either a session, for work an orchestrator handed out, or the server, for its own maintenance. The store records; the agent guard is what reads those facts back when grading a write. See doctrine.

A job is not a run. magus run build web is a run, and no job exists for it. A job causes runs: its check executes as one, and a server job records the invocation of its last one. Jobs are listed with magus ls jobs and in the console's Jobs view; runs are listed in the Runs view.

Run

One target executing under one magus invocation, such as magus run test web or magus affected ci. A run keeps its captured output behind an output reference. Every magus run is a run whether or not any job asked for it; see Job for how the two relate.

Lease

The grant a holder takes on a job: the write and read paths that job declared, enforced in the checkout that took it with magus job exec. A job is the piece of work; a lease is permission over it.

Advisor

One read-only check from the advice suite: it reads the changeset through magus and writes one titled section of findings. The same advisors run as a pull request comment in CI and inside magus diff --impact locally.

Conventions

Placeholders

Angle brackets mark a value you replace with your own - never type the brackets:

magus run <target>
magus completion <shell>    # e.g. bash, zsh, fish

<target>, <path>, <shell>, <name> and the like are stand-ins, not literal text.