diff options
Diffstat
| -rw-r--r-- | CONCEPT.md | 691 | +0 −691 |
| -rw-r--r-- | LICENSES/CC-BY-4.0.txt | 156 | +156 −0 |
| -rw-r--r-- | README.md | 118 | +117 −1 |
| -rw-r--r-- | REUSE.toml | 10 | +8 −2 |
| -rw-r--r-- | docs/mirumfile.md | 590 | +590 −0 |
| -rw-r--r-- | docs/whitepaper.md | 594 | +594 −0 |
6 files changed, 1465 insertions, 694 deletions
diff --git a/CONCEPT.md b/CONCEPT.md deleted file mode 100644 --- a/CONCEPT.md +++ /dev/null @@ -1,691 +0,0 @@ -# The Concept Document - -> [!CAUTION] -> This document describes how I would like to design the system; -> it doesn't reflect the current state. Almost everything is still unimplemented. - -The really portable CI platform with VM-first isolation, -programmable pipelines, and local execution parity. - -## [Why](https://xkcd.com/927/) - -The goal of this project is to build a CI system that's both convenient for tiny -projects and suitable for gigantic C++ codebases like Chromium or llvm. - -For this to work, several key decisions need to be made: - -**Open source.** -You can and should build SaaS, but the user must be able to deploy the entire -system themselves. Self-hosted runners aren't enough. - -**Portability and cross-platform support.** -You should be able to run on at least x64, arm64, riscv64, powerpc, s390x, and -loongarch64, as well as Linux, {Free,Open,Net}BSD, Windows, and macOS. Users -should be able to add exotic features like Haiku and Plan9, or even a custom kernel. - -**Hermitic builds.** -All official platforms should support sealed builds with declarative environment -descriptions (like Docker, yes). Hosted builds should work everywhere. - -**Different deployment models.** -Not everyone can deploy themselves, and not everyone wants to. Offer open-source -SaaS, cloud and self-hosted runners, and fully autonomous solutions. The degree -of autonomy is the user's choice. - -**Dynamic pipelines.** -Don't assume that all tasks are described by a static YAML/TOML configuration. -This is often the case, but you should have a path for dynamic pipelines when they -are needed. - -**Security and Isolation.** -CI is the most security-sensitive platform, affecting testing, deployment to production, -and releases. -CI has access to both the code and the production environment. -Since the "just hide it behind a VPN" option doesn't work for either SaaS or open -source projects where CI must be public, you can't be overly paranoid about architectural -decisions. Consider that the code for your pacemaker might be tested here. - -**Local debugging.** -You should be able to run the build locally, get the console into the sandbox, and -attach a debugger. There's nothing more pointless and merciless than trying to fix -automation than the cycle of "test commit -> push to Git Forge -> hope it works." - -- `mirum run` — run the pipeline locally with the same VMs but with a local copy - of your code and a debugger -- `mirum ssh` — connect to a failed VM over SSH and debug on real hardware -- `mirum try` — run a build from local changes on the cluster without creating - throwaway commits. - -## How - -The architecture is built on two key solutions: -[Virtual Machines](https://en.wikipedia.org/wiki/Virtualization#Hardware_virtualization) -and [Starlark](https://starlark-lang.org/). - -### Virtualization vs. Containers vs. Host - -Modern CI must provide a reproducible and hermetic environment for every task. -Despite the popularity of container isolation, it's a Linux-specific technology -with many limitations. As soon as you need Windows, a specific kernel version, -or even systemd, you're forced to revert to bare, stateful runners you've manually -configured. - -A solution was proposed in Sourcehut: use ephemeral VMs from a snapshot for isolated -builds. Unlike a container, a VM can run any guest system, can emulate inaccessible -architectures, provides a full stack including the kernel, and provides sufficient -isolation to allow a user to access the build machine via SSH. - -The idea is to split the runner into two layers: - -- mirum-agent, a highly portable statically linked C binary that can copy files, - execute bash commands, and collect logs and resources. -- mirum-worker, a full-fledged runtime that launches a disposable VM for each task, - launching and managing mirum-agent within the VM. - -This architecture allows for full support on official platforms, including isolation, -SSH access, and so on. On the other hand, if you're testing an exotic system -(for example, on bare metal without any OS at all), port mirum-agent and you'll -be able to connect to a regular mirum server. You can also offer specialized versions -of mirum-worker for containers/clouds/lambdas, or any custom environment. -Simply run mirum-agent in a sandbox and issue commands to it. - -Using VMs also significantly simplifies infrastructure. -One x64 host can run Linux, Windows, and *BSD, -while one arm64 mac mini can run macOS, Linux, and Windows on arm64. - -### Starlark - -Today, there are two ways to describe pipelines: statically in yaml or by writing -a script in a scripting language like JavaScript/Python/Ruby. - -Mirum occupies a niche between the two and offers Starlark as a configuration language. -Starlark is a specialized embedded programming language developed for the Bazel -build system. On the one hand, you have variable conditions and loops, imports, -objects, and arrays. - -Unlike yaml, you don't have to reinvent the wheel. Settings are variables, tasks -are functions, build matrices are loops, and the reusable actions library is -a simple import. Python syntax is well-known and doesn't require learning your DSL. - -On the other hand, it's not an algorithmically complete language. There are no side -effects, no need for a separate sandbox, and no need to drag in a runtime. The CI -server has total control over Starlark execution. But unlike Kotlin DSL (TeamCity), -Groovy (Jenkins), or Python (Buildbot), Starlark forbids side effects: no network, -no filesystem, no arbitrary imports. Eval is safe for untrusted code -(PRs from external contributors), deterministic, and cacheable. - -## Modules - -Four executable modules: - -``` -┌──────────────────────────────────────────────────────┐ -│ mirum-server │ -│ watch registry, task queue, log aggregation, WebUI │ -└─────────────────────────────────────────────────────┘ - ↑ gRPC (worker-initiated) ↑ gRPC - │ │ -┌────────┴──────────┐ ┌─────────┴─────────┐ -│ mirum-worker │ │ mirum-worker │ -│ Linux (KVM) │ │ macOS (Vz) │ -│ │ vsock │ │ │ vsock │ -│ ↓ │ │ ↓ │ -│ ┌────────────┐ │ │ ┌────────────┐ │ -│ │mirum-agent │ │ │ │mirum-agent │ │ -│ │ inside VM │ │ │ │ inside VM │ │ -│ └────────────┘ │ │ └────────────┘ │ -└───────────────────┘ └───────────────────┘ - -┌───────────────────┐ -│ mirum (CLI) │ -│ Starlark eval, │ -│ spawns worker │ -└───────────────────┘ -``` - -**mirum-server** — the orchestrator. -Contains a database, stores users, organizations, project and pipeline settings, -provides a WebUI and API, responds to webhooks, and distributes tasks to workers. -Collects logs and build results from workers. - -It's a control plane. - -**mirum-worker** — the task executor -Connects to the server via outbound gRPC, declares its capabilities, -and picks tasks from the queue that it can execute. - -Executes starlark (project + pipeline functions, coroutine model) and notifies -the server of changes (for example, new watch settings). -Starts a VM with tasks and returns the results to the server. If starlark is blocked -on yeald, it leaves the blocked coroutine in the queue. - -It's a data plane. _Only the worker has access to the user's secrets and code._ - -Workers come in different kinds: KVM worker for local VMs, -macOS worker with Vz.framework, windows worker with Hyper-V, -EC2 worker for cloud VMs, host worker for direct execution. -A new worker type joins the cluster and starts picking up tasks with no changes -to the server. - -**mirum-agent** — static binary pre-installed in every VM. - -A tiny bridge between the worker on the host and the tasks running in the VM. -Written in C99, no dependencies, posix-only. Implements a simple TLV for communicating -with the host (via virtio-vsock or tcp socket, depending on the hypervisor). - -Channels: control, stdin, stdout, stderr, file transfers, interactive shell sessions. - -Small and simple enough to be auditable. -This component lives close to the actual code, making it highly security-sensitive. - -**mirum (CLI)** — developer tool. -Contains a Starlark runtime for local eval — takes the server's role locally. - -Technically, it is a lightweight disposable server that runs a real worker locally -to replicate real-world conditions in a cluster as closely as possible. - -`mirum run`: eval pipeline → spawn worker → dispatch tasks → display logs. -`mirum try`: send a diff to the server. -`mirum attach`: shell into a VM. -`mirum eval`: show the DAG without running anything. - -## Configuration Model - -Mirum operates on projects, a project consists of pipelines, -pipelines consist of tasks. A task is the minimum unit of execution — it always -runs in a single VM. Pipelines are DAGs (directed acyclic graphs) that invoke multiple -tasks and pass state between them. A project tracks a list of pipelines, their trigger -rules (watch, cron, manual, ...), and result notifications. - -A quick example pipeline, all in one starlark file: - -```python -# /.mirum/main.star — entry point and the only file required - -# A single pipeline describes a matrix of multiple operating systems and architectures. -# A single Linux host with KVM serves Linux, Windows, and BSD guests. -# A macOS host serves macOS, Linux, and BSD. -# Tasks adapt to the platform via `ctx.os` — they don't choose it. - -# ctx.run takes an optional setup function to indicate which steps -# are environment setup only, so the resulting image can be cached -def setup(ctx): - ctx.shell("apt-get update && apt-get install -y cargo") - -def build(ctx): - ctx.checkout() - ctx.shell("cargo build --release") - ctx.upload("target/release/myapp", artifact="bin") - -def test(ctx): - # test knows nothing about build — only that it needs an artifact. - # that artifact could have been built right here, come from cache - # or a registry, or even uploaded from a developer's laptop - ctx.download("bin", dest=".") - ctx.shell("cargo test") - -# Write a function that describes your pipeline. -def build_pipeline(ctx): - src = ctx.source() - - # Build steps are simply functions that the pipeline calls in the VM sandbox. - b = ctx.run(build, setup=setup, source=src, image="mirum/ubuntu-24.04") - ctx.run(test, setup=setup, depends=b, image="mirum/ubuntu-24.04") - -# Describe your project, what pipelines exist, and how to launch them. -def project(ctx): - ctx.pipeline("build", fn=build_pipeline, watch=ctx.watch(events=["push"], manual=True)) -``` - -### Functions All the Way Down - -Mirum configuration is nested function composition in Starlark. -`project` is the entry point and the only required function — it registers -pipelines and the rules for "what, from where, and when to run." - -A pipeline is a function that imperatively executes tasks one by one and passes -dependencies between them. A task's result can be used to launch further tasks, -enabling dynamic task creation. You don't need separate syntax for -"allow failure / retry / skip_if / matrix" — it's just if/for in Starlark. - -``` -# /.mirum/main.star — the entry point the server looks for - -project(ctx) → event routing "when to trigger" - -pipeline(ctx) → DAG of tasks "what to run, on which platforms" - -task(ctx) → scripts + artifacts "how to build" -``` - -Starlark supports imports, so splitting a large project works out of the box. -The only requirement is a `.mirum/` directory at the repository root with `main.star` -as the entry point. The internal structure of `.mirum/` is free-form. - -The fixed `.mirum/` path is not a convention but an architectural requirement. -The server reads configuration via the forge contents API (`GET /repos/Org/repo/contents/.mirum/`), -not via git clone. This allows: fetching only configuration files without accessing -source code, filtering webhooks (push with no changes in `.mirum/` → skip re-eval), -caching configuration by the directory's commit SHA. - -To reuse code both within and outside the project, starlark load is used. - -``` -@mirum// Standard library (services, apt, bazel helpers) -@pkg// External packages (from deps.star, pinned by commit) -// Local files (relative to .mirum/ root) -``` - -### Coroutine Eval: Dynamic DAGs - -Pipeline functions execute as coroutines. `ctx.run()` without accessing results -is non-blocking — the server accumulates pending tasks. Accessing a result (`.output()`) -is a yield point: the server dispatches all pending tasks, waits for the needed -result, and resumes the pipeline function. - -```python -def build(ctx): - ... - -def discover_tests(ctx): - ... - -def run_test(ctx): - ... - -def pipeline(ctx): - src = ctx.source() # where sources come from: git, mirum try, or a local directory - - # All ctx.run() calls before the first .output() accumulate and run in parallel - builds = {} - for os in ["linux", "mac"]: - builds[os] = ctx.run(build, source=src, - image="mirum/%s" % os, args={"os": os}) - - discovery = ctx.run(discover_tests, source=src, - image="mirum/linux") - - # Yield: server dispatches builds + discovery in parallel, - # waits for discovery to complete, resumes with result - test_modules = discovery.output("modules") - - # Dynamic phase: use runtime data - for module in test_modules: - ctx.run(run_test, args={"module": module}, - depends=list(builds.values())) -``` - -If a pipeline function never calls `.output()`, the entire DAG is built in -a single pass. A static DAG is a special case of the dynamic model. - -`mirum eval` executes the pipeline function locally without dispatching tasks. -For static DAGs it prints the full graph. For dynamic DAGs — everything up to -the first yield point, marked "depends on runtime data beyond this point." - -### Example: Everything in One File - -```python -# /.mirum/main.star — complete CI - -# ctx.run takes an optional setup function to indicate which steps -# are environment setup only, so the resulting image can be cached -def setup(ctx): - ctx.shell("apt-get update && apt-get install -y cargo") - -def build(ctx): - ctx.checkout() - ctx.shell("cargo build --release") - ctx.upload("target/release/myapp", artifact="bin") - -def test(ctx): - # test knows nothing about build — only that it needs an artifact. - # that artifact could have been built right here, come from cache - # or a registry, or even uploaded from a developer's laptop - ctx.download("bin", dest=".") - ctx.shell("cargo test") - -def ci(ctx): - src = ctx.source() - b = ctx.run(build, setup=setup, source=src, image="mirum/ubuntu-24.04") - ctx.run(test, setup=setup, depends=b, image="mirum/ubuntu-24.04") - -def project(ctx): - ctx.pipeline("ci", fn=ci, watch=ctx.watch(events=["push"])) -``` - -### Example: Cross-Platform Project - -```python -# tasks/setup.star -def cpp_toolchain(ctx): - if ctx.os == "linux": - ctx.shell("apt-get update && apt-get install -y cmake ninja-build") - elif ctx.os == "freebsd": - ctx.shell("pkg install -y cmake ninja") - elif ctx.os == "windows": - ctx.shell("choco install -y cmake ninja visualstudio2022-workload-vctools") - elif ctx.os == "macos": - ctx.shell("brew install cmake ninja") - -# tasks/build.star -def build(ctx): - ctx.checkout() - if ctx.os == "windows": - ctx.shell('cmake -G "Visual Studio 17 2022" -B build .') - elif ctx.os == "macos": - ctx.shell("cmake -B build -DCMAKE_OSX_DEPLOYMENT_TARGET=12.0 .") - else: - ctx.shell("cmake -B build .") - ctx.shell("cmake --build build --config Release") - ctx.upload("build/out/*", artifact="pkg") - -def publish(ctx): - ctx.download("pkg", dest="release/") - ctx.shell("gh release create $TAG release/* --generate-notes") - -# pipelines/release.star -load("//tasks/setup.star", "cpp_toolchain") -load("//tasks/build.star", "build", "publish") - -IMAGES = { - "linux": "mirum/ubuntu-24.04", - "freebsd": "mirum/freebsd-14", - "windows": "mirum/windows-2025", - "macos": "mirum/macos-15", -} - -# Irregular matrix — just a list. No exclude needed. -PLATFORMS = [ - ("linux", "amd64"), - ("linux", "arm64"), - ("macos", "arm64"), - ("windows", "amd64"), - ("freebsd", "amd64"), -] - -def pipeline(ctx): - src = ctx.source(ref=ctx.event.tag) - - # Build: loop over platforms, each task gets its own handle - builds = {} - for os, arch in PLATFORMS: - builds[(os, arch)] = ctx.run(build, - setup=cpp_toolchain, - source=src, - image=IMAGES[os], - args={"os": os, "arch": arch}) - - # Publish: fan-in, waits for all builds - ctx.run(publish, depends=list(builds.values())) - -# .mirum/main.star -def project(ctx): - ctx.pipeline("release", file="pipelines/release.star", - watch=ctx.watch(events=["tag"], pattern="v*")) -``` - -The pipeline decides WHERE (image, platforms). Setup decides WITH WHAT -(toolchain, cached snapshot). The task decides HOW (checkout, build, upload). -Tasks don't know what platform they're running on — `ctx.os` and `ctx.arch` are -injected by the pipeline. - -## Images - -The most tedious and time-consuming task is preparing images for various operating -systems. Some distributions distribute qcow2, some support cloud-init, and some only -offer an ISO installer. Some require a network connection for configuration, while -others work offline. - -Another problem is distribution. The reason containers are popular is the OCI registry. -A container is easy to upload to a server, and just as easy to download and deploy. -Nothing similar exists for VMs. - -An elegant solution was found in Tart by CirrusCI: use OCI as a black box for storing -the VM image. Load it into your existing infrastructure, easily update, and distribute. -A single distribution format for all platforms. Images are stored as compressed -raw disk chunks: - -``` -OCI Image Manifest: - config: - mediaType: "application/vnd.mirum.image.config.v1+json" - { mirum.version, os, arch, distro, distro_version, - agent_version, disk_size, chunk_size } - - layers: - - mediaType: "application/vnd.mirum.disk.raw.v1+zstd" - annotations: { "mirum.offset": "0", "mirum.length": "67108864" } - - ... - - # macOS additionally: - - mediaType: "application/vnd.mirum.aux.v1+zstd" - - mediaType: "application/vnd.mirum.hwmodel.v1+json" -``` - -Each chunk is independently zstd-compressed. The worker downloads and decompresses -in parallel. 64MB chunks for a 4GB disk ≈ 64 layers. On pull worker reassembles -raw disk from chunks → converts to hypervisor format -(qcow2, vhdx, Vz native) → caches → CoW clone per task. - -### Three Layers (VM Runtime) - -VM images are larger than container images, but there are fewer of them. -We can borrow the layer caching idea and apply it to image snapshots (like in qcow2+). - -``` -Layer 0: Base image (from OCI registry) - Golden (Mirum-maintained) or organization (external). - Worker downloads, converts to hypervisor format, caches locally. - -Layer 1: Setup (function from ctx.run(setup=...)) - Declared in the pipeline. Worker executes, takes a snapshot. - Cache is local, best-effort, evicted by LRU. - -Layer 2: Ephemeral overlay - CoW clone of setup cache (or base). Per-task. Destroyed. -``` - -### Setup as a Function - -The pipeline passes two functions to `ctx.run()`: -`setup` (optional) and the main task. The image is also specified in the pipeline: - -```python -def cpp_setup(ctx): - if ctx.os == "linux": - ctx.shell("apt-get update && apt-get install -y cmake ninja-build") - elif ctx.os == "freebsd": - ctx.shell("pkg install -y cmake ninja") - -def build(ctx): - ctx.checkout() - ctx.shell("cmake -B build . && cmake --build build") - -def pipeline(ctx): - ctx.run(build, setup=cpp_setup, source=ctx.source(), - image="mirum/ubuntu-24.04", - args={"os": "linux"}) -``` - -The worker hashes `(image_digest, setup_function_hash, os, arch)`. -Cache hit → CoW clone, boot in milliseconds. Miss → boot base, run setup, -snapshot, cache. - -Setup is an ordinary Starlark function, composable via `load()`: - -```python -# @pkg//acme/setup.star -def cpp_toolchain(ctx): - if ctx.os == "linux": - ctx.shell("apt-get update && apt-get install -y cmake ninja-build") - elif ctx.os == "windows": - ctx.shell("choco install -y cmake ninja") -``` - -```python -load("@pkg//acme/setup.star", "cpp_toolchain") - -def pipeline(ctx): - for os in ["linux", "windows"]: - ctx.run(build, setup=cpp_toolchain, source=src, - image=IMAGES[os], args={"os": os}) -``` - -One `cpp_toolchain` across the entire organization — one hash — one snapshot per worker. - -Three levels, cleanly separated: - -- **Pipeline**: WHERE (image, platforms, source) -- **Setup**: WITH WHAT (toolchain, dependencies — cached snapshot) -- **Task**: HOW (checkout, build, test, upload) - -Organizations that need a fully pre-built image can publish it to any OCI registry -using external tooling and reference it directly: - -```python -# not need setup for preconfigured image -ctx.run(build, source=src, image="acme-registry.com/ci-base:latest") -``` - -### Golden Images - -Golden images are built by the Mirum team using Packer, not by users: - -| Platform | Packer builder | Install method | -| ------------------------- | ---------------------- | ----------------------------------- | -| Linux (Ubuntu, Fedora...) | `qemu` | cloud-init / preseed / kickstart | -| macOS | `tart` (Packer plugin) | VZMacOSInstaller + VNC boot_command | -| Windows | `qemu` | autounattend.xml (evaluation ISO) | -| NetBSD | `qemu` | sysinst auto | -| FreeBSD | `qemu` | bsdinstall scripted | -| OpenBSD | `qemu` | autoinstall response file | - -Windows: evaluation ISO is freely downloadable. The evaluation period (180 days) -is irrelevant for ephemeral CI VMs. Users activate with their own key if needed. - -macOS: `.ipsw` installed via Virtualization.framework. Setup Assistant automated -via VNC keystroke injection. Requires Apple hardware for building and running. - -## Extensibility - -Starlark simplifies building plugins and libraries. You already have `load`, -so you don't need to invent your own systems like reusable actions. - -**Transparent worker optimizations** — invisible to the task. -Configured in `worker.yaml`. The worker configures the VM environment before running -any scripts: apt mirror, cargo/npm cache mount, HTTP proxy. -`./ci.sh` with `apt-get install` inside simply runs faster. -Bash scripts speed up for free. - -```yaml -# worker.yaml -optimizations: - apt_mirror: "http://apt-cache.internal:3142" - http_proxy: "http://squid.internal:3128" - cargo_cache: "/mnt/shared/cargo" -``` - -**Starlark stdlib (@mirum//)** — for things that require an explicit decision. -This is a standard Starlark that, although it comes with an agent, -doesn't require any additional APIs (see the Bazel vs Buck configurations). -Users can read, fork, or write their own. - -Starlark sees capabilities via `ctx.worker.has("docker")`. -The stdlib adapts. No hidden magic — the code is readable. - -The dividing principle: if an optimization can be applied without changing -task behavior — it's a transparent worker optimization. If the task needs to know -(e.g. a postgres address) — it's Starlark stdlib with graceful degradation. - -Checks worker capabilities and adapts: - -```python -load("@mirum//services", "service") - -def test(ctx): - # docker on the worker? → sidecar container - # no docker? → install and run inside the VM - service(ctx, "postgres", image="postgres:16", port=5432) - ctx.shell("make test") -``` - -Workers declare capabilities on registration: - -```yaml -# worker.yaml -capabilities: - kvm: true - gpu: false - docker: true -``` - -**Server plugins (traits)** — extend the platform. Implement trait interfaces. -Configured in `server.yaml`: - -| Trait | AGPL built-in | External plugin | -| ---------------- | ------------- | ---------------------- | -| AuthBackend | Token, basic | SAML, OIDC, LDAP | -| Source provider | Git, VSC, s3 | Mercurial, Perforse | -| SecretProvider | Env, systemd | Vault, AWS KMS | -| NotificationSink | — | Slack, email, webhooks | -| BillingHook | Noop | Usage metering | - -## Comparison - -We were inspired by many wonderful tools -- Buildbot: centralized master, property model, `try` for pre-commit testing, dynamic build steps; -- TeamCity: vsc roots, multi-tenant, role model, breadth of tool support; -- Concourse: the idea of universal input/output; -- SourceHut: SSH into VMs for debugging, BSD support; -- Cirrus CI: ephemeral VMs, bring-your-own cloud, Starlark, agent inside VM; -- GitHub Actions: how not to do it. - -### vs GitHub Actions - -Paid, closed, inseparable from Microsoft, only ubuntu/windows/macOS, -only x64/arm64, nodejs required in runtime, -yaml configs (and only for the current repository). -No cross-OS matrices out of the box (macOS runners are a paid add-on). -No local execution. No dynamic DAG. Caching is an action, not a primitive. - -### vs GitLab CI - -GitLab CI is part of GitLab. YAML. DAG via `needs:` — a hack on top of a stage-based model. -Runners are stateful machines or Docker. No VM isolation. `include:` / YAML anchors — fragile reuse. -Dynamic child pipelines — via YAML generation. - -### vs Jenkins - -Groovy DSL is powerful but allows arbitrary code (RCE when eval'ing PRs). -Plugin ecosystem is huge but fragile (Security Advisories every month). -Agents are stateful, workspace persists. Shared Libraries are Groovy classes, trusted/untrusted. - -### vs TeamCity - -Powerful and popular, with clever ideas (dedicated VCS root, for example). -But it's paid and expensive, closed-source, and difficult to use. -Kotlin DSL is typed with IDE support, but allows side effects (HTTP, filesystem). -Agents are stateful and require maintenance. Snapshot dependencies equal our source consistency. - -Templates (1:1) vs our `load()` (N:N) — Starlark is strictly more powerful. - -### vs Buildbot - -The most portable and flexible of all. However, it's outdated, difficult to configure, -and designed for hosted builds. Its architecture doesn't support SaaS. -Pure Python configurations on both the master and agents. No IaC out of the box. - -### vs Cirrus CI - -Great tool, but unfortunately still closed source and dependent on gcloud. -Starlark is only available as an advanced mode with yaml. -It lacks support for many BSDs (but FreeBSD is available!). - -## Licensing - -**AGPL-3.0** — all four modules, all built-in traits, all runtimes, standard library, -full CLI, basic Web UI, SQLite, single-tenant auth. - -**Commercial license** — For companies unwilling to use the AGPL, offer a commercial -license, certifications, and SLAs in SaaS. Don't hesitate to take money -from enterprises and spend it on open source. diff --git a/LICENSES/CC-BY-4.0.txt b/LICENSES/CC-BY-4.0.txt new file mode 100644 --- /dev/null +++ b/LICENSES/CC-BY-4.0.txt @@ -0,0 +1,156 @@ +Creative Commons Attribution 4.0 International + + Creative Commons Corporation (“Creative Commons”) is not a law firm and does not provide legal services or legal advice. Distribution of Creative Commons public licenses does not create a lawyer-client or other relationship. Creative Commons makes its licenses and related information available on an “as-is” basis. Creative Commons gives no warranties regarding its licenses, any material licensed under their terms and conditions, or any related information. Creative Commons disclaims all liability for damages resulting from their use to the fullest extent possible. + +Using Creative Commons Public Licenses + +Creative Commons public licenses provide a standard set of terms and conditions that creators and other rights holders may use to share original works of authorship and other material subject to copyright and certain other rights specified in the public license below. The following considerations are for informational purposes only, are not exhaustive, and do not form part of our licenses. + +Considerations for licensors: Our public licenses are intended for use by those authorized to give the public permission to use material in ways otherwise restricted by copyright and certain other rights. Our licenses are irrevocable. Licensors should read and understand the terms and conditions of the license they choose before applying it. Licensors should also secure all rights necessary before applying our licenses so that the public can reuse the material as expected. Licensors should clearly mark any material not subject to the license. This includes other CC-licensed material, or material used under an exception or limitation to copyright. More considerations for licensors. + +Considerations for the public: By using one of our public licenses, a licensor grants the public permission to use the licensed material under specified terms and conditions. If the licensor’s permission is not necessary for any reason–for example, because of any applicable exception or limitation to copyright–then that use is not regulated by the license. Our licenses grant only permissions under copyright and certain other rights that a licensor has authority to grant. Use of the licensed material may still be restricted for other reasons, including because others have copyright or other rights in the material. A licensor may make special requests, such as asking that all changes be marked or described. Although not required by our licenses, you are encouraged to respect those requests where reasonable. More considerations for the public. + +Creative Commons Attribution 4.0 International Public License + +By exercising the Licensed Rights (defined below), You accept and agree to be bound by the terms and conditions of this Creative Commons Attribution 4.0 International Public License ("Public License"). To the extent this Public License may be interpreted as a contract, You are granted the Licensed Rights in consideration of Your acceptance of these terms and conditions, and the Licensor grants You such rights in consideration of benefits the Licensor receives from making the Licensed Material available under these terms and conditions. + +Section 1 – Definitions. + + a. Adapted Material means material subject to Copyright and Similar Rights that is derived from or based upon the Licensed Material and in which the Licensed Material is translated, altered, arranged, transformed, or otherwise modified in a manner requiring permission under the Copyright and Similar Rights held by the Licensor. For purposes of this Public License, where the Licensed Material is a musical work, performance, or sound recording, Adapted Material is always produced where the Licensed Material is synched in timed relation with a moving image. + + b. Adapter's License means the license You apply to Your Copyright and Similar Rights in Your contributions to Adapted Material in accordance with the terms and conditions of this Public License. + + c. Copyright and Similar Rights means copyright and/or similar rights closely related to copyright including, without limitation, performance, broadcast, sound recording, and Sui Generis Database Rights, without regard to how the rights are labeled or categorized. For purposes of this Public License, the rights specified in Section 2(b)(1)-(2) are not Copyright and Similar Rights. + + d. Effective Technological Measures means those measures that, in the absence of proper authority, may not be circumvented under laws fulfilling obligations under Article 11 of the WIPO Copyright Treaty adopted on December 20, 1996, and/or similar international agreements. + + e. Exceptions and Limitations means fair use, fair dealing, and/or any other exception or limitation to Copyright and Similar Rights that applies to Your use of the Licensed Material. + + f. Licensed Material means the artistic or literary work, database, or other material to which the Licensor applied this Public License. + + g. Licensed Rights means the rights granted to You subject to the terms and conditions of this Public License, which are limited to all Copyright and Similar Rights that apply to Your use of the Licensed Material and that the Licensor has authority to license. + + h. Licensor means the individual(s) or entity(ies) granting rights under this Public License. + + i. Share means to provide material to the public by any means or process that requires permission under the Licensed Rights, such as reproduction, public display, public performance, distribution, dissemination, communication, or importation, and to make material available to the public including in ways that members of the public may access the material from a place and at a time individually chosen by them. + + j. Sui Generis Database Rights means rights other than copyright resulting from Directive 96/9/EC of the European Parliament and of the Council of 11 March 1996 on the legal protection of databases, as amended and/or succeeded, as well as other essentially equivalent rights anywhere in the world. + + k. You means the individual or entity exercising the Licensed Rights under this Public License. Your has a corresponding meaning. + +Section 2 – Scope. + + a. License grant. + + 1. Subject to the terms and conditions of this Public License, the Licensor hereby grants You a worldwide, royalty-free, non-sublicensable, non-exclusive, irrevocable license to exercise the Licensed Rights in the Licensed Material to: + + A. reproduce and Share the Licensed Material, in whole or in part; and + + B. produce, reproduce, and Share Adapted Material. + + 2. Exceptions and Limitations. For the avoidance of doubt, where Exceptions and Limitations apply to Your use, this Public License does not apply, and You do not need to comply with its terms and conditions. + + 3. Term. The term of this Public License is specified in Section 6(a). + + 4. Media and formats; technical modifications allowed. The Licensor authorizes You to exercise the Licensed Rights in all media and formats whether now known or hereafter created, and to make technical modifications necessary to do so. The Licensor waives and/or agrees not to assert any right or authority to forbid You from making technical modifications necessary to exercise the Licensed Rights, including technical modifications necessary to circumvent Effective Technological Measures. For purposes of this Public License, simply making modifications authorized by this Section 2(a)(4) never produces Adapted Material. + + 5. Downstream recipients. + + A. Offer from the Licensor – Licensed Material. Every recipient of the Licensed Material automatically receives an offer from the Licensor to exercise the Licensed Rights under the terms and conditions of this Public License. + + B. No downstream restrictions. You may not offer or impose any additional or different terms or conditions on, or apply any Effective Technological Measures to, the Licensed Material if doing so restricts exercise of the Licensed Rights by any recipient of the Licensed Material. + + 6. No endorsement. Nothing in this Public License constitutes or may be construed as permission to assert or imply that You are, or that Your use of the Licensed Material is, connected with, or sponsored, endorsed, or granted official status by, the Licensor or others designated to receive attribution as provided in Section 3(a)(1)(A)(i). + +b. Other rights. + + 1. Moral rights, such as the right of integrity, are not licensed under this Public License, nor are publicity, privacy, and/or other similar personality rights; however, to the extent possible, the Licensor waives and/or agrees not to assert any such rights held by the Licensor to the limited extent necessary to allow You to exercise the Licensed Rights, but not otherwise. + + 2. Patent and trademark rights are not licensed under this Public License. + + 3. To the extent possible, the Licensor waives any right to collect royalties from You for the exercise of the Licensed Rights, whether directly or through a collecting society under any voluntary or waivable statutory or compulsory licensing scheme. In all other cases the Licensor expressly reserves any right to collect such royalties. + +Section 3 – License Conditions. + +Your exercise of the Licensed Rights is expressly made subject to the following conditions. + + a. Attribution. + + 1. If You Share the Licensed Material (including in modified form), You must: + + A. retain the following if it is supplied by the Licensor with the Licensed Material: + + i. identification of the creator(s) of the Licensed Material and any others designated to receive attribution, in any reasonable manner requested by the Licensor (including by pseudonym if designated); + + ii. a copyright notice; + + iii. a notice that refers to this Public License; + + iv. a notice that refers to the disclaimer of warranties; + + v. a URI or hyperlink to the Licensed Material to the extent reasonably practicable; + + B. indicate if You modified the Licensed Material and retain an indication of any previous modifications; and + + C. indicate the Licensed Material is licensed under this Public License, and include the text of, or the URI or hyperlink to, this Public License. + + 2. You may satisfy the conditions in Section 3(a)(1) in any reasonable manner based on the medium, means, and context in which You Share the Licensed Material. For example, it may be reasonable to satisfy the conditions by providing a URI or hyperlink to a resource that includes the required information. + + 3. If requested by the Licensor, You must remove any of the information required by Section 3(a)(1)(A) to the extent reasonably practicable. + + 4. If You Share Adapted Material You produce, the Adapter's License You apply must not prevent recipients of the Adapted Material from complying with this Public License. + +Section 4 – Sui Generis Database Rights. + +Where the Licensed Rights include Sui Generis Database Rights that apply to Your use of the Licensed Material: + + a. for the avoidance of doubt, Section 2(a)(1) grants You the right to extract, reuse, reproduce, and Share all or a substantial portion of the contents of the database; + + b. if You include all or a substantial portion of the database contents in a database in which You have Sui Generis Database Rights, then the database in which You have Sui Generis Database Rights (but not its individual contents) is Adapted Material; and + + c. You must comply with the conditions in Section 3(a) if You Share all or a substantial portion of the contents of the database. +For the avoidance of doubt, this Section 4 supplements and does not replace Your obligations under this Public License where the Licensed Rights include other Copyright and Similar Rights. + +Section 5 – Disclaimer of Warranties and Limitation of Liability. + + a. Unless otherwise separately undertaken by the Licensor, to the extent possible, the Licensor offers the Licensed Material as-is and as-available, and makes no representations or warranties of any kind concerning the Licensed Material, whether express, implied, statutory, or other. This includes, without limitation, warranties of title, merchantability, fitness for a particular purpose, non-infringement, absence of latent or other defects, accuracy, or the presence or absence of errors, whether or not known or discoverable. Where disclaimers of warranties are not allowed in full or in part, this disclaimer may not apply to You. + + b. To the extent possible, in no event will the Licensor be liable to You on any legal theory (including, without limitation, negligence) or otherwise for any direct, special, indirect, incidental, consequential, punitive, exemplary, or other losses, costs, expenses, or damages arising out of this Public License or use of the Licensed Material, even if the Licensor has been advised of the possibility of such losses, costs, expenses, or damages. Where a limitation of liability is not allowed in full or in part, this limitation may not apply to You. + + c. The disclaimer of warranties and limitation of liability provided above shall be interpreted in a manner that, to the extent possible, most closely approximates an absolute disclaimer and waiver of all liability. + +Section 6 – Term and Termination. + + a. This Public License applies for the term of the Copyright and Similar Rights licensed here. However, if You fail to comply with this Public License, then Your rights under this Public License terminate automatically. + + b. Where Your right to use the Licensed Material has terminated under Section 6(a), it reinstates: + + 1. automatically as of the date the violation is cured, provided it is cured within 30 days of Your discovery of the violation; or + + 2. upon express reinstatement by the Licensor. + + c. For the avoidance of doubt, this Section 6(b) does not affect any right the Licensor may have to seek remedies for Your violations of this Public License. + + d. For the avoidance of doubt, the Licensor may also offer the Licensed Material under separate terms or conditions or stop distributing the Licensed Material at any time; however, doing so will not terminate this Public License. + + e. Sections 1, 5, 6, 7, and 8 survive termination of this Public License. + +Section 7 – Other Terms and Conditions. + + a. The Licensor shall not be bound by any additional or different terms or conditions communicated by You unless expressly agreed. + + b. Any arrangements, understandings, or agreements regarding the Licensed Material not stated herein are separate from and independent of the terms and conditions of this Public License. + +Section 8 – Interpretation. + + a. For the avoidance of doubt, this Public License does not, and shall not be interpreted to, reduce, limit, restrict, or impose conditions on any use of the Licensed Material that could lawfully be made without permission under this Public License. + + b. To the extent possible, if any provision of this Public License is deemed unenforceable, it shall be automatically reformed to the minimum extent necessary to make it enforceable. If the provision cannot be reformed, it shall be severed from this Public License without affecting the enforceability of the remaining terms and conditions. + + c. No term or condition of this Public License will be waived and no failure to comply consented to unless expressly agreed to by the Licensor. + + d. Nothing in this Public License constitutes or may be interpreted as a limitation upon, or waiver of, any privileges and immunities that apply to the Licensor or You, including from the legal processes of any jurisdiction or authority. + +Creative Commons is not a party to its public licenses. Notwithstanding, Creative Commons may elect to apply one of its public licenses to material it publishes and in those instances will be considered the “Licensor.” Except for the limited purpose of indicating that material is shared under a Creative Commons public license or as otherwise permitted by the Creative Commons policies published at creativecommons.org/policies, Creative Commons does not authorize the use of the trademark “Creative Commons” or any other trademark or logo of Creative Commons without its prior written consent including, without limitation, in connection with any unauthorized modifications to any of its public licenses or any other arrangements, understandings, or agreements concerning use of licensed material. For the avoidance of doubt, this paragraph does not form part of the public licenses. + +Creative Commons may be contacted at creativecommons.org. diff --git a/README.md b/README.md index 8292f0d..56684af 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,122 @@ # Mirum -An experimental CI built around virtual machines and Starlark +> [!CAUTION] +> This document describes the design and rationale of the system. It does +> not reflect the current state — almost everything is still +> unimplemented. For the canonical user-facing API of the configuration +> file, see [`mirumfile.md`](./docs/mirumfile.md). + +An experimental portable CI platform with VM-first isolation, +programmable pipelines, local execution parity and Starlark configs. + +## [Why](https://xkcd.com/927/) + +The goal of this project is to build a CI system that's both convenient for tiny +projects and suitable for gigantic C++ codebases like Chromium or llvm. + +For this to work, several key decisions need to be made: + +**Open source.** +You can and should build SaaS, but the user must be able to deploy the entire +system themselves. Self-hosted runners aren't enough. + +**Portability and cross-platform support.** +You should be able to run on at least x64, arm64, riscv64, powerpc, s390x, and +loongarch64, as well as Linux, {Free,Open,Net}BSD, Windows, and macOS. Users +should be able to add exotic features like Haiku and Plan9, or even a custom kernel. + +**Hermitic builds.** +All official platforms should support sealed builds with declarative environment +descriptions (like Docker, yes). Hosted builds should work everywhere. + +**Different deployment models.** +Not everyone can deploy themselves, and not everyone wants to. Offer open-source +SaaS, cloud and self-hosted runners, and fully autonomous solutions. The degree +of autonomy is the user's choice. + +**Dynamic pipelines.** +Don't assume that all tasks are described by a static YAML/TOML configuration. +This is often the case, but you should have a path for dynamic pipelines when they +are needed. + +**Security and Isolation.** +CI is the most security-sensitive platform, affecting testing, deployment to production, +and releases. +CI has access to both the code and the production environment. +Since the "just hide it behind a VPN" option doesn't work for either SaaS or open +source projects where CI must be public, you can't be overly paranoid about architectural +decisions. Consider that the code for your pacemaker might be tested here. + +**Local debugging.** +You should be able to run the build locally, get the console into the sandbox, and +attach a debugger. There's nothing more pointless and merciless than trying to fix +automation than the cycle of "test commit -> push to Git Forge -> hope it works." + +- `mirum task` — run a single task on the host without spinning up a VM, + for fast iteration during development +- `mirum run` — run the pipeline locally with the same VMs but with a local copy + of your code and a debugger +- `mirum ssh` — connect to a failed VM over SSH and debug on real hardware +- `mirum try` — run a build from local changes on the cluster without creating + throwaway commits + +## How + +The architecture is built on two key solutions: +[Virtual Machines](https://en.wikipedia.org/wiki/Virtualization#Hardware_virtualization) +and [Starlark](https://starlark-lang.org/). + +### Virtualization vs. Containers vs. Host + +Modern CI must provide a reproducible and hermetic environment for every task. +Despite the popularity of container isolation, it's a Linux-specific technology +with many limitations. As soon as you need Windows, a specific kernel version, +or even systemd, you're forced to revert to bare, stateful runners you've manually +configured. + +A solution was proposed in Sourcehut: use ephemeral VMs from a snapshot for isolated +builds. Unlike a container, a VM can run any guest system, can emulate inaccessible +architectures, provides a full stack including the kernel, and provides sufficient +isolation to allow a user to access the build machine via SSH. + +The idea is to split the runner into two layers: + +- mirum-agent, a highly portable statically linked C binary that can copy files, + execute bash commands, and collect logs and resources. +- mirum-worker, a full-fledged runtime that launches a disposable VM for each task, + launching and managing mirum-agent within the VM. + +This architecture allows for full support on official platforms, including isolation, +SSH access, and so on. On the other hand, if you're testing an exotic system +(for example, on bare metal without any OS at all), port mirum-agent and you'll +be able to connect to a regular mirum server. You can also offer specialized versions +of mirum-worker for containers/clouds/lambdas, or any custom environment. +Simply run mirum-agent in a sandbox and issue commands to it. + +Using VMs also significantly simplifies infrastructure. +One x64 host can run Linux, Windows, and *BSD, +while one arm64 mac mini can run macOS, Linux, and Windows on arm64. + +### Starlark + +Today, there are two ways to describe pipelines: statically in yaml or by writing +a script in a scripting language like JavaScript/Python/Ruby. + +Mirum occupies a niche between the two and offers Starlark as a configuration language. +Starlark is a specialized embedded programming language developed for the Bazel +build system. On the one hand, you have variable conditions and loops, imports, +objects, and arrays. + +Unlike yaml, you don't have to reinvent the wheel. Settings are variables, tasks +are functions, build matrices are loops, and the reusable actions library is +a simple import. Python syntax is well-known and doesn't require learning your DSL. + +On the other hand, it's not an algorithmically complete language. There are no side +effects, no need for a separate sandbox, and no need to drag in a runtime. The CI +server has total control over Starlark execution. But unlike Kotlin DSL (TeamCity), +Groovy (Jenkins), or Python (Buildbot), Starlark forbids side effects: no network, +no filesystem, no arbitrary imports. Eval is safe for untrusted code +(PRs from external contributors), deterministic, and cacheable. ## Installation diff --git a/REUSE.toml b/REUSE.toml index 2be7092..8cd1ea7 100644 --- a/REUSE.toml +++ b/REUSE.toml @@ -7,8 +7,6 @@ version = 1 path = [ ".mailmap", "CLA.md", - "CONCEPT.md", - "README.md", "VERSION", "cmd/mirum-server/web/*.json", "go.mod", @@ -19,3 +17,11 @@ path = [ ] SPDX-FileCopyrightText = "2026 Nikolay Govorov <me@govorov.online>" SPDX-License-Identifier = "AGPL-3.0-or-later" + +[[annotations]] +path = [ + "README.md", + "docs/**.md", +] +SPDX-FileCopyrightText = "2026 Nikolay Govorov <me@govorov.online>" +SPDX-License-Identifier = "CC-BY-4.0" diff --git a/docs/mirumfile.md b/docs/mirumfile.md new file mode 100644 --- /dev/null +++ b/docs/mirumfile.md @@ -0,0 +1,590 @@ +# Mirumfile + +> [!CAUTION] +> This document is the API reference for the `Mirumfile` format. The +> implementation is in progress and the runtime does not yet match this +> document. + +`Mirumfile` is a single Starlark file at the repository root. It defines +**tasks** (units of work, run on a host or in a VM) and **pipelines** +(graphs of tasks, scheduled across one or more VMs). The same file is +read by the local `mirum` CLI and by mirum-server in a cluster. + +## CLI + +``` +mirum task <name> [args...] Run a registered task. Positional args after + <name> become positional arguments to the + task function after tctx: + mirum task build linux amd64 + → build(tctx, "linux", "amd64") + +mirum run <pipeline> Run a registered pipeline. +``` + +## File format and discovery + +`Mirumfile` lives at the repository root. This is the only required file +for Mirum. + +The file is standard Starlark with the following predeclared globals: + +| Name | Kind | Purpose | +|-------------|---------------------|--------------------------------------| +| `task` | builtin function | Register a task | +| `pipeline` | builtin function | Register a pipeline | +| `fail` | standard Starlark | Abort with a message | +| `print` | standard Starlark | Write to the task log | +| `struct` | standard Starlark | Build anonymous records | + +Multi-file projects use standard Starlark `load()`: + +```python +load("//tasks/build.star", "build", "test") +load("//tasks/release.star", "package") +``` + +The mirum standard library is mounted under `@mirum//`: + +```python +load("@mirum//on.star", "git", "cron", "any_of") +load("@mirum//pkg.star", "install") +``` + +## Naming convention + +By convention, the first parameter of a function is named according to the +ctx kind it expects: + +- `tctx` for task functions and helpers that operate on a task ctx +- `pctx` for pipeline functions and helpers that operate on a pipeline ctx +- `event` for trigger predicates + +This convention is not enforced by the runner — it is just a readability +aid. A reader of `def lint(tctx):` immediately knows it expects a task +ctx. + +## Registration + +Tasks and pipelines are registered as **side effects** of top-level +calls, not as values bound to names. The function is defined first, then +registered: + +```python +def build(tctx): + """Build mirum-server""" + tctx.exec(["go", "build", "-o", "build/mirum-server", "./cmd/mirum-server"]) +task(build) + +def test(tctx): + """Run tests""" + tctx.need(build) + tctx.exec(["go", "test", "-race", "-count=1", "./..."]) +task(test) + +load("@mirum//on.star", "git") + +def ci(pctx): + """CI: build + test on Linux""" + pctx.run(test, image="mirum/ubuntu-24.04") +pipeline(ci, on=[git.push(branches=["main"])]) +``` + +The function being registered remains an ordinary callable. It can be +called directly (`build(tctx)`), passed as a value (`pctx.run(test, ...)`), +and referenced by `tctx.need`. Registration is metadata for the CLI and +the server — it does not wrap or replace the function. + +A function defined at the top level that is **not** registered is a +**helper**: invisible to `mirum list`, not invocable as `mirum task`, +no special calling convention. Helpers are just regular functions called +from registered ones. There is no `_` prefix convention. + +```python +def check_gofmt(tctx): # helper, not registered + r = tctx.exec(["gofmt", "-l", "."], check=False) + if r.stdout.strip(): + fail("gofmt: needs formatting:\n" + r.stdout) + +def lint(tctx): + """Run static checks""" + tctx.exec(["go", "vet", "./..."]) + check_gofmt(tctx) +task(lint) +``` + +### `task(fn, name=None)` + +Registers `fn` as a task. The default name is the Starlark function name; +the optional `name` keyword overrides it. Returns `None`. Re-registration +of the same name is a hard error at eval time. + +### `pipeline(fn, name=None, on=None)` + +Registers `fn` as a pipeline. Same naming rules as `task`. The `on` +parameter is a single predicate or a list of predicates that decide when +mirum-server triggers the pipeline (see [Triggers](#triggers)). Returns +`None`. + +## Triggers + +A trigger is a **predicate function**: it takes an event and returns a +bool. mirum-server runs each registered pipeline whose `on=` predicate +returns True for an incoming event. With a list, the pipeline runs if +**any** predicate matches (OR semantics). + +Predicates are ordinary Starlark functions, defined either by the user or +by the standard library. There is no separate trigger DSL. + +### Writing predicates by hand + +```python +def main_push(event): + return (event.kind == "git" + and event.type == "push" + and event.branch == "main") + +def go_changes(event): + if event.kind != "git": + return False + if event.type != "push" and event.type != "pull_request": + return False + return any([p.endswith(".go") or p == "go.mod" for p in event.paths]) + +def by_release_bot(event): + return (event.kind == "git" + and event.type == "push" + and event.author == "release-bot") + +def ci(pctx): + ... +pipeline(ci, on=[main_push, go_changes]) +``` + +### Standard library factories + +For common cases the `@mirum//on.star` stdlib provides predicate +factories — Starlark functions that return predicates. They are +themselves written in plain Starlark; users can fork the file or write +their own factories the same way. Factories for source-specific events +are grouped by source kind, exposed as structs: + +```python +load("@mirum//on.star", "git", "cron", "manual", "any_of", "all_of") + +def ci(pctx): + ... +pipeline(ci, on=[ + git.push(branches=["main"], paths=["**.go", "go.mod"]), + git.pull_request(branches=["main"]), +]) + +def release(pctx): + pctx.run(build_release, image="mirum/ubuntu-24.04") +pipeline(release, on=[git.tag(names=["v*"])]) + +def nightly(pctx): + ... +pipeline(nightly, on=[cron("0 6 * * *")]) +``` + +A factory is just a closure-returning function — there is nothing +special about it on the runtime side. A trimmed version of the stdlib's +`git.push`: + +```python +# @mirum//on.star +def _git_push(branches=None, paths=None): + def predicate(event): + if event.kind != "git" or event.type != "push": + return False + if branches and event.branch not in branches: + return False + if paths and not _any_glob(event.paths, paths): + return False + return True + return predicate + +git = struct( + push = _git_push, + tag = _git_tag, + pull_request = _git_pull_request, +) +``` + +When a non-git source is added — Perforce, Mercurial, Subversion — its +factories live in their own namespace alongside `git`: + +```python +load("@mirum//on.star", "git", "perforce") + +pipeline(ci, on=[ + git.push(branches=["main"]), + perforce.submit(branches=["//depot/main/..."]), +]) +``` + +User-written factories compose with stdlib ones identically. To require +that a build runs only when both `main` is pushed AND a release-bot is +the author, mix stdlib and hand-written predicates with `all_of`: + +```python +load("@mirum//on.star", "git", "all_of") + +def by_release_bot(event): + return (event.kind == "git" + and event.type == "push" + and event.author == "release-bot") + +def bot_release(pctx): + ... +pipeline(bot_release, on=[ + all_of(git.push(branches=["main"]), by_release_bot), +]) +``` + +### `EventCtx` + +The single argument passed to every predicate. Same value is also +available inside pipeline functions as `pctx.event`. EventCtx is +read-only. + +Only two fields are guaranteed on every event: + +| Field | Type | Meaning | +|-------------------|------|----------------------------------------| +| `event.kind` | str | Source family that produced the event: `"git"`, `"perforce"`, `"hg"`, `"cron"`, `"manual"`, `"s3"`, `"webhook"`, … | +| `event.timestamp` | int | Unix epoch seconds of when the event occurred | + +**Everything else depends on `event.kind`.** Each kind documents its +own fields. Source families that have multiple distinct event types +(e.g. `git` has push, tag, pull_request) also expose `event.type` as a +sub-discriminator. + +A predicate that wants to read kind-specific fields **must check +`event.kind` first** (and `event.type` if the kind has multiple types). +Accessing a field that does not exist for the current event raises an +error. Predicates that do not recognize a kind should return `False`, +not crash. New event kinds and types may be added at any time; existing +predicates that check for known kinds remain valid. + +**`kind == "manual"`** + +| Field | Type | Meaning | +|------------------|-------------|----------------------------------------| +| `event.user` | str | Identifier of the user who triggered | +| `event.reason` | str \| None | Optional reason supplied by the user | + +**`kind == "cron"`** + +| Field | Type | Meaning | +|------------------|------|---------------------------------------------| +| `event.schedule` | str | The cron expression that fired | + +**`kind == "git"`** + +Fields common to all git events: + +| Field | Type | Meaning | +|------------------|------|--------------------------------------------------| +| `event.type` | str | `"push"`, `"tag"`, or `"pull_request"` | +| `event.source` | str | Name of the source set | +| `event.commit` | str | Commit SHA | +| `event.author` | str | Author / committer / tagger / PR author | + +Additional fields by `event.type`: + +`type == "push"`: + +| Field | Type | Meaning | +|------------------|-----------|----------------------------------------| +| `event.branch` | str | Branch pushed to | +| `event.message` | str | Commit message | +| `event.paths` | list[str] | Paths changed by the push | + +`type == "tag"`: + +| Field | Type | Meaning | +|------------------|------|--------------------------------------------------| +| `event.tag` | str | Tag name | +| `event.message` | str | Tag annotation, if any | + +`type == "pull_request"`: + +| Field | Type | Meaning | +|------------------|-----------|----------------------------------------| +| `event.number` | int | PR number | +| `event.title` | str | PR title | +| `event.base` | str | Target branch | +| `event.head` | str | Source branch | +| `event.draft` | bool | Draft state | +| `event.labels` | list[str] | Labels currently applied | +| `event.paths` | list[str] | Paths changed in the PR | + +Additional event kinds (other VCS systems, S3 object events, generic +webhooks, file watchers, external systems) bring their own field sets +and are documented as they are added. Existing predicates are +unaffected: they match the kinds they know and return `False` for +everything else. + +## TaskCtx + +The argument passed to every task function. The only ctx that can +execute commands. + +### Execution + +``` +tctx.shell(script, **opts) -> Result +tctx.exec(argv, **opts) -> Result +``` + +`shell` runs `script` through a POSIX/bash interpreter with full support +for pipes, redirects, command substitution, here-docs, variable expansion, +and control flow. Same shell semantics on every platform; no `/bin/bash` +dependency. + +`exec` runs a single command without shell parsing. Use it when you have +an argv list and want zero shell interpretation. + +Common keyword options for both: + +| Option | Default | Meaning | +|------------|-------------|-----------------------------------------------------------| +| `env` | `{}` | Additional environment variables (added to `tctx.env`) | +| `cwd` | `None` | Working directory (relative to current `tctx.cwd`) | +| `stdin` | `None` | None / str / bytes / path | +| `stdout` | `None` | None (capture+stream) / path / `DEVNULL` | +| `stderr` | `None` | None (capture+stream) / path / `DEVNULL` / `STDOUT` | +| `append` | `False` | Open `stdout` / `stderr` paths in append mode | +| `timeout` | `None` | Seconds; kill on expiry | +| `check` | `True` | Non-zero exit raises `fail()` automatically | +| `capture` | `True` | Populate `Result.stdout` / `Result.stderr` | + +`Result` is a struct value with attributes: + +```python +result.stdout # str +result.stderr # str +result.code # int +result.ok # bool (code == 0) +``` + +`check=True` is the default — non-zero exits abort the task. Tasks that +want to inspect the exit code use `check=False`: + +```python +r = tctx.exec(["test", "-f", path], check=False) +if r.ok: + ... +``` + +### Immutable derive + +``` +tctx.with_env({"K": "V"}) -> tctx +tctx.with_cwd("subdir") -> tctx +``` + +Both return a new ctx with the modification applied. The original ctx is +unchanged. + +### Introspection + +``` +tctx.cwd # str — current working directory (absolute) +tctx.env # dict-like — current environment +tctx.os # "linux" | "darwin" | "freebsd" | "windows" | ... +tctx.arch # "amd64" | "arm64" | "riscv64" | ... +``` + +### `tctx.need(fn, *args, **kwargs) -> Result | None` + +Runs `fn(tctx, *args, **kwargs)` if it has not already been called with +the same arguments in this invocation; otherwise returns the cached +result. Use it for "make sure this happened" semantics; use a direct +call (`build(tctx)`) for "definitely run this now" semantics. + +Dedup key is `(fn, args, kwargs)`. Different arguments to the same +function are different invocations: + +```python +def build(tctx, goos="linux", goarch="amd64"): + ... + +def all_platforms(tctx): + for goos, goarch in [("linux", "amd64"), ("darwin", "arm64")]: + tctx.need(build, goos, goarch) # two distinct invocations +``` + +### `tctx.checkout(name="main", ref=None) -> str` + +Materializes a source set inside the task's environment and returns +the absolute path to it. The runtime is responsible for fetching the +right files; from the task's perspective the call is idempotent — +calling it again returns the same path without doing extra work. + +When `ref` is omitted, the runtime picks the natural ref for the +triggering event: + +| Event | Default ref | +|-----------------------------|---------------------------| +| `git push` | The pushed commit | +| `git tag` | The tagged commit | +| `git pull_request` | The PR head | +| `cron` | Default branch | +| `manual` / `mirum task` | Current working tree | + +To use a different ref explicitly, pass `ref=` (a SHA, branch, or tag +name). Pipelines that need to override the default usually do so by +forwarding event data through `args`: + +```python +def build_at(tctx, ref): + src = tctx.checkout(ref=ref) + ... +task(build_at) + +def replay(pctx): + pctx.run(build_at, image="...", args={"ref": pctx.event.commit}) +``` + +Multiple source sets are accessed by name: + +```python +def integration(tctx): + src = tctx.checkout() # default source set + fixtures = tctx.checkout("fixtures") # named source set + tctx.with_env({"FIXTURES": fixtures}).exec(["go", "test", "./tests/integration/..."]) +``` + +Source set names are defined in [Source sets](#source-sets), not in +pipeline or task code. A task that needs source files should call +`tctx.checkout()` explicitly — both as documentation of intent and +because some workers may stage source lazily on first call. + +### `tctx.upload(path, artifact="name")` and `tctx.download(name, dest=".")` + +`upload` registers a file as a named artifact under the current task. +`download` retrieves a previously uploaded artifact by name into a +destination directory. + +Locally, artifacts are tracked in `.mirum/local-artifacts.json` in the +repo root and persist between `mirum` invocations. In the cluster they +move through mirum-server. The task functions are unchanged either way. + +`download` of an artifact that was never uploaded fails with a clear +error. + +## PipelineCtx + +The argument passed to every pipeline function. **Cannot execute shell +commands.** Pipeline code may come from untrusted PRs; it is restricted +to orchestration. + +### `pctx.event` + +Read-only [`EventCtx`](#eventctx) for the event that triggered this +pipeline. The same value that was passed to the trigger predicates. + +When `mirum run` is invoked locally, the event is synthetic +(`{"kind": "manual"}` by default; overridable via CLI flags). + +### `pctx.run(task_fn, image=, args={}, setup=None, depends=[]) -> handle` + +Dispatches a task into a new VM. Returns a handle that can be passed as +`depends=[handle, ...]` to subsequent `pctx.run` calls. + +| Argument | Meaning | +|------------|--------------------------------------------------------------------------| +| `task_fn` | Task function to invoke (the function value, not the registered name) | +| `image` | OCI reference to the VM image | +| `args` | Keyword arguments forwarded to `task_fn(tctx, **args)` | +| `setup` | Optional setup function for cached snapshots | +| `depends` | Handles from prior `pctx.run` calls. This VM does not start until they finish, and inherits their `upload`'d artifacts | + +Source materialization is a task-side concern, not a pipeline-side one +— see [`tctx.checkout`](#tctxcheckoutnamemain---str). The pipeline does +not pass source refs to its tasks; each task asks for the source sets +it needs by name, and the runner resolves them based on the triggering +event. + +`depends=` expresses **VM topology**, not task dependency. Within a +single VM, task functions compose via `tctx.need(...)`. Across VMs, the +pipeline orchestrates via `pctx.run(..., depends=[...])`. These are two +distinct mechanisms and are not interchangeable. + +Example: cross-platform release that fans out builds and fans in publish. + +```python +load("@mirum//on.star", "git") + +PLATFORMS = [ + ("linux", "amd64"), ("linux", "arm64"), + ("darwin", "arm64"), + ("windows", "amd64"), + ("freebsd", "amd64"), +] + +IMAGES = { + "linux": "mirum/ubuntu-24.04", + "darwin": "mirum/macos-15", + "windows": "mirum/windows-2025", + "freebsd": "mirum/freebsd-14", +} + +def release(pctx): + builds = [] + for goos, goarch in PLATFORMS: + builds.append(pctx.run(build, + image=IMAGES[goos], + args={"goos": goos, "goarch": goarch})) + + pctx.run(publish, image=IMAGES["linux"], depends=builds) +pipeline(release, on=[git.tag(names=["v*"])]) +``` + +## Source sets + +Source URLs, refs, auth, and repo locations are **configuration**. +Pipeline and task code reference source sets only by **name**. + +In a cluster, source sets are configured per project in mirum-server's +WebUI. The server resolves names to URLs and credentials when staging a +VM, and credentials never reach the VM or the task code. + +Locally, source sets are defined in `.mirum/sources.json` in the repo +root: + +```json +{ + "main": ".", + "fixtures": "/home/user/work/test-fixtures" +} +``` + +Values are absolute paths to local checkouts. The `main` entry can be +omitted; it defaults to the directory containing `Mirumfile`. The file +is opt-in for git tracking — projects with shared layout may commit it, +projects with user-specific paths typically gitignore it. + +A `tctx.checkout("name")` for a name not in the configuration is a hard +error. + +## Safe shell interpolation + +Dynamic values are passed through the `env=` keyword and referenced as +quoted shell variables in the script body. This gives POSIX single-word +expansion semantics — the value becomes one literal argument and is +never re-parsed: + +```python +# Even if branch == "; rm -rf /", this is safe — it is one literal arg +tctx.shell('git log --format=%H "$BRANCH"', env={"BRANCH": branch}) +``` + +Never build shell scripts by string concatenation. Always pass dynamic +values through `env=`. + +`tctx.exec(argv, ...)` bypasses the shell entirely; use it when you +already have an argv list and want zero chance of shell interpretation. diff --git a/docs/whitepaper.md b/docs/whitepaper.md new file mode 100644 --- /dev/null +++ b/docs/whitepaper.md @@ -0,0 +1,594 @@ +# Mirum Whitepaper + +## Modules + +Four executable modules: + +``` +┌──────────────────────────────────────────────────────┐ +│ mirum-server │ +│ watch registry, task queue, log aggregation, WebUI │ +└─────────────────────────────────────────────────────┘ + ↑ gRPC (worker-initiated) ↑ gRPC + │ │ +┌────────┴──────────┐ ┌─────────┴─────────┐ +│ mirum-worker │ │ mirum-worker │ +│ Linux (KVM) │ │ macOS (Vz) │ +│ │ vsock │ │ │ vsock │ +│ ↓ │ │ ↓ │ +│ ┌────────────┐ │ │ ┌────────────┐ │ +│ │mirum-agent │ │ │ │mirum-agent │ │ +│ │ inside VM │ │ │ │ inside VM │ │ +│ └────────────┘ │ │ └────────────┘ │ +└───────────────────┘ └───────────────────┘ + +┌───────────────────┐ +│ mirum (CLI) │ +│ Starlark eval, │ +│ spawns worker │ +└───────────────────┘ +``` + +**mirum-server** — the orchestrator. +Contains a database, stores users, organizations, project and pipeline settings, +provides a WebUI and API, responds to webhooks, and distributes tasks to workers. +Collects logs and build results from workers. + +It's a control plane. + +**mirum-worker** — the task executor +Connects to the server via outbound gRPC, declares its capabilities, +and picks tasks from the queue that it can execute. + +Evaluates pipeline Starlark (coroutine model), starts a VM for each +task, runs the task function inside, and returns the results to the +server. If a pipeline yields on a task result it is not yet ready for, +the worker leaves the suspended coroutine in the queue and resumes it +when the awaited task completes. + +It's a data plane. _Only the worker has access to the user's secrets and code._ + +Workers come in different kinds: KVM worker for local VMs, +macOS worker with Vz.framework, windows worker with Hyper-V, +EC2 worker for cloud VMs, host worker for direct execution. +A new worker type joins the cluster and starts picking up tasks with no changes +to the server. + +**mirum-agent** — static binary pre-installed in every VM. + +A tiny bridge between the worker on the host and the tasks running in the VM. +Written in C99, no dependencies, posix-only. Implements a simple TLV for communicating +with the host (via virtio-vsock or tcp socket, depending on the hypervisor). + +Channels: control, stdin, stdout, stderr, file transfers, interactive shell sessions. + +Small and simple enough to be auditable. +This component lives close to the actual code, making it highly security-sensitive. + +**mirum (CLI)** — developer tool. +Contains a Starlark runtime for local eval and embeds a host worker for +in-process execution. Takes the server's role locally. + +Technically, it is a lightweight disposable server that runs a real worker locally +to replicate real-world conditions in a cluster as closely as possible. + +- `mirum task <name>`: run a single registered task on the host, no VM. +- `mirum run <pipeline>`: eval pipeline → spawn worker → dispatch tasks → display logs. +- `mirum list`: list registered tasks and pipelines. +- `mirum try <pipeline>`: send a local diff to the server. +- `mirum ssh <vm>`: shell into a failed VM. +- `mirum eval <pipeline>`: show the DAG without running anything. + +## Configuration Model + +Mirum operates on projects, a project consists of pipelines, +pipelines consist of tasks. A task is the minimum unit of execution — it always +runs in a single VM. Pipelines are DAGs (directed acyclic graphs) that invoke multiple +tasks and pass state between them. A project tracks a list of pipelines, their trigger +rules (watch, cron, manual, ...), and result notifications. + +A quick example, all in one starlark file: + +```python +# /Mirumfile — the only file required + +load("@mirum//on.star", "git") + +# A single pipeline describes a matrix of multiple operating systems and architectures. +# A single Linux host with KVM serves Linux, Windows, and BSD guests. +# A macOS host serves macOS, Linux, and BSD. +# Tasks adapt to the platform via `tctx.os` — they don't choose it. + +# pctx.run takes an optional setup function to indicate which steps +# are environment setup only, so the resulting image can be cached +def setup(tctx): + tctx.shell("apt-get update && apt-get install -y cargo") + +def build(tctx): + tctx.checkout() + tctx.shell("cargo build --release") + tctx.upload("target/release/myapp", artifact="bin") +task(build) + +def test(tctx): + # test knows nothing about build — only that it needs an artifact. + # that artifact could have been built right here, come from cache + # or a registry, or even uploaded from a developer's laptop + tctx.download("bin", dest=".") + tctx.shell("cargo test") +task(test) + +# A pipeline is a function that dispatches tasks. It is registered the +# same way tasks are, with a list of triggers. +def ci(pctx): + b = pctx.run(build, setup=setup, image="mirum/ubuntu-24.04") + pctx.run(test, setup=setup, depends=b, image="mirum/ubuntu-24.04") +pipeline(ci, on=[git.push(branches=["main"])]) +``` + +### Functions All the Way Down + +Mirum configuration is nested function composition in Starlark. There +are two kinds of registered things — **tasks** and **pipelines** — and +both are registered as side effects of top-level calls in the file: + +```python +def build(tctx): + ... +task(build) + +def ci(pctx): + ... +pipeline(ci, on=[git.push(branches=["main"])]) +``` + +A pipeline is a function that imperatively dispatches tasks and passes +dependencies between them. A task's result can be used to launch further +tasks, enabling dynamic task creation. There is no separate syntax for +"allow failure / retry / skip_if / matrix" — it is just `if`/`for` in +Starlark. + +``` +Mirumfile → the file the server looks for (one per repo, at root) + +triggers → event routing "when to run" + (predicates passed via pipeline(..., on=[...])) + +pipeline(pctx) → DAG of tasks "what to run, on which platforms" + +task(tctx) → scripts + artifacts "how to build" +``` + +Starlark supports imports, so splitting a large project works out of the +box. The only requirement is a `Mirumfile` at the repository root. + +The server reads `Mirumfile` (and any files it transitively `load()`s) +via the forge contents API, not via git clone. This allows: fetching +only configuration files without accessing source code, filtering +webhooks (push with no changes in `Mirumfile`'s closure → skip re-eval), +caching configuration by file SHAs. + +To reuse code both within and outside the project, Starlark `load` is used. + +``` +@mirum// Standard library (triggers, services, apt, bazel helpers) +@pkg// External packages (from deps.star, pinned by commit) +// Local files (relative to repo root) +``` + +### Coroutine Eval: Dynamic DAGs + +Pipeline functions execute as coroutines. `pctx.run()` without accessing +results is non-blocking — the server accumulates pending tasks. +Accessing a result (`.output()`) is a yield point: the server dispatches +all pending tasks, waits for the needed result, and resumes the pipeline +function. + +```python +def build(tctx): + ... +task(build) + +def discover_tests(tctx): + ... +task(discover_tests) + +def run_test(tctx, module): + ... +task(run_test) + +def ci(pctx): + # All pctx.run() calls before the first .output() accumulate and run in parallel + builds = {} + for os in ["linux", "mac"]: + builds[os] = pctx.run(build, + image="mirum/%s" % os, args={"os": os}) + + discovery = pctx.run(discover_tests, image="mirum/linux") + + # Yield: server dispatches builds + discovery in parallel, + # waits for discovery to complete, resumes with result + test_modules = discovery.output("modules") + + # Dynamic phase: use runtime data + for module in test_modules: + pctx.run(run_test, args={"module": module}, + depends=list(builds.values())) +pipeline(ci, on=[git.push(branches=["main"])]) +``` + +If a pipeline function never calls `.output()`, the entire DAG is built +in a single pass. A static DAG is a special case of the dynamic model. + +`mirum eval` executes the pipeline function locally without dispatching +tasks. For static DAGs it prints the full graph. For dynamic DAGs — +everything up to the first yield point, marked "depends on runtime data +beyond this point." + +### Example: Everything in One File + +```python +# /Mirumfile — complete CI + +load("@mirum//on.star", "git") + +# pctx.run takes an optional setup function to indicate which steps +# are environment setup only, so the resulting image can be cached +def setup(tctx): + tctx.shell("apt-get update && apt-get install -y cargo") + +def build(tctx): + tctx.checkout() + tctx.shell("cargo build --release") + tctx.upload("target/release/myapp", artifact="bin") +task(build) + +def test(tctx): + # test knows nothing about build — only that it needs an artifact. + # that artifact could have been built right here, come from cache + # or a registry, or even uploaded from a developer's laptop + tctx.download("bin", dest=".") + tctx.shell("cargo test") +task(test) + +def ci(pctx): + b = pctx.run(build, setup=setup, image="mirum/ubuntu-24.04") + pctx.run(test, setup=setup, depends=b, image="mirum/ubuntu-24.04") +pipeline(ci, on=[git.push(branches=["main"])]) +``` + +### Example: Cross-Platform Project + +```python +# tasks/setup.star +def cpp_toolchain(tctx): + if tctx.os == "linux": + tctx.shell("apt-get update && apt-get install -y cmake ninja-build") + elif tctx.os == "freebsd": + tctx.shell("pkg install -y cmake ninja") + elif tctx.os == "windows": + tctx.shell("choco install -y cmake ninja visualstudio2022-workload-vctools") + elif tctx.os == "macos": + tctx.shell("brew install cmake ninja") + +# tasks/build.star +def build(tctx): + tctx.checkout() + if tctx.os == "windows": + tctx.shell('cmake -G "Visual Studio 17 2022" -B build .') + elif tctx.os == "macos": + tctx.shell("cmake -B build -DCMAKE_OSX_DEPLOYMENT_TARGET=12.0 .") + else: + tctx.shell("cmake -B build .") + tctx.shell("cmake --build build --config Release") + tctx.upload("build/out/*", artifact="pkg") +task(build) + +def publish(tctx): + tctx.download("pkg", dest="release/") + tctx.shell('gh release create "$TAG" release/* --generate-notes') +task(publish) + +# /Mirumfile +load("@mirum//on.star", "git") +load("//tasks/setup.star", "cpp_toolchain") +load("//tasks/build.star", "build", "publish") + +IMAGES = { + "linux": "mirum/ubuntu-24.04", + "freebsd": "mirum/freebsd-14", + "windows": "mirum/windows-2025", + "macos": "mirum/macos-15", +} + +# Irregular matrix — just a list. No exclude needed. +PLATFORMS = [ + ("linux", "amd64"), + ("linux", "arm64"), + ("macos", "arm64"), + ("windows", "amd64"), + ("freebsd", "amd64"), +] + +def release(pctx): + # Build: loop over platforms, each task gets its own handle + builds = {} + for os, arch in PLATFORMS: + builds[(os, arch)] = pctx.run(build, + setup=cpp_toolchain, + image=IMAGES[os], + args={"os": os, "arch": arch}) + + # Publish: fan-in, waits for all builds + pctx.run(publish, depends=list(builds.values())) +pipeline(release, on=[git.tag(names=["v*"])]) +``` + +The pipeline decides WHERE (image, platforms). Setup decides WITH WHAT +(toolchain, cached snapshot). The task decides HOW (checkout, build, upload). +Tasks don't know what platform they're running on — `tctx.os` and `tctx.arch` +are injected by the pipeline. + +## Images + +The most tedious and time-consuming task is preparing images for various operating +systems. Some distributions distribute qcow2, some support cloud-init, and some only +offer an ISO installer. Some require a network connection for configuration, while +others work offline. + +Another problem is distribution. The reason containers are popular is the OCI registry. +A container is easy to upload to a server, and just as easy to download and deploy. +Nothing similar exists for VMs. + +An elegant solution was found in Tart by CirrusCI: use OCI as a black box for storing +the VM image. Load it into your existing infrastructure, easily update, and distribute. +A single distribution format for all platforms. Images are stored as compressed +raw disk chunks: + +``` +OCI Image Manifest: + config: + mediaType: "application/vnd.mirum.image.config.v1+json" + { mirum.version, os, arch, distro, distro_version, + agent_version, disk_size, chunk_size } + + layers: + - mediaType: "application/vnd.mirum.disk.raw.v1+zstd" + annotations: { "mirum.offset": "0", "mirum.length": "67108864" } + - ... + + # macOS additionally: + - mediaType: "application/vnd.mirum.aux.v1+zstd" + - mediaType: "application/vnd.mirum.hwmodel.v1+json" +``` + +Each chunk is independently zstd-compressed. The worker downloads and decompresses +in parallel. 64MB chunks for a 4GB disk ≈ 64 layers. On pull worker reassembles +raw disk from chunks → converts to hypervisor format +(qcow2, vhdx, Vz native) → caches → CoW clone per task. + +### Three Layers (VM Runtime) + +VM images are larger than container images, but there are fewer of them. +We can borrow the layer caching idea and apply it to image snapshots (like in qcow2+). + +``` +Layer 0: Base image (from OCI registry) + Golden (Mirum-maintained) or organization (external). + Worker downloads, converts to hypervisor format, caches locally. + +Layer 1: Setup (function from pctx.run(setup=...)) + Declared in the pipeline. Worker executes, takes a snapshot. + Cache is local, best-effort, evicted by LRU. + +Layer 2: Ephemeral overlay + CoW clone of setup cache (or base). Per-task. Destroyed. +``` + +### Setup as a Function + +The pipeline passes two functions to `pctx.run()`: +`setup` (optional) and the main task. The image is also specified in the pipeline: + +```python +def cpp_setup(tctx): + if tctx.os == "linux": + tctx.shell("apt-get update && apt-get install -y cmake ninja-build") + elif tctx.os == "freebsd": + tctx.shell("pkg install -y cmake ninja") + +def build(tctx): + tctx.checkout() + tctx.shell("cmake -B build . && cmake --build build") +task(build) + +def ci(pctx): + pctx.run(build, setup=cpp_setup, + image="mirum/ubuntu-24.04", + args={"os": "linux"}) +pipeline(ci, on=[git.push(branches=["main"])]) +``` + +The worker hashes `(image_digest, setup_function_hash, os, arch)`. +Cache hit → CoW clone, boot in milliseconds. Miss → boot base, run setup, +snapshot, cache. + +Setup is an ordinary Starlark function, composable via `load()`: + +```python +# @pkg//acme/setup.star +def cpp_toolchain(tctx): + if tctx.os == "linux": + tctx.shell("apt-get update && apt-get install -y cmake ninja-build") + elif tctx.os == "windows": + tctx.shell("choco install -y cmake ninja") +``` + +```python +load("@pkg//acme/setup.star", "cpp_toolchain") + +def ci(pctx): + for os in ["linux", "windows"]: + pctx.run(build, setup=cpp_toolchain, + image=IMAGES[os], args={"os": os}) +``` + +One `cpp_toolchain` across the entire organization — one hash — one snapshot per worker. + +Three levels, cleanly separated: + +- **Pipeline**: WHERE (image, platforms) +- **Setup**: WITH WHAT (toolchain, dependencies — cached snapshot) +- **Task**: HOW (checkout, build, test, upload) + +Organizations that need a fully pre-built image can publish it to any OCI registry +using external tooling and reference it directly: + +```python +# no need for setup with a preconfigured image +pctx.run(build, image="acme-registry.com/ci-base:latest") +``` + +### Golden Images + +Golden images are built by the Mirum team using Packer, not by users: + +| Platform | Packer builder | Install method | +| ------------------------- | ---------------------- | ----------------------------------- | +| Linux (Ubuntu, Fedora...) | `qemu` | cloud-init / preseed / kickstart | +| macOS | `tart` (Packer plugin) | VZMacOSInstaller + VNC boot_command | +| Windows | `qemu` | autounattend.xml (evaluation ISO) | +| NetBSD | `qemu` | sysinst auto | +| FreeBSD | `qemu` | bsdinstall scripted | +| OpenBSD | `qemu` | autoinstall response file | + +Windows: evaluation ISO is freely downloadable. The evaluation period (180 days) +is irrelevant for ephemeral CI VMs. Users activate with their own key if needed. + +macOS: `.ipsw` installed via Virtualization.framework. Setup Assistant automated +via VNC keystroke injection. Requires Apple hardware for building and running. + +## Extensibility + +Starlark simplifies building plugins and libraries. You already have `load`, +so you don't need to invent your own systems like reusable actions. + +**Transparent worker optimizations** — invisible to the task. +Configured in `worker.yaml`. The worker configures the VM environment before running +any scripts: apt mirror, cargo/npm cache mount, HTTP proxy. +`./ci.sh` with `apt-get install` inside simply runs faster. +Bash scripts speed up for free. + +```yaml +# worker.yaml +optimizations: + apt_mirror: "http://apt-cache.internal:3142" + http_proxy: "http://squid.internal:3128" + cargo_cache: "/mnt/shared/cargo" +``` + +**Starlark stdlib (@mirum//)** — for things that require an explicit decision. +This is a standard Starlark that, although it comes with an agent, +doesn't require any additional APIs (see the Bazel vs Buck configurations). +Users can read, fork, or write their own. + +Starlark sees capabilities via `tctx.worker.has("docker")`. +The stdlib adapts. No hidden magic — the code is readable. + +The dividing principle: if an optimization can be applied without changing +task behavior — it's a transparent worker optimization. If the task needs to know +(e.g. a postgres address) — it's Starlark stdlib with graceful degradation. + +Checks worker capabilities and adapts: + +```python +load("@mirum//services", "service") + +def test(tctx): + # docker on the worker? → sidecar container + # no docker? → install and run inside the VM + service(tctx, "postgres", image="postgres:16", port=5432) + tctx.shell("make test") +task(test) +``` + +Workers declare capabilities on registration: + +```yaml +# worker.yaml +capabilities: + kvm: true + gpu: false + docker: true +``` + +**Server plugins (traits)** — extend the platform. Implement trait interfaces. +Configured in `server.yaml`: + +| Trait | AGPL built-in | External plugin | +| ---------------- | ------------- | ---------------------- | +| AuthBackend | Token, basic | SAML, OIDC, LDAP | +| Source provider | Git, VSC, s3 | Mercurial, Perforse | +| SecretProvider | Env, systemd | Vault, AWS KMS | +| NotificationSink | — | Slack, email, webhooks | +| BillingHook | Noop | Usage metering | + +## Comparison + +We were inspired by many wonderful tools +- Buildbot: centralized master, property model, `try` for pre-commit testing, dynamic build steps; +- TeamCity: vsc roots, multi-tenant, role model, breadth of tool support; +- Concourse: the idea of universal input/output; +- SourceHut: SSH into VMs for debugging, BSD support; +- Cirrus CI: ephemeral VMs, bring-your-own cloud, Starlark, agent inside VM; +- GitHub Actions: how not to do it. + +### vs GitHub Actions + +Paid, closed, inseparable from Microsoft, only ubuntu/windows/macOS, +only x64/arm64, nodejs required in runtime, +yaml configs (and only for the current repository). +No cross-OS matrices out of the box (macOS runners are a paid add-on). +No local execution. No dynamic DAG. Caching is an action, not a primitive. + +### vs GitLab CI + +GitLab CI is part of GitLab. YAML. DAG via `needs:` — a hack on top of a stage-based model. +Runners are stateful machines or Docker. No VM isolation. `include:` / YAML anchors — fragile reuse. +Dynamic child pipelines — via YAML generation. + +### vs Jenkins + +Groovy DSL is powerful but allows arbitrary code (RCE when eval'ing PRs). +Plugin ecosystem is huge but fragile (Security Advisories every month). +Agents are stateful, workspace persists. Shared Libraries are Groovy classes, trusted/untrusted. + +### vs TeamCity + +Powerful and popular, with clever ideas (dedicated VCS root, for example). +But it's paid and expensive, closed-source, and difficult to use. +Kotlin DSL is typed with IDE support, but allows side effects (HTTP, filesystem). +Agents are stateful and require maintenance. Snapshot dependencies equal our source consistency. + +Templates (1:1) vs our `load()` (N:N) — Starlark is strictly more powerful. + +### vs Buildbot + +The most portable and flexible of all. However, it's outdated, difficult to configure, +and designed for hosted builds. Its architecture doesn't support SaaS. +Pure Python configurations on both the master and agents. No IaC out of the box. + +### vs Cirrus CI + +Great tool, but unfortunately still closed source and dependent on gcloud. +Starlark is only available as an advanced mode with yaml. +It lacks support for many BSDs (but FreeBSD is available!). + +## Licensing + +**AGPL-3.0** — all four modules, all built-in traits, all runtimes, standard library, +full CLI, basic Web UI, SQLite, single-tenant auth. + +**Commercial license** — For companies unwilling to use the AGPL, offer a commercial +license, certifications, and SLAs in SaaS. Don't hesitate to take money +from enterprises and spend it on open source. |