diff --git a/docs/designs/compass/decisions/server/DL-539.md b/docs/designs/compass/decisions/server/DL-539.md new file mode 100644 index 0000000..2283295 --- /dev/null +++ b/docs/designs/compass/decisions/server/DL-539.md @@ -0,0 +1,6 @@ +--- +id: DL-539 +decision: "Each stack child that runs as a container becomes its own pod: the Server is a one-replica Recreate Deployment with exactly one Runner pod per Server, and NATS is a one-replica StatefulSet; Postgres and the OTel collector are operator-provided outside the renderer, and no LLM gateway is deployed." +status: "Active (Matt, 2026-10-17)" +record: ../../server-on-kubernetes.md#design +--- diff --git a/docs/designs/compass/decisions/server/DL-540.md b/docs/designs/compass/decisions/server/DL-540.md new file mode 100644 index 0000000..336f86a --- /dev/null +++ b/docs/designs/compass/decisions/server/DL-540.md @@ -0,0 +1,6 @@ +--- +id: DL-540 +decision: "ghcr.io/rigelbuild/compass-server is built from the .#compass-server nix closure on a digest-pinned distroless static-debian12 base, runs as 65532:65532, is published by digest through the shared --lane runner|server publish and retag scripts, and is gated by SERVER_IMAGE_CLOSURE_PATHS." +status: "Active (Matt, 2026-10-17)" +record: ../../server-on-kubernetes.md#design +--- diff --git a/docs/designs/compass/decisions/server/DL-541.md b/docs/designs/compass/decisions/server/DL-541.md new file mode 100644 index 0000000..c49fbb2 --- /dev/null +++ b/docs/designs/compass/decisions/server/DL-541.md @@ -0,0 +1,6 @@ +--- +id: DL-541 +decision: "Migrations stay in Server boot with no migration Job, and the renderer never renders Postgres; the operator brings a Postgres reached through a direct session-mode endpoint whose login role owns the database and is either a CREATEROLE plus BYPASSRLS role on PostgreSQL 16 or later or a superuser on any version, and that answers at boot." +status: "Active (Matt, 2026-10-17)" +record: ../../server-on-kubernetes.md#design +--- diff --git a/docs/designs/compass/decisions/server/DL-542.md b/docs/designs/compass/decisions/server/DL-542.md new file mode 100644 index 0000000..3799f7a --- /dev/null +++ b/docs/designs/compass/decisions/server/DL-542.md @@ -0,0 +1,6 @@ +--- +id: DL-542 +decision: "The Server renderer names Secrets but holds no values; Server secrets are read through dotenv:// from a read-only mounted Secret file whose master key never changes, forge App ids and secret names render as --forge-* args, XDG_STATE_HOME and the state dir sit on an emptyDir, and the admin token Secret mounts directly at 0440 under pod fsGroup 65532 with no init container, which the Server accepts once it follows the volume symlink and allows group-read for a group it holds." +status: "Active (Matt, 2026-10-17)" +record: ../../server-on-kubernetes.md#design +--- diff --git a/docs/designs/compass/decisions/server/DL-543.md b/docs/designs/compass/decisions/server/DL-543.md new file mode 100644 index 0000000..7059770 --- /dev/null +++ b/docs/designs/compass/decisions/server/DL-543.md @@ -0,0 +1,6 @@ +--- +id: DL-543 +decision: "In-cluster NATS requires a user and password carried in COMPASS_NATS_URL, and a NetworkPolicy admits port 4222 only from the Server pod." +status: "Active (Matt, 2026-10-17)" +record: ../../server-on-kubernetes.md#design +--- diff --git a/docs/designs/compass/decisions/server/DL-544.md b/docs/designs/compass/decisions/server/DL-544.md new file mode 100644 index 0000000..2e4545c --- /dev/null +++ b/docs/designs/compass/decisions/server/DL-544.md @@ -0,0 +1,6 @@ +--- +id: DL-544 +decision: "The Server terminates its own TLS from an operator-supplied kubernetes.io/tls Secret and loads it once, so a renewed certificate takes effect on restart; no ingress terminates TLS, serverAddr is the in-cluster Service, external access is TCP passthrough, and a private CA reaches the Runner through serverCA and clients through --ca." +status: "Active (Matt, 2026-10-17)" +record: ../../server-on-kubernetes.md#design +--- diff --git a/docs/designs/compass/decisions/server/DL-545.md b/docs/designs/compass/decisions/server/DL-545.md new file mode 100644 index 0000000..236a95a --- /dev/null +++ b/docs/designs/compass/decisions/server/DL-545.md @@ -0,0 +1,6 @@ +--- +id: DL-545 +decision: "A Runner-cluster entry whose issuer serves no anonymous discovery uses a jwksFile rendered from the cluster's public keys, so no Go change is needed; the rendered Runner gets serverCA, an optional serverName that sets the Runner's TLS server name through --server-name so a certificate naming only the public host verifies against the in-cluster serverAddr, and an egressAllow list rendered as --egress-allow, and its node exposes KVM as the kvmResourceName extended resource, stages the seccompProfilePath profile, and supports hostUsers: false; the Runner's microVM volume root is the volumes subdirectory of the session-volume hostPath mount, and the operator puts the project quota on that subdirectory, never on the hostPath itself, so the quota check has an unprojected reference above it." +status: "Active (Matt, 2026-10-17)" +record: ../../server-on-kubernetes.md#design +--- diff --git a/docs/designs/compass/decisions/server/DL-546.md b/docs/designs/compass/decisions/server/DL-546.md new file mode 100644 index 0000000..0e57d65 --- /dev/null +++ b/docs/designs/compass/decisions/server/DL-546.md @@ -0,0 +1,6 @@ +--- +id: DL-546 +decision: "Public compass holds tools/server-manifests/ and tools/runner-manifests/ while the operator holds values, Secrets, certificates, node config and apply; the head commit of every main push tags both the Server and Runner images :main--, where n is its first-parent count and each tag names that lane's newest publish at or before the commit, so a numerical image-automation policy on n bumps both images together with no operator bump job, and the automation writes only the digest, never the tag, into a pod spec." +status: "Active (Matt, 2026-10-17)" +record: ../../server-on-kubernetes.md#design +--- diff --git a/docs/designs/compass/decisions/server/DL-547.md b/docs/designs/compass/decisions/server/DL-547.md new file mode 100644 index 0000000..a069388 --- /dev/null +++ b/docs/designs/compass/decisions/server/DL-547.md @@ -0,0 +1,6 @@ +--- +id: DL-547 +decision: "Every Compass image (agent, runner, server, guest image, gateway) is public, renderers emit no imagePullSecrets, every registry read in the publish, index and release tooling is anonymous and credentials are used only for the write destination, except the pre-push probe of the tag a push is about to write, which is part of the write and uses the login file because GHCR answers an anonymous probe of a package that does not exist yet with 403, and each publish job fails when its package is not public, because no GitHub API sets package visibility." +status: "Active (Matt, 2026-10-17)" +record: ../../server-on-kubernetes.md#design +--- diff --git a/docs/designs/compass/decisions/server/DL-560.md b/docs/designs/compass/decisions/server/DL-560.md new file mode 100644 index 0000000..201636f --- /dev/null +++ b/docs/designs/compass/decisions/server/DL-560.md @@ -0,0 +1,6 @@ +--- +id: DL-560 +decision: "Every Compass image (agent, runner, server, guest image, gateway) is built in compass for linux/amd64 and linux/arm64 and published as an OCI index whose members are first pushed as :git--, the retag and main-tag tooling verifies an index by its digest and member list, and the guest boot path chooses its kernel file (bzImage or Image) and serial console (ttyS0 or ttyAMA0) per arch." +status: "Active (Matt, 2026-10-17)" +record: ../../server-on-kubernetes.md#design +--- diff --git a/docs/designs/compass/decisions/server/DL-577.md b/docs/designs/compass/decisions/server/DL-577.md new file mode 100644 index 0000000..af51748 --- /dev/null +++ b/docs/designs/compass/decisions/server/DL-577.md @@ -0,0 +1,6 @@ +--- +id: DL-577 +decision: "The arm64 microVM guest gets no pre-merge boot: the P7 arm64 build gate is the only pre-merge proof, the Runner's startup boot canary fails closed on a node that cannot boot the guest, Runner nodes stay amd64 until an arm64 KVM boot passes, and no self-hosted arm64 CI runner is added." +status: "Active (Matt, 2026-10-18)" +record: ../../server-on-kubernetes.md#resolved-decisions +--- diff --git a/docs/designs/compass/server-on-kubernetes.md b/docs/designs/compass/server-on-kubernetes.md new file mode 100644 index 0000000..361fe1d --- /dev/null +++ b/docs/designs/compass/server-on-kubernetes.md @@ -0,0 +1,1100 @@ +# Compass Server on Kubernetes + +> Freezes on merge; later changes supersede by citation, never rewrite. +> Unless a path says otherwise, every path is in +> [RigelBuild/compass](https://github.com/RigelBuild/compass) at `main`. + +**Named decisions this record introduces:** D1–D10, inline in +[Design](#design), ledgered as DL-539 to DL-547 and DL-560 in +`decisions/server/`. The OQ-13 ruling is DL-577. DL-545 (D7) also covers +`serverName` (P9). + +## Resolved decisions + +Question numbers match the review threads on this record. + +- **OQ-1 — Admin token from a Secret.** A Go change (P4). The supplied token + file may be a symlink, and it may be group-readable when the process holds + the file's group. There is no init container (D4). +- **OQ-2 — Server secret provider.** `dotenv://` from a read-only mounted + Secret file (D4). +- **OQ-3 — LLM gateway.** Not deployed by this record (D1). +- **OQ-4 — Server certificate.** The operator supplies the certificate. The + product carries a private CA through the Runner's `serverCA` and the + client `--ca` (D6). +- **OQ-5 — Postgres.** The operator brings a Postgres that meets the D3 + contract. +- **OQ-6 — Image lane reuse.** `tools/server-image/` reuses the pure cores of + `tools/runner-image/` behind `--lane runner|server` (P1). +- **OQ-7 — Image visibility.** Every Compass image is public (D9, P5). +- **OQ-8 — Sessions across a Server restart.** The Runner must survive a + Server restart. That is its own record. Until it lands, see + [Known limitation](#known-limitation). +- **OQ-9 — Who runs Postgres.** Never the renderer (D1, D3). +- **OQ-10 — Digest bumps.** Paired per-push tags that image automation + follows (D8, P6). +- **OQ-11 — Retired.** It asked how one operator moves its own state, which + is not a product change. +- **OQ-12 — Multi-arch images.** Every Compass image is published for + `linux/amd64` and `linux/arm64` (D10, P7). +- **OQ-13 — arm64 microVM boot proof.** No pre-merge arm64 boot; the P7 + arm64 build gate is the only pre-merge proof, and P7 is unchanged. No + GitHub-hosted arm64 runner exposes `/dev/kvm`. The Runner's startup boot + canary (`runMicroVMPreflight`, `go/cmd/compass-runner/main.go`) fails + closed on a node that cannot boot the guest, and the Runner README says + amd64 only until an arm64 boot passes (P7). A self-hosted arm64 runner was + rejected: GitHub says self-hosted runners + ["should almost never be used for public repositories"](https://docs.github.com/en/actions/reference/security/secure-use#hardening-for-self-hosted-runners) + (DL-577). + +## Problem / Intent + +The Runner runs on Kubernetes: it has a digest-published image, a manifest +renderer, and projected-token identity. Nothing deploys a Compass Server +there, so the Runner's `serverAddr` has no target. This record lists the +product changes that let an operator run Server plus Runner on a cluster, and +the requirements the product places on that operator. How one operator runs +its own clusters is out of scope; see +[self-host and managed](https://github.com/RigelBuild/compass/blob/main/docs/concepts/self-host-and-managed.md) +and the [OSS-core boundary record](meta/oss-core-managed-boundary/design.md). + +## Approach + +### Ground truth + +- **Stack children.** `spawnChain` (`go/internal/stack/stack.go`) starts + Postgres, the OTel collector, NATS, the LLM gateway, compass-server, then + compass-runner. The first four are podman containers. The Server gets the + door as `--listen-fd 3` (`serverSpec`, `go/internal/stack/spec.go`). +- **Boot order.** `Serve` (`go/server/serve.go`) binds listeners, then + `openFabricAndStore` opens NATS before the store. `Open` + (`go/internal/store/store.go`) pings once without retry ("store: ping"), + then applies the embedded migrations under the session-scoped advisory + lock `migrationLockKey`. +- **RLS roles.** `0001_init.sql` runs `CREATE ROLE compass_system NOLOGIN + BYPASSRLS`, always re-runs `ALTER ROLE compass_system BYPASSRLS`, and grants + `compass_app` and `compass_system` to `current_user`. PostgreSQL lets only + a superuser or a role holding BYPASSRLS set that attribute + ([CREATE ROLE](https://www.postgresql.org/docs/current/sql-createrole.html)). + `WithSystemRole` (`go/internal/store/tenant_tx.go`) runs `SET LOCAL ROLE + compass_system` at runtime. +- **Door.** `loadNetworkTLS` (`go/server/network_door.go`) loads the keypair + once, with a TLS 1.3 floor and h2. It has no reload. +- **Health.** `go/server/health.go` serves `GET /healthz` and grpc health. It + reports NOT_SERVING once drain starts. `compass.runner` is SERVING only + while a Runner is enrolled. +- **Secrets.** `resolveMasterKey` (`go/server/serve.go`) fails boot without a + 64-hex `COMPASS_MASTER_KEY`, and `reconcileKeyState` fails boot when it + differs from the stored key. The resolver in + `go/internal/secrets/resolver.go` names the dotenv file when it cannot open + it ("secrets: dotenv provider file %s: %w"), and secretspec writes an audit + log under `$XDG_STATE_HOME`. +- **Admin token.** `readSuppliedAdminToken` (`go/server/network_door.go`) + `Lstat`s the path and fails on `info.Mode()&(os.ModeType|0o077) != 0`. A + Secret volume file is a symlink through `..data`. With `fsGroup`, kubelet's + `changeFilePermission` (`pkg/volume/volume_linux.go` in + kubernetes/kubernetes) runs `os.Lchown(filename, -1, int(*fsGroup))` and + adds `roMask = os.FileMode(0440)` on a read-only volume. The file fails + both checks today. +- **Runner.** `renderRunnerManifests` (`tools/runner-manifests/render-core.ts`) + takes an absolute `https` `serverAddr`, and now `serverCA` and + `egressAllow`. The hub holds one Runner (`go/internal/runnerhub/hub.go`). +- **Runner TLS name.** `NewCATrustClient` (`go/internal/runner/tls.go`) sets + only `RootCAs` and `MinVersion`, and without `--ca` the Runner uses + `http.DefaultClient`. Go's `addTLS` (`net/http/transport.go`) fills an + empty `ServerName` with the dialed host, so the Runner always checks the + certificate against the `--server` host. +- **Agent egress.** The microVM arms a default-deny firewall. + `--egress-allow` (`go/cmd/compass-runner/main.go`) is the only way to open + a host. `isValidHost` (`go/internal/runtime/host.go`) admits DNS names and + IP literals, never `host:port`. +- **Issuer keys.** `fetchHTTPS` (`go/internal/auth/workload_identity.go`) + sends no `Authorization` header. A `jwksFile` entry never fetches. + `validateRunnerClusters` (`go/internal/auth/runner_clusters.go`) requires + issuers to be unique only among the configured entries. +- **Image lanes.** Each lane publishes `:git-` by digest only when its + closure set changed. `resolveAncestor` (`tools/runner-image/retag-core.ts`) + walks first-parent ancestors and returns the newest published one; + `assertNoClosureChange` fails when a skipped commit changed the closure. + `release-runner-image` and `release-server-image` + (`.github/workflows/release.yml`) use them to mint `:vX.Y.Z` by digest. +- **Tag order.** No published tag sorts by commit. Commit timestamps are not + unique: 29 of the last 100 main commits share their second with another + commit. Flux's `RegexFilter.Apply` (`internal/policy/filter.go` in + fluxcd/image-reflector-controller) stores `f.filtered[tag] = item`, so two + tags with the same extracted value collapse to one. +- **Node shape** ([spike findings](platform/compass-runner-containerization/spike-findings.md)). + `TestFullBoot` passed on a bare-metal node with `/dev/kvm` `0666`, a + device plugin and `hostUsers: false`. A host with AppArmor needed + `kernel.apparmor_restrict_unprivileged_userns=0`. + +### Design + +**Topology (D1).** Each stack child that runs as a container today becomes +its own pod. The Server container supervises nothing. + +| Child | On Kubernetes | +| --- | --- | +| compass-server | Deployment, `replicas: 1`, `strategy: Recreate` | +| NATS | StatefulSet, 1 replica, PVC, `DefaultNatsImage`, the stack's R1 file JetStream config | +| Postgres | operator-provided, outside the renderer (D3) | +| OTel collector | operator-provided; `OTEL_EXPORTER_OTLP_ENDPOINT` points at it | +| LLM gateway | not deployed; see the [gateway record](server/compass-server-llm-gateway/design.md) | +| compass-runner | the existing DaemonSet, on exactly one KVM node | + +One Server replica, because the hub holds one Runner and the +[delivery-cutover record](infra/runtime/compass-managed-delivery-cutover/design.md) +keeps a single Server. `Recreate` also keeps two Servers from migrating at +once. Exactly one Runner pod per Server: `enroll` (`go/internal/runnerhub/hub.go`) +replaces the attached Runner and drives its bound accounts offline, so a +second Runner drops the first one's sessions. Only `compass-stack` needs a +gateway ("gateway image is required; pass --gateway-image or +--gateway-external", `go/internal/stack/stack.go`); `compass-server` has no +gateway flag. Agents keep a raw provider key plus an egress allowlist (D7). + +**Server image (D2).** `ghcr.io/rigelbuild/compass-server` follows the Runner +image's shape. `server-image/Dockerfile` copies the realised +`.#compass-server` closure onto a digest-pinned distroless `static-debian12` +base. The image runs as `65532:65532`, with `/nix/store/.compass-server` as +its entrypoint. The closure carries secretspec and the wrapper's bash. +`SERVER_IMAGE_CLOSURE_PATHS` gates publishing. + +**Data and migrations (D3).** Migrations stay in Server boot; there is no +Job. The renderer never renders Postgres. The operator's Postgres must meet +this contract: + +- The DSN names a direct, session-mode endpoint. The migration lock is + session-scoped, so a transaction-mode pooler breaks it. +- The login role owns the target database, so it can create objects in + `public`. +- PostgreSQL ≥16 with a login role that holds CREATEROLE plus BYPASSRLS, or + a superuser on any version. `0001_init.sql` creates and re-asserts the + BYPASSRLS role. Before 16, "You must be a superuser to create a new role + having the BYPASSRLS attribute" + ([CREATE ROLE, 15](https://www.postgresql.org/docs/15/sql-createrole.html)); + from 16, "Only superuser roles or roles with BYPASSRLS can specify + BYPASSRLS" ([CREATE ROLE, 16](https://www.postgresql.org/docs/16/sql-createrole.html)). + A service whose owner role cannot hold BYPASSRLS cannot run today's schema. +- The database answers at boot. A suspended database fails the single ping, + and the pod restarts until it answers. +- One replica fits the default pool of 10. + +**Secrets (D4).** The renderer names Secrets but never holds a value. The +operator creates every Secret and backs up the master key; losing it is +unrecoverable (`docs/self-host.md`). + +- `COMPASS_DATABASE_DSN`, `COMPASS_NATS_URL` and `COMPASS_S3_*` come from + Secrets as env. +- Server secrets come from one Secret key, `server.env`, mounted read-only + and read through `--secret-provider dotenv:///etc/compass/secrets/server.env`. + The file holds `COMPASS_MASTER_KEY` and the `SERVER_` values. + `SpecResolver.buildManifest` (`go/internal/secrets/resolver.go`) declares + exactly those names. Unlike `env://`, the file keeps the master key out of + the process env. +- `SERVER_*` values re-resolve on use. `COMPASS_MASTER_KEY` is read once at + boot and must never change in the mounted Secret. +- `XDG_STATE_HOME` and `--state-dir` sit on one emptyDir under `/run/compass`. + The state dir is a child the Server creates at `0700` (`ensurePrivateDir`, + `go/server/socket.go`). +- The admin token Secret mounts directly, `defaultMode: 0440`, under pod + `fsGroup: 65532`. `--admin-token-file` names the projected file. P4 makes + the Server accept it. No init container and no init image. +- Agents get forge tokens only from the Server's App broker. + `buildGitCredentialBroker` (`go/server/git_credential.go`) returns nil + unless the primary App id is set, and `IsReservedGitHubEnvName` + (`go/internal/secrets/secrets.go`) strips user `GH_TOKEN`-style secrets. So + the renderer takes a `forge` value and renders `--forge-*` args: ids and + secret names, never values. The reviewer App is optional; + `wireForgeWriteCaller` (`go/server/serve.go`) wires agent `ForgeCall` + writes only when both Apps are set. +- An admin stores each agent provider key with `compass secret set + --kind provider --provider --delivery file`. + +**NATS access (D5).** The stack's NATS has no auth because it listens on +loopback. In a cluster, NATS requires a user and password, and a +NetworkPolicy admits port 4222 only from the Server pod. The password reaches +the NATS config through a variable reference, and the credentials ride in +`COMPASS_NATS_URL`. The fabric's connect path keeps URL credentials out of its +errors (`go/internal/fabric/fabric.go`). + +**Door, TLS and serverAddr (D6).** The Server terminates its own TLS. No +ingress may terminate it: the door refuses cleartext, and the Runner's bidi +stream needs h2 end to end. + +- **Certificate.** The operator supplies a `kubernetes.io/tls` Secret whose + SANs cover the in-cluster name and any external name. A certificate that + names only the public host also works when the Runner sets `serverName` + (P9). A private CA works; the Runner's `serverCA` value and the client + `--ca` carry it. +- **Renewal.** The door loads its keypair once, so a renewed certificate + takes effect on restart (see [Known limitation](#known-limitation)). +- **Services.** Service `compass-server` (ClusterIP) maps 443 to 50052. An + optional LoadBalancer Service on 50052 passes TCP through. +- **serverAddr** is `https://compass-server..svc`. +- **Clients.** The CLI uses `--server-addr`, `--ca` and the admin token + (`go/cmd/compass/client.go`). The desktop app uses client mode with + `ca_cert` (`docs/self-host.md`). A browser origin comes from + `corsAllowedOrigin`. +- **Probes.** Startup and readiness are HTTPS `httpGet` on `/healthz`. + Readiness never depends on `compass.runner`, because the Runner needs the + Service before it can enroll. There is no liveness probe. + +**Runner values (D7).** + +- **Identity.** A cluster whose issuer serves no anonymous discovery uses a + `jwksFile` entry: the operator reads `/openid/v1/jwks` at render time and + passes it as a `jwks` value, and the Server renderer writes it into the + runner-clusters ConfigMap. This needs no Go change. A signing-key rotation + then needs a re-render and a Server rollout. An issuer with anonymous + discovery keeps `jwksURI`. +- **CA.** `serverCA` mounts the CA and sets `COMPASS_RUNNER_CA`. +- **Server name.** `serverName` sets `COMPASS_SERVER_NAME`, the name the + Runner checks on the certificate (P9). +- **Egress.** `egressAllow` renders one `--egress-allow=` arg. The + renderer applies the `isValidHost` grammar, so `host:port` fails at render + time. A session needs the provider API host, `github.com` and + `api.github.com`. +- **Node.** The KVM node exposes `/dev/kvm` as the extended resource named by + `kvmResourceName`, stages the seccomp profile at `seccompProfilePath` + under the kubelet seccomp root, and supports `hostUsers: false`. +- **Session volumes.** `hostPaths.sessionVolumeRoot` is a directory with no + project id. The operator creates `volumes/` inside it and puts the project + id, `FS_XFLAG_PROJINHERIT` and the limits on `volumes/`. The Runner's + volume root is that subdirectory (P8). The quota check needs an + unprojected reference above the volume root, and in the pod the only one + is the top of the hostPath mount. + +**Delivery split and paired tags (D8).** Public compass holds +`tools/server-manifests/` and `tools/runner-manifests/`: values in, plain +objects out, digest-only images, no Secrets. The operator holds the values +for each cluster, the Secrets, the certificate, node config, and how the +output is applied. The head commit of every main push gets +`:main--` on both the Server and Runner images (P6). `n` is that +commit's first-parent count on main. Each lane's tag names that lane's newest +publish at or before the commit, re-tagged by digest with no rebuild. Both +tags of one `n` therefore pair the images built from the same Go inputs, as +long as both closure sets cover all of `go/**` (compass PR #2357). Main +requires linear history and forbids force-pushes, so `n` is unique and rises +with each push. A push of several commits tags only its head, so `n` has +gaps; a numerical policy does not care. Image automation follows the tags +with a numerical policy on `n`: + +```yaml +filterTags: + pattern: '^main-(?P[1-9][0-9]*)-[0-9a-f]{12}$' + extract: '$n' +policy: + numerical: + order: asc +digestReflectionPolicy: IfNotPresent +``` + +Only the digest may change a pod spec. The tag changes on every push even +when a lane's digest does not, so a pod spec that carries the tag would roll +the Server and the Runner on every push and drop every session. The +renderers keep rejecting a tagged ref (`assertRunnerImageDigest`, +`assertServerImageDigest`). The operator's setter therefore writes only the +`:digest` field of the policy (`{"$imagepolicy": "::digest"}`), +for example on a kustomize `images[].digest` override of the rendered +output. Values files are JSON and carry no setter comment. + +One automation over both policies bumps both digests in one commit. No +operator bump job is needed. The tagger writes the Server tag, then the +Runner tag, seconds apart. A scan between the two writes sees a new Server +with the previous Runner for one interval; the next scan pairs them. + +**Public images (D9).** Every Compass image is public: +`compass-agent`, `compass-runner`, `compass-server`, `compass-guest-image` +and `compass-gateway` under `ghcr.io/rigelbuild`. On 2026-10-17 all five +answered `skopeo list-tags --no-creds`. Neither renderer emits +`imagePullSecrets`. A command that only reads the registry (`skopeo +inspect`, `podman manifest add`) sends no credentials; a write (`skopeo +copy`, `podman manifest push`) uses them for the destination only (P5). +One read is part of a write: the probe of the exact tag a push is about to +write uses the login file. GHCR answers an anonymous request for a package +that does not exist yet with 403, and every probe classifier aborts on +403, so an anonymous probe would stop the push that creates the package. +GitHub's REST packages API can list, get, delete and restore a +package, but has no call that sets visibility. A new package starts private, +and making it public is a one-way switch in the package settings: a manual +step at its first publish +([package visibility](https://docs.github.com/en/packages/learn-github-packages/configuring-a-packages-access-control-and-visibility)). + +**Multi-arch images (D10).** Every Compass image is published for +`linux/amd64` and `linux/arm64`, built in compass. `:git-`, +`:main--` and `:vX.Y.Z` name an OCI index of the two members, and +each member is pushed first as `:git--`. This is the agent +lane's shape today: `publish-image-amd64` and `publish-image-arm64` +(`.github/workflows/release.yml`) push the members, and +`publishImageIndex` (`tools/agent-image-index/index.ts`) composes and checks +the index against `EXPECTED_PLATFORMS = "linux/amd64,linux/arm64"`. The +other four lanes are single-arch now: + +- Runner: `IMAGE_PLATFORM = "linux/amd64"` and an x64-only host check + (`tools/runner-image/build.ts`). +- Server: "server-image targets linux/amd64, but this host is + ${process.arch}" (`tools/server-image/build.ts`). +- Gateway: `PLATFORM = "linux/amd64"`, and `layoutDigest` + (`tools/gateway-image/core.ts`) requires exactly one manifest. +- Guest image: one artifact manifest. `fetchGuestManifest` + (`go/internal/stack/materialize.go`) accepts only a single manifest. + +`flake.nix` lists `systems = [ "x86_64-linux" ]`. `manifestIdentity` +(`tools/runner-image/retag-core.ts`) refuses an index ("an image index, or +malformed"), so retag and the P6 tagger must learn to verify one. + +The guest boot path is also amd64-only, in three ways. The +[guest agent pin record](infra/runtime/compass-guest-agent-pin-multiarch/design.md) +named them and left them to a follow-up (its T5). P7 is that follow-up. + +- **Kernel file.** At the pinned nixpkgs, `linuxPackages.kernel.target` is + `bzImage` on `x86_64-linux` and `Image` on `aarch64-linux`. `bzImage` is + hard-coded in `kernelImagePath` (`tools/runner-image/build-core.ts`), the + kernel path in `tools/guest-image/publish.ts`, `guestImageEnv` + (`tools/microvm-boot-test/run-core.ts`), the `microvm` job in + `.github/workflows/ci.yml`, and the kernel `annotationName` in + `guestAssets` (`go/internal/stack/materialize.go`). That name must equal + `ASSET_FILENAMES.kernel` (`tools/guest-image/publish-core.ts`), because + `validateGuestLayer` checks the layer annotation. +- **Boot modules.** `moduleConfigCheck` (`guest-image/default.nix`) fails + unless eight symbols are `=m`. The aarch64 kernel config has + `CONFIG_VIRTIO_PCI`, `CONFIG_VIRTIO_BLK`, `CONFIG_VIRTIO_NET` and + `CONFIG_PACKET` as `=y`, so the arm64 initrd build fails. A built-in module + is harmless: kmod `modprobe` exits 0 for it, and `makeModulesClosure` + records it as a `builtin` dependency. +- **Console.** `vmmArgs` (`go/internal/runtime/microvm/launch.go`) always + appends `console=ttyS0`. The arm64 serial device is `ttyAMA0`, as + cloud-hypervisor's AArch64 example shows, so guestd's stderr would miss + the captured console. + +P7 fixes all three, with a unit test per arch for each. No CI job can boot +an arm64 guest: GitHub's hosted arm64 runners expose no `/dev/kvm` +([runner-images #14062](https://github.com/actions/runner-images/issues/14062)). +So P7 adds an arm64 build gate, not a boot gate. When P7 lands, the Server +and NATS run on a node of either arch. The Runner image ships both members, +but the Runner is supported on amd64 only until an arm64 microVM boot +passes (OQ-13). + +### Known limitation + +A Server restart ends every live session. A dropped session stream ends +`Run` (`go/internal/runner/run.go`, "runner sessions loop: %w"), and the +Runner exits. The Runner has no liveness probe because "a pid-1 restart tears +down all node sessions" (`tools/runner-manifests/render-core.ts`). Each +Server deploy and each certificate renewal therefore drops sessions, and D8 +makes deploys frequent. A separate record makes the Runner survive a Server +restart: it re-dials, re-enrolls with the hub, and keeps its microVMs. + +### Alternatives considered + +- **One pod running `compass-stack up`.** Rejected: the supervisor drives + podman and host processes. +- **An init container that copies the admin token.** Rejected: it adds an + image value and a pod step for a check the Server can make itself (P4). +- **A renderer-owned Postgres StatefulSet.** Rejected: no upgrade path and no + point-in-time recovery, and it would read as a supported surface. +- **Ingress-terminated TLS.** Rejected: an extra h2 hop and a second + certificate. +- **A migration Job.** Rejected: no migrate-only entrypoint exists, and the + advisory lock serializes migrations. +- **An operator bump job, or per-image automation on `:git-`.** + Rejected: the first is a scheduled job per operator; the second has no + order and pairs nothing. +- **A timestamp tag.** Rejected: equal timestamps collapse in the policy. + +## Plan + +### Global Constraints + +- **Public repo.** No operator names, hosts, zones, cluster names or private + product detail. +- **PRs.** Each task is one compass PR, red then green. P2 merges after P4, + because its output boots only with P4. P5, P6 and P7 form one stack in + that order: all three edit `tools/runner-image/` and `release.yml`. P8 + edits only `tools/runner-manifests/`. P9 edits the Runner's TLS client, + its flags and `tools/runner-manifests/`, but no line that P8 edits. P8 and + P9 are independent of the rest. +- **Images.** Digest only. Renderers reject tags, as `assertRunnerImageDigest` + does. Every image is an OCI index of `linux/amd64` and `linux/arm64` once + P7 lands. +- **Registry auth.** Reads send no credentials; writes use credentials for + the destination only. The probe of the tag a push is about to write is + part of the write and uses the login file (D9). +- **Secrets.** Renderers emit no Secret object and no secret value. +- **Kubernetes.** ≥1.33, for user namespaces. +- **PostgreSQL.** ≥16 for a non-superuser owner role (D3). +- **Server pod.** uid and gid 65532, `fsGroup: 65532`, + `readOnlyRootFilesystem`, drop ALL, `RuntimeDefault` seccomp, + `automountServiceAccountToken: false`. +- **Lanes.** TypeScript is Bun with `bun:test`. Go DB tests use the `pgtest` + build tag. +- **Spec impact.** P4 changes the requirement "The network door mints and + persists a bootstrap admin token" in `docs/specs/product/compass.md`. P9 + adds a `compass-runner --server-name` requirement to the same file. No + other task changes a spec. + +### Order + +```text +P1 server image (#2355, merged) ─┐ +P3 runner values (#2360, merged) ┤ +P8 runner volume root ───────────┼──► operators render and deploy +P9 runner TLS server name ───────┤ +P4 admin token (Go) ──► P2 server renderer (#2363) ┘ +P5 anonymous reads ──► P6 paired main tags ──► P7 multi-arch images +Deferred: F1 door TLS reload, F2 tokenFile +``` + +### P1 — Server image and publish lane + +Merged as compass PR #2355, after #2357 widened `RUNNER_IMAGE_CLOSURE_PATHS` +to `go/**`. Its smoke checks `--version`, a missing-DSN exit 1, user +`65532:65532`, a read-only `dotenv://` boot with Postgres and NATS, and a +reproducible digest. Files: `server-image/{Dockerfile,moon.yml,.dockerignore}`, +`tools/server-image/{build.ts,build-core.ts}`, `--lane` in +`tools/runner-image/{publish.ts,retag.ts}`, and the `publish-server-image` and +`release-server-image` jobs. `SERVER_IMAGE_CLOSURE_PATHS` is +`server-image/**`, `tools/server-image/**`, `tools/runner-image/**`, `go/**`, +`tools/toolchain/secretspec-env.nix`, `devenv.lock`, `flake.nix`, +`flake.lock`, `version.txt` and `.github/workflows/release.yml`. + +### P2 — Server manifests renderer + +The first renderer merged as compass PR #2363. A follow-up PR against main must drop +`initImage`, the `init-admin-token` init container and the `postgres` value +with its StatefulSet, Service and NetworkPolicy. It must add `forge`, +`nodeSelector` and `toleration`. Tests use golden output plus invariants: + +- `replicas: 1` with `Recreate`. +- `serverImage` and `natsImage` accept only `repo@sha256:<64 hex>`. A + tagged ref, `repo:tag@sha256:…` included, fails parse with the field name. +- No `Secret` kind, no `initContainers`, no Postgres object. +- `parseServerDeployValues` rejects a `postgres` or `initImage` key with an + error naming the key, so an older values file fails instead of rendering + without them. +- The admin token volume has `defaultMode: 0o440`, and `--admin-token-file` + names the file in that volume. +- The NATS NetworkPolicy admits 4222 only from the Server label. +- Probes hit `/healthz`; no liveness probe. +- `XDG_STATE_HOME` and `--state-dir` sit on the emptyDir, and the state dir + is not its root. +- A `runnerClusters` entry with `jwks` renders a ConfigMap key and a matching + `jwksFile`; `jwks` plus `jwksURI` is rejected. +- `forge` renders `--forge-host`, `--forge-app-id`, `--forge-installation-id`, + `--forge-app-key-secret`, `--forge-app-webhook-secret`, and `--forge-repos` + when set. Ids are positive integers. `forge.reviewer` adds the three + `--forge-reviewer-*` args. An absent value renders no arg. +- `nodeSelector` and `toleration` render as in the Runner renderer, with + `operator: "Equal"`; absent values render none. + +The README covers the render steps, the JWKS read, the `server.env` lines, +and the never-change rule for `COMPASS_MASTER_KEY`. It states the D3 +Postgres contract with its floor: PostgreSQL ≥16 for a CREATEROLE plus +BYPASSRLS owner, or a superuser on any version. It states that image +automation must write only the digest (D8). + +### P3 — Runner renderer `serverCA` and `egressAllow` + +Merged as compass PRs #2358 and #2360. `serverCA` takes exactly one of +`configMapName` or `secretName`, plus `key`, mounts at +`/etc/compass/server-ca` and sets `COMPASS_RUNNER_CA`. `egressAllow` renders +one `--egress-allow=` arg and rejects hosts outside `isValidHost`. + +### P4 — Admin token file from a Secret volume (Go) + +`readSuppliedAdminToken` opens the path once with +`O_RDONLY|O_NONBLOCK|O_CLOEXEC`, which follows the symlink, and checks the +open file's `Stat` before it reads. `O_NONBLOCK` keeps the open of a FIFO +from blocking boot; today `Lstat` fails a FIFO before any open. A non-regular +file fails before any read. Checking the open file removes the gap between +check and read. The file must be regular, with no setuid, setgid or sticky +bit, no group-write bit and no other bit. Group-read is allowed only when +the file's gid is the process's `os.Getegid()` or in `os.Getgroups()`. +kubelet's 0440 file under `fsGroup` passes; a 0640 file in a group the +process does not hold fails. `writeSuppliedAdminTokenFile` still copies the +token to a 0600 file under the state dir, and `reusableAdminToken` keeps its +strict `Lstat` check for that copy. Both are unchanged. + +Tests, red first: + +- `go/server/admin_token_mode_test.go` (`//go:build unix`), on the pure + check: 0600 passes; 0640 and 0440 pass with a held gid; 0640 fails with an + unheld gid; 0660, 0604, a directory, a FIFO and a setgid file fail. +- `TestIssueAndWriteAdminTokenRejectsMalformedSuppliedFiles` + (`go/server/admin_token_reuse_pgtest_test.go`) drops its "group readable" + and "symlink input" rejection cases. It adds "group writable" (0660), + "other readable" (0604), and "fifo": a `syscall.Mkfifo` at 0600 with no + writer, plus a symlink to it. Both fail with "must be a private regular + file" within a 5-second test deadline. A plain `os.Open` would block there. +- A new pgtest builds the Secret volume layout: `..data/token` at 0440 in + `os.Getegid()`, and `token -> ..data/token`. Boot registers the token, + `assertAdminTokenAuthenticates` passes, and the state-dir copy is 0600. + +Spec impact: the requirement "The network door mints and persists a +bootstrap admin token" (`docs/specs/product/compass.md`) says the file +"SHALL name a private regular file". It changes to: a regular file, reached +directly or through symlinks, with no other bits and no group-write bit, and +group-read only for a group the Server process holds. The +`--admin-token-file` help in `go/cmd/compass-server/main.go` ("Private file +containing the supplied bootstrap-admin token") changes to match. + +### P5 — Anonymous registry reads + +The five packages are public already. P5 makes every registry read in +compass anonymous except the destination probe, so D9 holds, and keeps the +packages public. + +Dropping `--authfile` is not enough. Every publish job writes +`REGISTRY_AUTH_FILE=$RUNNER_TEMP/ghcr-auth.json` to `$GITHUB_ENV` +(`.github/workflows/release.yml`). skopeo's `sharedImageFlags` defaults +`--authfile` to `os.Getenv("REGISTRY_AUTH_FILE")`. podman's +`GetDefaultAuthFile` returns the same variable first, and `podman manifest +add` has no `--no-creds`. So each read names its anonymity: + +- Every `skopeo inspect` of a registry ref passes `--no-creds`. skopeo then + sets an empty `DockerAuthConfig`, which `GetCredentials` + (`containers/image`, `pkg/docker/config`) returns before any auth file. +- The exception is the destination probe: the `inspect` of the exact tag a + push is about to write, whose answer decides publish, skip or abort. It is + part of the write and uses the same credentials as the push. On + 2026-10-17, an anonymous probe of a package that does not exist got + "Requesting bearer token: received unexpected HTTP status: 403 Forbidden"; + a missing tag on an existing public package got "manifest unknown". The + pre-push classifiers free the tag only on a not-found answer: + `tagDisposition` (`tools/guest-image/publish-core.ts`, + `tools/gateway-image/core.ts`), `guard_immutable` + (`agent-image/publish.sh`) and `isManifestUnknown` + (`tools/agent-image-index/index.ts`). So an anonymous probe would abort + the push that creates a new package. +- Every `skopeo copy` passes `--src-no-creds` and `--dest-authfile `, + never a bare `--authfile`. The rule holds for a local source too, where + the source flag is a no-op. +- `podman manifest add` passes `--authfile /anonymous-auth.json`, + a file holding `{}`. A named auth file turns off every fallback path in + `getAuthFilePaths` (`if !userSpecifiedPath`). `podman manifest push` keeps + the login file. + +The sites: + +- `tools/runner-image/`: `retag.ts` builds its probe, copy and verify argv + with new `inspectArgv` and `copyArgv` in `retag-core.ts`. Its copy today + passes no auth flag and leans on the environment; it now requires + `REGISTRY_AUTH_FILE` and fails `EXIT.usage` without it. `tagArgs` + (`publish-core.ts`) moves from `--authfile` to the copy pair. +- `tools/agent-image-index/index.ts`: `inspectCommand` drops `authFile` and + passes `--no-creds`. `guardImmutableTag` probes `:git-` with a new + `destinationProbeCommand` that carries the login file. `composeLocalIndex` + adds members with the empty auth file. +- `tools/guest-image/publish.ts` and `tools/gateway-image/publish.ts`: the + pre-push probe keeps its `auth` spread, the login file when + `REGISTRY_AUTH_FILE` is set. The post-push re-read passes `--no-creds`. + The copy passes `--src-no-creds`, plus `--dest-authfile` when + `REGISTRY_AUTH_FILE` is set. +- `tools/release-notes/index.ts`: `gatherImage` adds `--no-creds`. + `tools/guest-image/pin-agent-image.ts` passes it already. +- The `release-image` job: three `skopeo inspect --authfile + "$REGISTRY_AUTH_FILE"` reads (the ancestor probe and two digest checks) + and one `skopeo copy --multi-arch all`. All three are source or + verification reads, so all pass `--no-creds`. +- `agent-image/publish.sh`: the `guard_immutable` probe of `$dest` keeps + `--authfile "$REGISTRY_AUTH_FILE"`. The post-copy re-read passes + `--no-creds`, and the `copy "nix:$SPEC" "$dest"` passes the copy pair. + The pinned fork skopeo (`tools/toolchain/skopeo-nix2container-env.nix`, + 1.24.1) has `--no-creds`, `--src-no-creds` and `--dest-authfile`. + +`classifyProbe` still reads 403 as ambiguous, because a private package and +a missing package give the same 403. + +Stale text goes: the `release-runner-image` step comment "The runner package +is private, so even the probes need auth", the `retag.ts` header "so private +packages work", and the `retag-core.test.ts` case name "a private package +without auth (403 Forbidden) is ambiguous" and its "on this private package" +comment. + +A new check runs as the last step of each of the five publish jobs. It +reads the just-published tag anonymously and fails with the package settings +path when the package is not public. A new package therefore fails its first +publish until someone sets it public: the one manual step, because no API +sets visibility. On a new package, the anonymous post-push re-read fails +first, with 403. The re-run after the package is set public finds the tag +through the destination probe, skips the push, and passes the check. + +Tests, red first. Each asserts the flag on every read, and the login file on +the destination probe: + +- `retag-core.test.ts`: `inspectArgv` carries `--no-creds`; `copyArgv` + carries `--src-no-creds` and `--dest-authfile` and no `--authfile`. +- `publish-core.test.ts`: `tagArgs` carries the copy pair and no + `--authfile`. +- `tools/agent-image-index/index.test.ts`: the `guardImmutableTag` probe of + `:git-` carries `--authfile ` and no `--no-creds`; every other + recorded `skopeo inspect` carries `--no-creds`; every `podman manifest add` + carries the empty auth file, whose content is `{}`; only `podman manifest + push` and the destination probe carry the login file. +- `tools/guest-image/publish.test.ts`: the case "threads --authfile to + every registry call when the login pinned one" now requires `--authfile + /tmp/creds.json` on the pre-push `inspect`, `--src-no-creds --dest-authfile + /tmp/creds.json` on the `copy`, and `--no-creds` on the re-read. A new case + stubs skopeo to answer 403 to an `inspect` without `--authfile` and + "manifest unknown" to one with it, and requires the `copy` to run. +- `tools/gateway-image/publish.test.ts`: the case "passes the auth file to + probe, copy, and re-read" expects `inspect --authfile + /tmp/registry-auth.json --raw`, then `copy --src-no-creds --dest-authfile + /tmp/registry-auth.json`, then `inspect --no-creds --raw`. +- A new `tools/runner-image/registry-auth.test.ts` joins continuation lines + in `release.yml` and `agent-image/publish.sh`. Every skopeo `inspect` of a + non-`nix:` ref must carry `--no-creds`. Every `copy` must carry + `--src-no-creds` and `--dest-authfile`. Only `login` may carry + `--authfile`. It checks the `gatherImage` template the same way. One case + is allowed: the body of `guard_immutable` in `agent-image/publish.sh` + (from `guard_immutable() {` to the next line that is exactly `}`) holds + exactly one `inspect`, and it must carry `--authfile + "$REGISTRY_AUTH_FILE"` and no `--no-creds`. Today's sources fail the + first three rules. +- `assert-public-core.test.ts` maps a manifest to public, and a 403, a + "manifest unknown" and a timeout to failures that name the package. + +### P6 — Paired main tags + +A new job `tag-main-images` in `.github/workflows/release.yml` runs on every +main push and tags the push's head commit. It +`needs: [publish-runner-image, publish-server-image]`, so it runs after both +lanes publish or skip, and never when either failed. It has +`permissions: {contents: read, packages: write}`, its own concurrency group +with `cancel-in-progress: false` and `queue: max`, and a checkout with +`fetch-depth: 0`; a shallow clone gives a wrong `n`. + +`tag-main.ts` computes `n` with `git rev-list --count --first-parent `. +For each lane it runs `resolveAncestor` over `git rev-list --first-parent +--max-count=50 ` and `assertNoClosureChange` over the diff from the +resolved commit, as `retag.ts` does. It resolves both lanes before any write. +Then it copies each digest to `:main--` with `--preserve-digests`, +Server first, and checks each with `assertCoherent`. A lane that published +this commit resolves at position 1. + +Tests, red first: + +- `retag-core.test.ts`: `mainTag(1638, <40-hex>)` returns + `main-1638-`; 0, a fraction and a short sha throw `EXIT.usage`. +- `resolveMainPair`: both lanes found at the head; a lane absent at the head + with no closure change resolves its ancestor; a lane absent with a closure + change throws, and the injected writer records no call. +- `tools/runner-image/closure-paths.test.ts`: `go/gen/compass/v1/compass.pb.go` + hits both closure sets through `closureChanges`; `tag-main-images` needs + both publish jobs and checks out with `fetch-depth: 0`. +- The tag matches the D8 pattern, and Flux's extract yields `n`. + +### P7 — Multi-arch images + +Every lane publishes `linux/amd64` and `linux/arm64` members and one OCI +index, as the agent lane does (D10). + +- **Closures.** `flake.nix` `systems` adds `aarch64-linux`, so + `.#compass-server`, `.#compass-runner` and the guest assets evaluate on + both. `guest-image/default.nix` already selects its agent member per + system (`agentPin.platformFor`). +- **Build guards.** `hostPlatform` (`tools/runner-image/build-core.ts`) + maps the host (`x64` → `linux/amd64`, `arm64` → `linux/arm64`) and throws + on any other host. `tools/runner-image/build.ts` and + `tools/server-image/build.ts` call it; the server lane already imports + this module. `PLATFORM` in `tools/gateway-image/core.ts` becomes a + parameter. The `IMAGE_PLATFORM` comment in `tools/runner-image/build.ts` + ("the cluster nodes are amd64") goes. +- **Jobs.** Each of `publish-runner-image`, `publish-server-image`, + `publish-guest-image` and `publish-gateway-image` becomes a matrix over + `ubuntu-latest` and `ubuntu-24.04-arm`, as `publish-image-amd64` and + `publish-image-arm64` are. Each leg pushes `:git--` by digest. + The change gate runs once, in a `-gate` job that both legs need. +- **Index.** A new `publish--index` job per lane needs both legs and + composes `:git-`. `publishImageIndex` moves from the agent-only + `IMAGE_REF` to a `repo` input, so all five lanes share it and its + `EXPECTED_PLATFORMS` check. +- **Guest artifact.** The guest index carries two artifact manifests with + `platform` set. `fetchGuestManifest` (`go/internal/stack/materialize.go`) + accepts an index and picks the member for `runtime.GOARCH`; a digest that + names one manifest still works. +- **Retag and the P6 tagger.** `manifestIdentity` accepts an index and + returns the index digest plus each member's platform and digest. + `assertCoherent` compares the index digest and the member list. Every copy + uses `--multi-arch all --preserve-digests`. `tag-main-images` needs the + index jobs, not the legs. +- **Kernel file.** `kernelFileName` (`tools/runner-image/build-core.ts`) + maps `linux/amd64` to `bzImage` and `linux/arm64` to `Image`. It is the + only TypeScript table. `kernelImagePath`, `buildArgs`, the kernel path in + `tools/guest-image/publish.ts`, `assetFilenames` (it replaces + `ASSET_FILENAMES` in `tools/guest-image/publish-core.ts`) and + `guestImageEnv` take the platform and call it. Each entry point passes + `hostPlatform(process.arch)`. `tools/guest-image/moon.yml` and + `tools/microvm-boot-test/moon.yml` add `/tools/runner-image/*-core.ts` to + their inputs, as `tools/server-image/moon.yml` does. In Go, a new + `guestKernelName(goarch)` (`go/internal/stack/materialize.go`) returns + `Image` for `arm64` and `bzImage` otherwise, and the kernel entry of + `guestAssets` takes its `annotationName` from + `guestKernelName(runtime.GOARCH)`. An amd64 artifact then fails + `validateGuestLayer` on an arm64 host. `pinSmoke` (`guest-image/default.nix`), + which the `compass-guest-image:eval-arm64` task evaluates, also asserts + that `compass-guest-kernel.target` is `bzImage` for `x86_64-linux` and + `Image` for `aarch64-linux`, so a nixpkgs move that renames the file fails + a gate. The `microvm` job in `ci.yml` stays on amd64 and keeps + `$kernel/bzImage`. +- **Boot modules.** `moduleConfigCheck` (`guest-image/default.nix`) accepts + `=m` or `=y`. It fails only when a symbol is absent or `is not set`. It + becomes a function of the config file, so a fixture test can run it. +- **Console.** A new `serialConsole(goarch)` + (`go/internal/runtime/microvm/launch.go`) returns `ttyS0` for `amd64` and + `ttyAMA0` for `arm64`, and errors on any other arch. `launch` calls it + with `runtime.GOARCH` before it starts a child and passes the result to + `vmmArgs`, which appends `console=`. +- **arm64 build gate.** A new `guest-arm64` job in `.github/workflows/ci.yml` + runs on `ubuntu-24.04-arm` when `microvm_affected` is true, as + `agent-fod-arm64` does, and the rollup checks it the same way. It runs + `bun tools/microvm-boot-test/run.ts --build-only`. That mode builds the + guest and the VMM stack, fails when the kernel file is missing, and skips + `go test`. It cannot boot: hosted arm64 runners have no `/dev/kvm`. +- **arm64 Runner support (OQ-13).** `tools/runner-manifests/README.md` + gains the line "Runner nodes must be amd64." The PR that removes it + carries a passing `moon run compass-go:test-microvm` log from an arm64 + KVM host with GICv3. Until then, the Runner's boot canary fails closed on + a node that cannot boot the guest. +- **Docs.** `docs/self-host-guest-image.md` names `Image` beside `bzImage` + in its file table and adds an `org.compass.guest.layer.Image` row. + Comments that name only `ttyS0` or `bzImage` name both: + `go/internal/runtime/microvm/config.go`, `go/internal/guestd/guestd.go`, + `go/cmd/compass-guestd/main.go` and `guest-image/default.nix`. + +Tests, red first: + +- `retag-core.test.ts`: `manifestIdentity` on a two-member index returns + both members; an index missing `linux/arm64` throws `EXIT.badManifest`; + `assertCoherent` fails on a member digest change. +- `tools/agent-image-index/index.test.ts`: `publishImageIndex` with a + `compass-server` repo composes the index from the `-amd64` and `-arm64` + tags. +- `go/internal/stack/materialize_test.go`: an index resolves to the + `runtime.GOARCH` member; an index with no such member fails and names the + arch. `guestKernelName("arm64")` is `Image` and + `guestKernelName("amd64")` is `bzImage`. +- `tools/runner-image/closure-paths.test.ts`, which already parses + `release.yml`: each of the four publish jobs has an `ubuntu-latest` and an + `ubuntu-24.04-arm` leg, and each `publish--index` job needs both. +- `tools/runner-image/build-core.test.ts`: `hostPlatform("arm64")` returns + `linux/arm64`, and `hostPlatform("ia32")` throws. For `linux/arm64`, + `kernelImagePath` and `buildArgs(…).GUEST_KERNEL` end in `/Image`; for + `linux/amd64` they keep `/bzImage`. +- `tools/guest-image/publish-core.test.ts`: `layoutPlan` for `linux/arm64` + writes `org.compass.guest.layer.Image` and no `bzImage` key. +- `tools/microvm-boot-test/run-core.test.ts`: `guestImageEnv` for + `linux/arm64` sets `COMPASS_TEST_GUEST_KERNEL` to `/Image`. +- `go/internal/runtime/microvm/launch_cmdline_test.go`: `serialConsole` + returns `ttyS0` for `amd64` and `ttyAMA0` for `arm64`, and errors for + `riscv64`. `vmmArgs` given `ttyAMA0` carries `console=ttyAMA0` and no + `ttyS0`. +- A new `guest-image/module-check-tests.nix`, built by the + `compass-guest-image:test` task: a config with all eight symbols `=m` or + `=y` passes; one symbol `is not set`, or absent, fails with + "BUILD-BREAK". +- `compass-guest-image:eval-arm64`: the kernel-target assertion passes at + the pin, and fails when its expected map swaps the two names. + +### P8 — Runner volume root below the session mount + +`renderRunnerManifests` (`tools/runner-manifests/render-core.ts`) mounts +`hostPaths.sessionVolumeRoot` at `SESSION_MOUNT_PATH` +(`/var/lib/compass/sessions`) and sets `COMPASS_MICROVM_VOLUME_ROOT` to +`SESSION_MOUNT_PATH` too, with `COMPASS_MICROVM_QUOTA_REQUIRED=true`. +`mountRoot` (`go/internal/runtime/microvm_quota_linux.go`) walks up until +`st_dev` changes. In the pod, that boundary is the top of the hostPath bind +mount. So for the volume root, `mountRoot` returns `distinct=false`, and +`readVolumeQuota` fails startup: "point --microvm-volume-root (or +$COMPASS_MICROVM_VOLUME_ROOT) at a SUBDIRECTORY of the quota'd filesystem +rather than at its mount point". Every rendered Runner fails its preflight. + +The fix is a subdirectory. A new const `SESSION_VOLUME_ROOT` is +`${SESSION_MOUNT_PATH}/volumes`, and `COMPASS_MICROVM_VOLUME_ROOT` takes it. +The mount and the hostPath (`type: "Directory"`) are unchanged. + +The quota also moves down. The kernel projects the quota into `statfs` only +for an inode that has the project-inherit flag: `xfs_fs_statfs` +(`fs/xfs/xfs_super.c`) checks `XFS_DIFLAG_PROJINHERIT`, and `ext4_statfs` +(`fs/ext4/super.c`) checks `EXT4_INODE_PROJINHERIT`. If the hostPath itself +carried the project, its root would read projected totals, and the check +would compare the project with itself. So the operator puts the project on +`/volumes` and leaves `sessionVolumeRoot` without one. +The renderer cannot create or check the subdirectory. A missing one fails +closed: `readVolumeQuota` returns `statfs "": …`, as +`TestReadVolumeQuotaAbsentPath` requires. + +The alternative keeps the volume root at the mount and makes the hostPath +the quota'd directory. It does not work: the volume root is still the top of +the mount, so `distinct` stays false. Any layout that passes needs an +unprojected directory at the top of the mount and the project below it. + +The README's "Other node prerequisites" bullet for the session tree says to +put the project quota on `/volumes`, owned by uid and gid +65532, and not on `sessionVolumeRoot`. + +Tests, red first, in `tools/runner-manifests/render-core.test.ts`: + +- "uses runtime settings consumed by compass-runner": + `COMPASS_MICROVM_VOLUME_ROOT` is `/var/lib/compass/sessions/volumes`. +- New: the volume root is a strict subdirectory of the `session-volumes` + `mountPath`. The test reads both from the rendered DaemonSet, so it fails + on today's output, where they are equal. +- "mounts only the session tree, runtime tree, and projected token" keeps + `/var/lib/compass/sessions` as the mount path. + +The Runner renderer has no golden file. `tools/server-manifests/golden.yaml` +holds no Runner object, so it does not change. + +### P9 — Runner TLS server name + +An operator's certificate may name only the public host, while +`serverAddr` is the in-cluster Service (D6). The Runner then checks the +certificate against the Service name and fails with `x509.HostnameError` +(see [Ground truth](#ground-truth), Runner TLS name). The Server needs no +change: the door holds one keypair (`loadNetworkTLS`, +`go/server/network_door.go`), and Go's `getCertificate` +(`crypto/tls/common.go`) returns a lone certificate whatever the SNI. + +`compass-runner` gains `--server-name`, default `$COMPASS_SERVER_NAME`. It +sets `tls.Config.ServerName` in `go/internal/runner/tls.go`: the name the +Runner checks on the certificate and sends as SNI. It changes no trust +root, so the certificate must still chain to `--ca` or the host roots. A new +`runner.NewServerClient(caPath, serverName)` builds the client. An empty +`caPath` leaves `RootCAs` nil, which means the host roots, so the setting +also works without `--ca`. A non-empty `caPath` is read with the +`NewCATrustClient` errors. The client keeps TLS 1.3 minimum and HTTP/1.1 +plus HTTP/2. Like the `--ca` client today, it reads no proxy variables. +`NewCATrustClient` keeps its contract and its CLI and e2e callers; both +functions share one transport builder. + +In `go/cmd/compass-runner/main.go`, a new `runnerHTTPClient` replaces the +`--ca` block. It resolves both values with `orEnv`. When both are empty, it +returns `http.DefaultClient`, as `httpClientFor` (`go/cmd/compass/client.go`) +does. `Dial` (`go/internal/runner/runner.go`) already uses that client for a +nil one, so with no server name the behaviour is unchanged. + +The Runner renderer gains an optional `serverName`. It uses the +`isValidEgressHost` grammar, so a port or a URL fails at render time. When +set, the container gets `COMPASS_SERVER_NAME`, placed after +`COMPASS_RUNNER_CA`. When unset, the output is unchanged. In +`tools/runner-manifests/README.md`, "Two optional values:" becomes three. +The new `serverName` bullet says: set it when `serverAddr` is the in-cluster +Service and the certificate names only a public host. + +P9 is independent of P2 and P4. It shares three files in +`tools/runner-manifests/` with P8, but no lines: P8 edits the +`COMPASS_MICROVM_VOLUME_ROOT` entry, the runtime-settings test and the +README's node prerequisites. + +Spec impact: under `## Runner enrollment CLI` in +`docs/specs/product/compass.md`, a new requirement follows the `--ca` +requirement: "`compass-runner --server-name` sets the name checked on the +Server's certificate". The Runner SHALL check the certificate against that +name instead of the `--server` host, and SHALL trust the same roots. When it +is unset, the Runner SHALL check the `--server` host. Its scenario is the +first test below. + +Tests, red first: + +- `go/internal/runner/tls_test.go`, new `TestServerNameOverride`: a + `certgen` certificate for `compass.example.test` only, served by + `startTLSRunnerServer` at `127.0.0.1`, with that certificate as `caPath`. + `NewServerClient(caPath, "compass.example.test")` enrolls. + `NewServerClient(caPath, "")` fails at the handshake with + `x509.HostnameError`. +- `NewServerClient("", "compass.example.test")` has a nil `RootCAs`, that + `ServerName` and `MinVersion` TLS 1.3. A missing or non-PEM `caPath` fails + with the `NewCATrustClient` messages. +- `go/cmd/compass-runner/main_test.go`, new + `TestServerNameFlagAndEnvironmentFallback`, shaped like + `TestAgentBatchingFlagAndEnvironmentFallback`: both values unset returns + `http.DefaultClient`; `$COMPASS_SERVER_NAME` alone gives a client with + that `ServerName`; the flag overrides the environment. +- `tools/runner-manifests/render-core.test.ts`, new + `describe("serverName")` after `describe("serverCA")`: unset renders no + `COMPASS_SERVER_NAME`; set renders it; `""`, `compass.example.test:443` + and `https://compass.example.test` throw "serverName". + +### Interfaces + +```ts +// tools/server-manifests/render-core.ts (P2) +export type RunnerClusterValues = { + name: string; issuer: string; audience?: string; + namespace: string; serviceAccount: string; maxTokenLifetime?: string; + jwksURI?: string; jwks?: string; caFile?: string; +}; +export type ForgeValues = { + host?: string; repos?: string[]; + appId: number; installationId: number; + appKeySecret: string; appWebhookSecret: string; + reviewer?: { appId: number; installationId: number; appKeySecret: string }; +}; +export type ServerDeployValues = { + namespace: string; serverImage: string; natsImage: string; natsStorage: string; + tlsSecretName: string; dsnSecretName: string; natsSecretName: string; + serverSecretsSecretName: string; adminTokenSecretName: string; s3SecretName?: string; + corsAllowedOrigin?: string; otlpEndpoint?: string; + external?: { port: number; loadBalancerClass?: string }; + nodeSelector?: { key: string; value: string }; + toleration?: { key: string; value: string; effect: "NoSchedule" | "PreferNoSchedule" | "NoExecute" }; + forge?: ForgeValues; + runnerClusters: RunnerClusterValues[]; +}; +export function assertServerImageDigest(ref: string): void; +export function parseServerDeployValues(raw: unknown): ServerDeployValues; +export function renderServerManifests(v: ServerDeployValues): ReturnType; +// server args: --listen 0.0.0.0:50052 --tls-cert /etc/compass/tls/tls.crt +// --tls-key /etc/compass/tls/tls.key --state-dir /run/compass/state +// --socket /run/compass/state/server.sock +// --admin-token-file /var/run/compass/admin-token/token +// --runner-clusters /etc/compass/runner-clusters/clusters.yaml +// --secret-provider dotenv:///etc/compass/secrets/server.env [--forge-*] +// files: tools/server-manifests/{render.ts,values.example.json,README.md,moon.yml} + +// tools/runner-manifests/render-core.ts (P3, merged) +RunnerDeployValues.egressAllow?: readonly string[]; +RunnerDeployValues.serverCA?: + | { configMapName: string; secretName?: undefined; key: string } + | { secretName: string; configMapName?: undefined; key: string }; + +// tools/runner-manifests/render-core.ts (P8) +const SESSION_MOUNT_PATH = "/var/lib/compass/sessions"; // unchanged: session-volumes mountPath +const SESSION_VOLUME_ROOT = `${SESSION_MOUNT_PATH}/volumes`; // new: COMPASS_MICROVM_VOLUME_ROOT value +// RunnerDeployValues.hostPaths is unchanged; the operator creates /volumes with the project quota + +// tools/runner-manifests/render-core.ts (P9) +RunnerDeployValues.serverName?: string; // isValidEgressHost grammar; set → env COMPASS_SERVER_NAME after COMPASS_RUNNER_CA + +// tools/runner-image/assert-public-core.ts (P5) +export function publicVerdict(result: ProbeResult, ref: string): { ok: true } | { ok: false; message: string }; +// tools/runner-image/assert-public.ts: bun tools/runner-image/assert-public.ts --ref +// tools/runner-image/retag-core.ts (P5); retag.ts builds every registry argv with these +export function inspectArgv(ref: string): string[]; // ["inspect","--no-creds","--raw","docker://"] +export function copyArgv(sourceRef: string, destRef: string, destAuthFile: string): string[]; +// ["copy","--preserve-digests","--src-no-creds","--dest-authfile",,"docker://","docker://"] +// tools/runner-image/publish-core.ts (P5): signature unchanged, argv moves to the copy pair +export function tagArgs(repo: string, digest: string, sha12: string, authFile: string): string[]; +// tools/agent-image-index/index.ts (P5): authFile leaves every read except the destination probe +function inspectCommand(tag: string, ...args: string[]): string[]; // ["skopeo","inspect",...args,"--no-creds",tag] +function destinationProbeCommand(authFile: string, tag: string): string[]; // ["skopeo","inspect","--raw","--authfile",authFile,tag] +// guardImmutableTag probes :git- with destinationProbeCommand; nothing else calls it +// composeLocalIndex writes "{}" to /anonymous-auth.json and passes it to every `podman manifest add` +// tools/guest-image/publish.ts, tools/gateway-image/publish.ts (P5): +// pre-push probe ["inspect",...auth,"--raw","docker://"], auth = ["--authfile",REGISTRY_AUTH_FILE] or [] +// copy ["copy","--src-no-creds",...destAuth,...,"oci:","docker://"], destAuth = ["--dest-authfile",REGISTRY_AUTH_FILE] or [] +// re-read ["inspect","--no-creds","--raw","docker://"] +// agent-image/publish.sh (P5): guard_immutable keeps `inspect --raw --authfile "$REGISTRY_AUTH_FILE" "$dest"` + +// tools/runner-image/retag-core.ts (P6) +export function mainTag(firstParentCount: number, commit: string): string; // "main--" +export async function resolveMainPair(input: { + ancestors: readonly string[]; + probe: Record Promise>; + changedSince: (commit: string) => Promise; + closurePaths: Record; +}): Promise>; +// tools/runner-image/tag-main.ts (P6) +// bun tools/runner-image/tag-main.ts --sha <40-hex> \ +// --server-repo ghcr.io/rigelbuild/compass-server --runner-repo ghcr.io/rigelbuild/compass-runner +// .github/workflows/release.yml: job tag-main-images + +// tools/runner-image/retag-core.ts (P7); Platform comes from build-core.ts +export type ManifestIdentity = + | { kind: "manifest"; manifestDigest: string; configDigest: string } + | { kind: "index"; manifestDigest: string; members: { platform: Platform; digest: string }[] }; +export function manifestIdentity(raw: string): ManifestIdentity; +export function assertCoherent(source: ManifestIdentity, release: ManifestIdentity): void; +// tools/runner-image/build-core.ts (P7); tools/server-image/build.ts imports hostPlatform +export type Platform = "linux/amd64" | "linux/arm64"; +export function hostPlatform(arch: NodeJS.Architecture): Platform; // x64 → linux/amd64, arm64 → linux/arm64, else throws +export function kernelFileName(platform: Platform): "bzImage" | "Image"; +export function kernelImagePath(kernelDir: string, platform: Platform): string; // / +export function buildArgs(outputs: RunnerImageOutputs, platform: Platform): Record; +// buildctlArgs(contextDir, outputs, platform, output) passes its platform to buildArgs +// tools/guest-image/publish-core.ts (P7): replaces ASSET_FILENAMES +export function assetFilenames(platform: Platform): Readonly>; +export function layoutPlan( + assets: Readonly>, + provenance: { revision: string; agentImageDigest: string }, + platform: Platform, +): LayoutPlan; +// tools/microvm-boot-test/run-core.ts (P7) +export function guestImageEnv(outPaths: readonly string[], platform: Platform): Record; +// tools/microvm-boot-test/run.ts (P7): --build-only realises the guest and VMM attrs, +// checks / exists, and exits 0 without `go test` +// tools/agent-image-index/index.ts (P7) +export type PublishInput = { repo: string; sha: string; authFile: string; runnerTemp: string }; +export async function publishImageIndex(input: PublishInput, run: CommandRunner): Promise; +// tools/gateway-image/core.ts (P7): the PLATFORM const becomes a platform parameter +``` + +```go +// go/server/network_door.go (P4, //go:build unix) +func readSuppliedAdminToken(path string) (string, error) // os.OpenFile(path, O_RDONLY|O_NONBLOCK|O_CLOEXEC, 0), f.Stat, io.ReadAll(f) +// checkSuppliedTokenMode is pure; gid comes from info.Sys().(*syscall.Stat_t).Gid. +func checkSuppliedTokenMode(mode os.FileMode, gid uint32, held []int) error + +// go/internal/stack/materialize.go (P7, //go:build unix) +func selectGuestMember(raw []byte, arch string) (digest string, err error) // index → member digest for runtime.GOARCH +func guestKernelName(goarch string) string // "Image" for arm64, else "bzImage"; guestAssets' kernel annotationName + +// go/internal/runtime/microvm/launch.go (P7, //go:build unix) +func serialConsole(goarch string) (string, error) // amd64 → "ttyS0", arm64 → "ttyAMA0", else error +func vmmArgs(cfg BootConfig, consolePath, console string, opts launchOptions) []string // appends "console=" + +// go/internal/runner/tls.go (P9, //go:build unix) +// caPath "" → RootCAs nil (host roots), else the --ca pool with NewCATrustClient's errors; +// serverName "" → Go checks the dialed host, else tls.Config.ServerName; TLS 1.3, HTTP/1.1 and HTTP/2 +func NewServerClient(caPath, serverName string) (*http.Client, error) + +// go/cmd/compass-runner/main.go (P9, //go:build unix): flag --server-name, default $COMPASS_SERVER_NAME +// orEnv on both values; both empty → http.DefaultClient, else runner.NewServerClient +func runnerHTTPClient(caFlag, serverNameFlag string) (connecthttp.HTTPClient, error) +``` + +### Deferred follow-ups + +- **F1 — Door reloads its TLS keypair (Go).** Taken if renewals must not + restart the Server. Tests: after a rewrite the next handshake presents the + new serial; a corrupt rewrite keeps the old one. + +```go +// go/server/network_door.go +func newCertReloader(certFile, keyFile string, log *slog.Logger) (*certReloader, error) +func (r *certReloader) GetCertificate(*tls.ClientHelloInfo) (*tls.Certificate, error) +``` + +- **F2 — `tokenFile` on a Runner cluster (Go).** Taken when a cluster needs + key rotation without a rollout. Key fetches send `Authorization: Bearer` + with the file's contents, re-read on every fetch. + +```go +// go/internal/auth/runner_clusters.go +RunnerCluster.TokenFile string // yaml:"tokenFile", optional +// go/internal/auth/workload_identity.go +func fetchHTTPS(ctx context.Context, client *http.Client, u, bearer string) ([]byte, error) +``` + +## Tasks + +- [x] P1 — Server image and publish lane, compass PR #2355, after #2357 +- [ ] P2 — `tools/server-manifests` renderer: no Postgres, no init container, + `forge`, placement values, digest-only images (follow-up to compass PR #2363), after P4 +- [x] P3 — Runner renderer `serverCA` and `egressAllow`, compass PRs #2358 + and #2360 +- [ ] P4 — Admin token file from a Secret volume (Go), with the spec change +- [ ] P5 — Anonymous registry reads in every lane, except the authenticated + destination probe; publish fails on a private package +- [ ] P6 — Paired `:main--` tags on the head of every main push, + after P5 +- [ ] P7 — Multi-arch (`linux/amd64`, `linux/arm64`) images in every lane, + index-aware retag and tagger, per-arch guest kernel file, boot modules and + console, arm64 guest build gate, after P6 +- [ ] P8 — Runner renderer sets `COMPASS_MICROVM_VOLUME_ROOT` to + `/var/lib/compass/sessions/volumes`, with the README quota layout; + independent of P2 +- [ ] P9 — Runner `--server-name` (`$COMPASS_SERVER_NAME`) sets + `tls.Config.ServerName`, with or without `--ca`; renderer value + `serverName`, README and spec requirement; independent of P2 and P4