Isolated build executor

A build_image job turns one verified source snapshot into one pushed OCI image. It publishes that image’s digest through the lease-fenced publish_release_artifact contract (migration 0021). This document covers the current trust boundary and what is not built yet.

Flow

1. Targeting (crates/pushleft-scheduler/src/builds.rs). A queued build_image job stays untargeted until its source_snapshots row exists. That row exists only after the ingester has verified and stored the exact bytes. The scheduler checks that the job’s provenance (snapshot, digest, plan hash, repository, artifact ID) matches the snapshot. A mismatch fails the job. It then picks an active, schedulable node in the build pool, seen in the last 60 seconds, whose capabilities include {"oci_build":1,"isolated_build":1}. It uses select_node_in_pool with a nil network segment, because build nodes never join tenant networks. It adds project_id, archive_bytes, manifest_yaml and placement to the payload. No credential is ever written to jobs.payload.
2. Lease (crates/pushleft-agent-gateway/src/build.rs). Builds for inactive deployments are not leased. For a build_image lease, the gateway reads the operator registry credential (PUSHLEFT_BUILD_REGISTRY_USERNAME, PUSHLEFT_BUILD_REGISTRY_PASSWORD_FILE, mode 0600). It returns the credential only in the lease response, as payload.registry_auth.
3. Execution (crates/pushleft-agent/src/build.rs). The agent advertises oci_build/isolated_build only when it runs with --buildkit-image <repo@sha256:…>. For each attempt it does the following:
   • It creates a fresh 0700 workspace under --build-root (also 0700). Any leftover from a crashed attempt is removed first.
   • It downloads the archive from POST /v1/agent/jobs/{id}/source. It retries only on 503, with backoff. 409 means the lease is lost. Every other status fails the attempt. It requires the exact media type, Content-Length and identity headers. It bounds the body to the leased archive_bytes (at most pushleft_source::MAX_ARCHIVE_BYTES). It then checks the SHA-256 against the lease payload; the response headers alone are never trusted.
   • It calls pushleft_source::extract, which runs verify and then writes fresh files with explicit 0644/0755 modes.
   • It builds the root Dockerfile with rootless BuildKit (see below). The image is written as an OCI tarball to the container’s stdout.
   • It treats that tarball as untrusted. Only oci-layout, index.json and blobs/sha256/<hex> entries are accepted, and each blob is hash-checked as it is written. Exactly one OCI image manifest is required.
   • The agent pushes the blobs and the manifest by digest itself, using Basic auth. The digest the registry reports (Docker-Content-Digest on the PUT, then a HEAD by digest) must equal the manifest digest. A local image ID is never used.
   • It reports {"image_digest":"sha256:…","image":"<repo>@sha256:…","log":{…}}.
   • A drop guard removes the container and its anonymous volume, the per-build network and the workspace. This happens on success, on failure, and when the attempt future is dropped (lease loss, SIGINT).
4. Publication (gateway complete_job). In the same transaction that records the result and completes the job, and before the lease is cleared, the gateway calls publish_release_artifact(org, job, token, node, artifact, repo@digest). It then appends a build_completed deployment event. If the function rejects the call (wrong repository, a different digest after a prior publication, a stale or non-build node, lost lease), the gateway returns 409 artifact_rejected and commits nothing. A failed attempt appends a build_failed event with the bounded log, and the normal attempt/backoff logic runs. Migration 0024 creates the NOLOGIN role pushleft_agent_gateway with EXECUTE on the function. Operators grant this role to the gateway’s BYPASSRLS login role.

Build logs are a tail of at most 16 KiB with 2 KiB lines. Control characters are stripped. total_bytes, dropped_lines, truncated_lines and truncated report every cut. The log goes into the job result and the deployment event. Logs are not streamed live yet. They appear when the attempt finishes.

Isolation decision

We evaluated two options:

• Dedicated ephemeral build VMs (one microVM or VM per build, destroyed afterwards). These give the strongest boundary, because there is no shared kernel with other tenants. They need a VM provisioner (Hetzner API or Firecracker on bare metal), image boot time and a VM lifecycle. None of that exists yet.
• Rootless BuildKit in a per-build container on dedicated build-pool nodes. This slice implements this option.

The flags come from the BuildKit primary documentation (docs/rootless.md, “Containerized deployment” → Docker). The container runs moby/buildkit:v0.25.1-rootless pinned by index digest sha256:0c0ce4929976ace509f7a53878303504b2de8857829aea4932af9f7325a30f51, through buildctl-daemonless.sh, as UID 1000 inside its own user namespace. It runs with --security-opt seccomp=unconfined, apparmor=unconfined and systempaths=unconfined, which are the documented requirements for rootless BuildKit to create a user and PID namespace and a fresh /proc for each RUN step.

The container never gets:

• --privileged;
• the documented --oci-worker-no-process-sandbox, which would let RUN steps signal or ptrace the daemon;
• the Docker socket;
• host environment or secrets;
• the host network. It gets its own bridge network, pl-build-<job>-<lease>.

Its other limits:

• resources are capped at 2 CPUs, 4 GiB memory without swap, 4096 PIDs and a 30-minute timeout;
• the source is a read-only bind mount;
• the registry credential is never inside the container. Pushing from the agent was chosen so that tenant RUN steps cannot read the push identity from the daemon’s memory or environment.

Residual risk is stated plainly:

• seccomp and AppArmor are off for the BuildKit container, so the host kernel’s user-namespace attack surface is reachable from tenant RUN steps.
• RUN steps share the build container’s network namespace (rootless BuildKit always uses host-mode networking inside that container). They can reach anything the per-build bridge can reach, including the node’s other interfaces and cloud metadata, unless the node firewall blocks it. A build-node egress policy is not implemented.

This is why builds are only targeted to the build pool, which hosts no tenant runtime or databases. Dedicated ephemeral VMs are the upgrade path for hostile multi-tenant builds.

Registry credential: current trust boundary

Per-job, repository-scoped, short-lived registry tokens are not implemented. The gateway hands every build lease the same operator push credential. A build node (that is, the agent process, not the build container) can therefore push to any repository in that registry for as long as the credential is valid. Scoping is enforced only at publication: publish_release_artifact accepts only <snapshot.image_repository>@sha256:<64 hex>, so an image pushed elsewhere can never become a release artifact. It cannot stop a compromised build node from overwriting or adding blobs and tags in other repositories. The upgrade path is a registry token server (distribution auth.token) that mints a push token for repository:<scoped repo>:push,pull, valid for the lease, at lease time.

Registries must accept HTTP Basic auth; the bearer-token challenge flow is not implemented. Plain HTTP is used only for loopback registries (127.0.0.1/localhost, which local fixtures use). Every other registry requires TLS.

Not done or not verified

• Native Linux proof is missing. All runs were on macOS with Docker Desktop (Linux VM, arm64). That shows the flags work, but it says nothing about the host kernel, AppArmor profiles (kernel.apparmor_restrict_unprivileged_userns on Ubuntu 24.04+), cgroup enforcement or escape resistance on a real build node.
• There are no build-node egress firewall, layer cache, multi-platform images, .pl.yaml build section (only the root Dockerfile), live log streaming, provenance attestations or SBOM.
• Builds reserve no fleet allocation. Concurrency is bounded only by the agent’s single serial workload lane. A job targeted to a node that dies stays targeted to that node; no retargeting is implemented.
• Registry garbage collection and retention are not implemented.

Verification

npm run test:build-executor (scripts/build-executor-e2e.sh) first runs the migrated-PostgreSQL scheduler targeting tests and the gateway publication tests. It then runs the real path:

1. The real CLI uploads scripts/fixtures/web-only.
2. The scheduler targets the build to a build node.
3. The real gateway (mTLS, source delivery) leases it.
4. The real agent runs rootless BuildKit and pushes to a per-run registry:2.8.3 with htpasswd auth.

The script checks that the release_artifacts digest equals the registry’s Docker-Content-Digest and the SHA-256 of the manifest body, and that the password appears in no job, result or event. It checks that the workspace, container and network are gone. It also checks that a failing Dockerfile records a build_failed event containing its log and publishes nothing.

Terminal failures and lease eligibility

A build_image lease is granted only while the node is still active, in the build pool and advertising oci_build and isolated_build; a job targeted at a node that no longer qualifies stays queued (it is not re-targeted automatically). A permanent build failure, a retryable failure that exhausts max_attempts, or a final lease that expires without a result marks the deployment failed and appends a build_failed event with "terminal": true; non-terminal failures append build_failed with "terminal": false and leave the deployment building for the retry.