Source ingestion

pl up now packages the local checkout before creating a deployment, then uploads its source to PUT /v1/deployments/{id}/source. The CLI prints a verified source receipt, not a built/deployed URL. Isolated build execution, registry publication/provenance, application runtime, ingress and production authentication remain unfinished. Builder source download exists on the agent gateway (see below), but no build executor consumes it yet.

Enable ingestion with PUSHLEFT_SOURCE_ROOT (a dedicated private local directory; its parent must already exist) and PUSHLEFT_BUILD_REGISTRY (an administrator-controlled registry origin, with optional port, without a scheme/path). Both settings are required together. The current API still refuses non-loopback binding with development authentication. Storage is not exposed as a static directory or public download endpoint. Shared-UID attackers and concurrent hostile mutation of the checkout/private storage tree are outside this implementation’s filesystem threat model. A production deployment needs a dedicated service identity/private storage mount and durable backup or a verified immutable object-store adapter.

Transport and limits

The wire type is application/vnd.pushleft.source+tar.zstd, with X-Source-SHA256 containing the compressed bytes’ lowercase SHA-256. The server verifies the bytes itself and requires an embedded .pl.yaml byte-for-byte equal to the deployment’s immutable manifest. Inferred manifests are included by the CLI. Git metadata describes the checkout but is not proof that its contents match a commit; the snapshot digest is authoritative, including local changes.

Limits are 64 MiB compressed, 256 MiB expanded, 32 MiB per file, 20,000 files, 240-byte paths and 32 path components. There are two upload slots per API process, acquired before buffering, with a 30-second body deadline. The slot remains owned by blocking decode/store work even if the HTTP future is cancelled. Multi-replica/global rate limiting is still needed for a public service.

Only regular USTAR entries and one bounded-window Zstandard frame are accepted. Symlinks, hardlinks, devices, sparse files, extension headers, traversal, duplicate names, file/ancestor aliases, unsafe mode bits, malformed/truncated archives and trailing frames/data are rejected. No extraction occurs in the API. An eventual isolated extractor must create fresh private files with explicit modes and must not restore archive ownership or other privileged metadata. The CLI normalizes ownership/timestamps and preserves only whether a regular source file is executable.

.gitignore and .plignore are honored. Dependency/build directories, VCS internals and common credential paths/extensions are excluded independently of ignore negations. This filename policy is not secret scanning: credentials embedded in ordinary source/configuration files can still be uploaded. Use secret references, inspect source inclusion, and keep secrets out of repositories; content scanning and a source-inclusion preview remain future DX work. This first transport intentionally refuses symlink/hardlink source files rather than dereferencing them.

Admission, durability and retries

Authenticate, bound the request body and validate the archive/digest/manifest.
Lock the build job then active deployment, verify undispatched provenance, serialize organization storage admission and commit one immutable source_upload_claims row per deployment. This reserves the exact digest/object key, registry repository, byte/file counts and stable snapshot/artifact IDs before any object write.
Store only those exact bytes in a deployment/organization-scoped key using a private staging file, fsync and no-overwrite publication. Existing objects are byte-checked and fsynced on replay. A changed digest is rejected before object publication, not left as an unbounded orphan.
Recheck the exact durable claim, build intent and active deployment; atomically insert the verified snapshot, attach stable source/artifact provenance to the queued untargeted build job, and emit one source_verified event. No capacity, build completion or runtime readiness is inferred.

Cancellation after claim commit or object publication leaves one bounded resumable claim/object. Retrying the same upload resumes the same IDs; different bytes require a new deployment. Teardown winning before final publication prevents snapshot/job publication. Claims stay charged to quota until retention GC releases them (below), so cancelling an upload cannot evade storage admission. Defaults are 1 GiB and 1,000 unreleased claims per organization; PUSHLEFT_SOURCE_MAX_ORG_BYTES and PUSHLEFT_SOURCE_MAX_ORG_CLAIMS configure those administrator limits. Tenant SQL policies permit claim reads but never writes or assertions of verification.

npm run test:source-ingestion runs migrated PostgreSQL claim/quota/replay/lifecycle tests, body-deadline and blocking-cancellation checks, and real CLI-to-HTTP-to-object/SQL probes. cargo test -p pushleft-source covers deterministic packaging, exclusion rules, private immutable storage and hostile archive forms. Neither proves a production builder or deployed application exists.

Retention and verified garbage collection

Migration 0028_source_retention.sql and scheduler src/retention.rs. Admission rows are never deleted. A release is recorded once on the immutable row: source_upload_claims.released_at/release_reason (abandoned, retired, torn_down) and object_deleted_at, source_snapshots.object_released_at, and release_artifacts.image_released_at/image_release_status (deleted, unsupported, failed). The immutability trigger lets only these columns change, and only from NULL. The one exception is that a failed registry status may be retried. Identity changes and DELETE are still refused. Quota counts only claims where released_at IS NULL.

Eligibility:

Abandoned: a claim with no verified snapshot, older than PUSHLEFT_SOURCE_CLAIM_TTL seconds (default 86400), on a deployment that is not in a protected state.
Retired: the deployment is superseded, cancelled or failed, and its updated_at is older than PUSHLEFT_RELEASE_RETENTION seconds (default 604800). It must not be among the environment’s last PUSHLEFT_RETAINED_RELEASES (default 3) successful releases, where successful means a live/superseded deployment with an artifact. These releases are kept for rollback.
Torn down: the deployment is destroyed. It is released immediately.
Always: no application_instances row of the deployment is deploying/ready, and no ingress route of the deployment is publishing/active/draining. Protected states are never released: queued, building, deploying, ready, running, live, degraded and teardown_queued.

The scheduler runs at most one pass per minute, and only when PUSHLEFT_SOURCE_ROOT is set (it must be the same private tree as the API’s). Each pass has two phases:

Mark. One transaction locks eligible deployments/claims FOR UPDATE SKIP LOCKED. This skips an upload that holds the deployment FOR SHARE. The transaction sets the release columns and appends one source_released and/or artifact_released deployment event. No filesystem or registry I/O happens inside it.
Delete, outside DB locks. SourceStore::remove_verified checks the key grammar again and refuses symlinked components, non-regular and hard-linked files. It deletes only bytes that hash to the recorded digest, then fsyncs the parent. A missing object is a no-op. On success it sets object_deleted_at. On a mismatch it refuses and logs, and the next pass retries. Registry manifests are deleted with DELETE /v2/<name>/manifests/<digest> on the snapshot’s recorded repository, using the gateway’s operator credential (PUSHLEFT_BUILD_REGISTRY_USERNAME, PUSHLEFT_BUILD_REGISTRY_PASSWORD_FILE). 2xx and 404 record deleted, 405 (delete disabled) records unsupported, and anything else records failed, which is retried. A digest still referenced by an unreleased artifact is not deleted. Without the credential, images stay marked until one is configured.

A crash between the phases leaves rows marked and bytes present. The next pass deletes them. PUT /v1/deployments/{id}/source on a released claim returns 409 source_retired (“deployment is retired …”), and the CLI prints that message.

npm run test:retention runs the migrated-PostgreSQL eligibility, idempotency, crash and trigger tests, and a stub-registry status test. It then runs the real build E2E. That E2E ends by retiring its release against a delete-enabled registry:2 and checks the following: the object is gone, the manifest returns 404, the rows and events remain, and the quota dropped.

Remaining retention gaps:

Local filesystem store only. An object-store adapter needs the same verify-then-delete contract.
Registry blobs are reclaimed only by the registry’s own offline garbage-collect. Registries with delete disabled record unsupported and keep the image.
Bearer-token registries are unsupported (Basic auth only). Plain HTTP is used only for a loopback registry.
A rebuild that pushes an identical digest into the same repository between the reference check and the DELETE could lose its manifest. A registry-side reference lock is needed.
Claims on deployments stuck in queued/building are never abandoned. A reaper for stale queued deployments is still needed.
Retention settings are global administrator settings, not per organization or plan.
The throttle is in-process. Several scheduler replicas are safe (row locks) but do redundant work.

Builder source delivery

The mTLS agent gateway serves POST /v1/agent/jobs/{id}/source (see docs/agent-gateway.md). It returns the exact archive only to the node that holds the current targeted build_image lease. The node must be a build-pool node with isolated-build capabilities, the deployment must be active, and the claim, snapshot, artifact and plan provenance must agree. Bytes are read and checked against the claimed length and SHA-256 outside database locks, then authorization is repeated before any byte is sent.

Remaining gaps:

Storage is local only. The gateway and API must share one private filesystem tree, with a service identity that can read it. Objects are not replicated, there is no object-store adapter, and the gateway cannot serve from another host.
The gateway does not re-run full archive validation (verify). It trusts the ingest-time verification through the immutable digest, and the digest check catches any later change to the bytes.
No build executor, extractor or registry push consumes the archive yet. Delivery to an agent is not a build.
Slots are per gateway process. There is no global or per-organization download rate limit.
The migrated tests inject the teardown/lease-loss race between the read and the reauthorization by calling those steps directly. No test cancels a live HTTP download during the blocking read.