Registry outage

The OCI registry (PUSHLEFT_BUILD_REGISTRY) receives every build push and serves every image pull. It is external to Pushleft and must accept HTTP Basic auth (build-executor.md).

What fails, and how

Builds fail closed. The agent pushes blobs and the manifest itself, and requires the registry’s Docker-Content-Digest to equal the manifest digest. A failed push is a failed attempt. The gateway requeues a retryable failure with a 2^attempt-second backoff (2, 4, 8, 16 s) up to max_attempts (5), then marks the deployment failed with a terminal build_failed event. Nothing is published, and no artifact without a pushed digest can exist (publish_release_artifact).
New containers fail. ensure_application runs docker pull <repo>@sha256:... before every create (crates/pushleft-agent/src/application.rs). During an outage new instances and replicas fail and retry, then end in error. Running containers are not affected: they are never re-pulled.
Retention stalls. Manifest deletes fail. release_artifacts.image_release_status becomes failed and is retried every pass (alert registry_image_release_failing).
Credential problems look the same. A wrong or rotated password gives 401 on push, so check the credential before you blame the registry.

Symptoms

build_failures_infrastructure (terminal build failures across several organizations within 30 min), build_duration_slow, and registry_image_release_failing. pl up users see build_failed with a push error in the log tail.

Diagnosis

-- Recent build failures and their reason (the log tail is in the event payload).
SELECT occurred_at, org_id, deployment_id, payload->>'terminal' AS terminal, left(payload#>>'{log,tail}', 300) AS log
FROM deployment_events WHERE kind = 'build_failed' AND occurred_at > now() - interval '1 hour' ORDER BY occurred_at DESC;

-- Builds waiting or retrying.
SELECT status, attempt, count(*) FROM jobs WHERE kind = 'build_image' AND status IN ('queued','leased') GROUP BY 1, 2;

From the build node, run curl -fsS --netrc-file registry.netrc https://<registry>/v2/ (a 0600 netrc with the registry user and password; never put the password on the command line). A 200 means the registry and credential are healthy; a 401 is a credential problem; a timeout or 5xx is the registry.

Actions

Fix the registry or its storage. Pushleft has no secondary registry and no failover (gap).
If the credential was rotated, update both password files and restart the scheduler (secret rotation).
Once /v2/ answers, queued and backed-off builds continue by themselves. Deployments that already reached failed stay failed, by design. Their users run pl up again, which creates a new deployment. There is no in-place retry.

What to tell users

Builds are failing because our image registry is unavailable. Running applications are not affected. Deployments that failed during the outage will not resume by themselves: run pl up again once we confirm the registry is back. New replicas and new deployments cannot start until then.

Verification

A test pl up reaches build_completed and live, and build_failures_infrastructure clears within 30 min.

Do not

Do not insert release_artifacts rows or call publish_release_artifact by hand to “adopt” an image pushed elsewhere. There is no adopt-an-image path, and the scoped-digest check is the provenance guarantee.
Do not switch the agent to a tag, or drop digest pinning.
Do not reset jobs.attempt to get past max_attempts.