Integrate CLIPV2 build pipeline

Merged
LU
1y ago
+4896-1070

Summary by cubic

Integrates CLIP v2: builds images with buildah and publishes index-only .clip archives, skipping the runc build flow. Adds end-to-end OCI credentials for private layers so builds and workers can pull v2 image content securely.

  • New Features

    • Added CLIP v2 pipeline: render Dockerfile, build with buildah, push to a build registry, index via CreateFromOCIImage, and skip runc for build containers.
    • V2 Dockerfile rendering with standard pip (no uv-b9); coalesces pip/mamba steps; respects custom Dockerfile FROM and uses provided creds for authenticated base pulls; skips Python install for beta9 base images.
    • Deterministic image IDs updated for v2 (Dockerfile included); separate base image ID for v1.
    • Worker reads container config (env, user, workdir, entrypoint/cmd) from CLIP archive metadata for v2 and writes initial_config.json; caches source image refs; no runtime skopeo.
    • Pull path handles OCI vs S3 .clip formats; canonicalizes v2 index path; S3 fallback if local store miss.
    • Config: default clipVersion=2, new buildRegistry and buildRegistryInsecure flags; improved TLS handling for localhost registries; enable blobcache by default.
    • Ensures /workspace and /volumes exist via the overlay upper layer; hardens overlay mounts with fuse-overlayfs fallback.
    • Pass OCI registry credentials at runtime (Linear 7278): create workspace secrets during build, store on the image, scheduler attaches image_credentials, worker uses them for lazy layer pulls.
    • V2 builds support ENV lines and build secrets (as build-args) in generated/appended Dockerfiles and pass them to buildah.

  • Migration

    • Ensure buildah and skopeo are installed on workers.
    • Set imageService.buildRegistry (and buildRegistryInsecure for localhost) so build/push/index works.
    • Run migration 036_add_image_credential_fields to add image credential secret references.
    • If forcing legacy behavior, set imageService.clipVersion to 1.

[!NOTE]
Adds CLIP v2 image pipeline (buildah + index-only .clip) with v2 runtime support and end‑to‑end OCI credentials, updating worker, scheduler, registry, DB, and APIs.

  • Image build/runtime (v2):
    • Build via buildah, push to buildRegistry, index with CreateFromOCIImage; skip runc command flow for build containers.
    • Render/append Dockerfiles with ENV, ARG secrets, Python/pip/mamba coalescing; skip Python install on beta9 base images.
    • Pull/mount handles OCI index vs S3 archives; canonicalizes v2 index path; caches source refs and derives spec from clip metadata.
  • Credentials & DB:
    • New registry credential framework (pkg/registry/credentials); remove old image/credentials.
    • Create workspace secret for base image creds; persist on image; scheduler attaches image_credentials; worker uses provider for lazy layers.
    • Add columns and migration 036_add_image_credential_fields; repo methods to set/get; secret create/update signatures adjusted.
  • Config & tooling:
    • Default clipVersion: 2; add buildRegistry/buildRegistryInsecure; enable blobcache; adjust TLS/no‑creds behavior in skopeo; create /workspace and /volumes in overlay.
  • Image identity & APIs:
    • V2 image ID hashes Dockerfile + context; extensive tests added.
    • Proto/types add image_credentials; SDK updated.
  • Other:
    • Many unit tests for v2 Dockerfile/rendering, image IDs, lifecycle; dependency updates (containerregistry, docker libs, etc.).

Written by Cursor Bugbot for commit 9935f0970934aa8eddbb25221e9592a707a38785. This will update automatically on new commits. Configure here.