cleanstart-git-processor

Status: Live Source: cleanstart-dev/cleanstart-security-advisories (git) Type: git (repo baked into the image at build time by the cleanstart-git-data stage, then pulled at runtime with pure-Go go-git) Source slug: cleanstart Schedule: Runs weekly on Wednesdays at 03:00 UTC (cron(0 3 ? * WED *)). CPU 256 / memory 512 MB / expected_duration_minutes = 45.

Overview

CleanStart publishes OSV-format advisories for its hardened container image catalogue. This processor is the only ingest path for that catalogue: without it a CleanStart image consumer has no per-CVE record naming the CleanStart package that carries the fix.

Each run:

Refreshes the repo. processor.PullOrClone pulls /data/cleanstart-security-advisories (baked at build time with a .baked-sha). The container’s final stage is scratch, so there is no git binary — the pull runs through go-git.
Short-circuits on an unchanged HEAD. The HEAD SHA is compared against BulkDataDumpTracker.sha256 for source cleanstart_advisory; on a match (and without --force) the run reports task.nowork and exits 0.
Narrows to changed files. With a tracker present, processor.DetectChangedFiles collects paths touched in the last 3 days and the walk is filtered to them. A first run, --force, or a history too shallow to diff falls back to a full scan.
Walks advisories/ for every .json file. CleanStart advisory IDs vary (CSAB-* and others), so no filename prefix filter is applied.
Maps and stores through the shared pipeline (internal/processor/pipeline.go) in transactions of --batch-size (default 200) files, one SAVEPOINT per file so a single bad advisory cannot poison the batch.

osv.MapAdvisory derives the row key: the first CVE-YYYY-NNNN+ alias if the advisory carries one, otherwise the upstream advisory ID. Advisories that OSV flags as malicious-package records are skipped here — oss-malware-git-processor owns those.

Records produced

Condition	Records
Every stored advisory	`CVEMetadata` (`source="cleanstart"`, `sourceFileHash` = file SHA1, `lastFetchedAt` = `UnixMilli`), `CVEDescription` (summary + details), `CVEMetadataReferences` (`referenceSource="CleanStart"`), `CVEProblemType` (upstream `cwe_ids`), `CVEAffected` + `CVEAffectedVersion`
Advisory carries a CVSS vector	`CVEMetric` rows per upstream severity entry
Advisory carries descriptions	one derived `CVEMetric` (`containerType="vulnetix"`, `cvssV4_0`) from `cvss.DeriveV4FromDescription`
Always	`CVEAlias` edges via `db.InsertAliases` — the OSV ID, `aliases[]`, `related[]`, `upstream[]`, plus the same-`cveId` cross-source edges
Every affected package	`Dependency`, `DependencyRegistry`, `PackageVersion`, `PackageVersionCVE` (and `GitHubRepoDependency` when the advisory names a GitHub repo) via `db.EnrichAffectedWithDependency`
Clean run	`BulkDataDumpTracker` row `cleanstart_advisory` (`sha256` = HEAD SHA, `totalCVEs` = rows upserted)

Resume

Two independent layers:

Repo-level — the cleanstart_advisory tracker SHA (skips the whole run).
Per-file — db.LoadProcessedHashes("cleanstart") builds a {cveId → sourceFileHash} skip set from the read replica (5-minute budget). A file whose SHA1 still matches is counted unchanged and never re-upserted. --force bypasses both.

The tracker is not advanced when any file errored, or when the run context was cancelled mid-way, so the next run re-scans instead of skipping on a matching SHA.

AI enrichment

aienrich.NewFromEnv is wired into the pipeline, so when PIX_INFERENCE_ENABLED + an AI-Gateway token are present (always true in ECS) each stored record gets the shared post-batch passes — affected routines, ATT&CK mapping, CWE inference, TreeSitter queries — written outside the batch transaction. Local just runs are inference-free unless AIENRICH=true.

Failure modes

Symptom	Cause
`repository unchanged, skipping processing`	HEAD SHA equals the tracker SHA — normal on a week with no upstream commits
`failed to load resume hashes, processing all files`	read-replica query exceeded 5 minutes; the run continues and re-upserts everything
`soft deadline reached, stopping early`	`EXPECTED_DURATION_MINUTES − 10` elapsed; the tracker still advances only if nothing errored and the context is live
files errored > 0	tracker is left untouched so the next run retries; the run pages only when `notify.Finalize` judges the failures systemic

Flags

Flag	Default	Meaning
`--force`	`false`	Ignore both the tracker SHA and the per-file hash skip set
`--batch-size`	`200`	Files per transaction
`--repo`	`/data/cleanstart-security-advisories`	Clone location
`--data-dir`	(repo + `/advisories`)	Override the advisory directory

Local run: just go-cleanstart-git-backfill (add prod to target the production database read/write pair).

S3 Persistence

Archive path: cleanstart/files/{sha256}/{filename} ✓
Quarantine path: failed-feeds/cleanstart-git-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
Failure reasons emitted: parse-error, store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[cleanstart-git-processor] PROC -->|success| ARCHIVE[("S3: cleanstart/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/cleanstart-git-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.