wolfi-json-processor

Status: Live, but truncating — see Failure modes Source: https://packages.wolfi.dev/os/security.json (Alpine secdb format, ~4.8 MB, no auth) Type: json (single bulk feed) Source slug: wolfi — used as the CVEAffected vendor / ADP org id and the S3 prefix. No CVEMetadata rows carry source='wolfi'. Schedule: Runs daily at 09:00 UTC (cron(0 9 * * ? *)), 512 CPU / 1024 MB, expected_duration_minutes = 60.

Overview

Wolfi is Chainguard’s undistro — the base for cgr.dev/chainguard/* images. A CVE in a Wolfi image is almost always a CVE in an upstream package that Wolfi rebuilt, so the CVE record itself belongs to cve.org / NVD / GitHub. What only Wolfi can tell you is which apk version carries the fix. That is exactly what this processor persists: ADP-style affected/version rows attached to the existing CVE record, so a scan of a Wolfi-based image can be answered with a concrete upgrade target instead of “this CVE exists somewhere in your base image”.

The processor is a thin shim: cmd/wolfi-json-processor/main.go supplies a config and delegates to the shared internal/apksecdb package, which chainguard-json-processor also uses. The feed format and shared storage model are documented once on the APK Secdb Processors page; this page covers what is specific to Wolfi.

Config fieldValue
FeedURLhttps://packages.wolfi.dev/os/security.json
Sourcewolfi
TrackerSourcewolfi_secdb
CollectionURLhttps://wolfi.dev
RegistrySlugwolfi-images
RegistryImageReferencecgr.dev/chainguard/wolfi-base

Records produced

ConditionRecords
Vuln id already has a CVEMetadata rowCVEAffected (containerType='adp', adpOrgId='wolfi', vendor wolfi, product/packageName = the apk name, collectionURL='https://wolfi.dev', CPEs JSON) under the base source, plus CVEAffectedVersion rows for every fixed version
Vuln id has no CVEMetadata row and is CVE-* / GHSA-*a placeholder row via db.EnsureMinimalCVEMetadata under the natural authority (cve.org for CVE-, github for GHSA-) — never under wolfi, because a self-sourced placeholder would have no authority to enrich it later
Every affected writeContainerOriginAdvisory for registry wolfi-images / cgr.dev/chainguard/wolfi-base, via db.UpsertContainerOriginAdvisory
Every affected writedependency enrichment via db.EnrichAffectedWithDependency (own savepoint, errors swallowed by design)
Run that wrote rows with zero errorsBulkDataDumpTracker (source='wolfi_secdb', sha256 = SHA256 of the feed body)

Note cgr.dev/chainguard/wolfi-base is also the image reference used by chainguard-json-processor, so ContainerOriginAdvisory must be split by registryId (wolfi-images vs chainguard-images), not by imageReference. Production currently holds 12,366 wolfi-images rows and 15,988 chainguard-images rows.

Resume

Feed-level only: the SHA256 of the whole security.json body is stored in BulkDataDumpTracker.sha256. An unchanged feed short-circuits the run with NoWork. There is no per-package resume, which is what makes the truncation problem below permanent rather than self-healing.

Failure modes

  • Runs truncate at ~16% of the feed and then mark it done. The apply loop breaks on the soft deadline (internal/apksecdb/runner.go:130-134), but the tracker gate (runner.go:189) only requires AffectedUpserted > 0 && Errored == 0 && ctx.Err() == nil — a deadline-truncated run with no errors still writes the feed hash. Measured on the 2026-08-05 09:00 run: "soft deadline reached, stopping early" processed=235 total=1478 after 50 minutes, affectedUpserted=24976, errored=0, Outcome=success — and the wolfi_secdb tracker now holds exactly that run’s feed hash (10d9831968b7e42a…). Because there is no per-package resume, packages 236-1478 are not ingested until the feed body changes, and when it does the walk restarts at package 0 and truncates in the same place.
  • Hardcoded deadline margin. cmd/wolfi-json-processor/main.go:51 uses EXPECTED_DURATION_MINUTES − 10 rather than internal/rundeadline.Soft. It is correctly gated on the env var (with an explicit comment saying why), so local backfills run unbounded — the AGENTS.md backfill rule is satisfied.
  • Partial success is exit 0. The process exits non-zero only when Errored > 0 && AffectedUpserted == 0 (main.go:111), deliberately, so ECS does not retry a mostly-successful run.

Flags

None. Behaviour is entirely determined by the config in main.go and by EXPECTED_DURATION_MINUTES.

S3 Persistence

  • Archive path: wolfi/files/{sha256}/security.json ✓ — the whole feed body is archived once per successful parse; individual packages have no stable identity outside the bulk feed (internal/apksecdb/runner.go:85)
  • Quarantine path: failed-feeds/wolfi-json-processor/{YYYY-MM-DD}/parse-error/security.json ✓ (internal/apksecdb/runner.go:79)
  • Failure reasons emitted: parse-error

Uses s3client.Uploader from internal/s3client/uploader.go; the processor name is derived as {source}-json-processor inside the shared runner. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[packages.wolfi.dev/os/security.json] --> PROC[wolfi-json-processor] PROC -->|success| ARCHIVE[("S3: wolfi/files/{sha256}/security.json")] PROC -->|parse failure| Q[("S3: failed-feeds/wolfi-json-processor/{date}/parse-error/security.json")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.

Processing flow

flowchart TD START([Start]) --> FETCH[GET security.json, 60s budget] FETCH -->|error| FAIL([Errored → Exit 1]) FETCH --> SHA[SHA256 feed body] SHA --> FRESH{tracker.sha256 == feed hash?} FRESH -->|yes| NW([NoWork — feed unchanged]) FRESH -->|no| PARSE[Parse secdb] PARSE -->|error| Q[Quarantine parse-error → Exit 1] PARSE --> ARCH[Archive feed body to S3] ARCH --> BULK[bulkLookupBaseSources for every unique vuln id] BULK --> LOOP[For each package] LOOP -->|soft deadline| STOP[Stop early] LOOP --> TX[Short per-package tx] TX --> P1[SAVEPOINT per placeholder: EnsureMinimalCVEMetadata] P1 --> P2[SAVEPOINT per vuln: UpsertAffected + InsertVersions] P2 --> P3[EnrichAffectedWithDependency + UpsertContainerOriginAdvisory] P3 --> LOOP STOP --> GATE LOOP -->|all packages| GATE{AffectedUpserted > 0
and Errored == 0
and not cancelled?} GATE -->|yes| TRACK[UpsertTracker wolfi_secdb = feed hash] GATE -->|no| SKIP[Leave tracker stale] TRACK --> FIN([Completed / Errored]) SKIP --> FIN