bitnami-git-processor

Status: Live Source: bitnami/vulndb (data/**/BIT-*.json) Type: git (repository baked into the container image at /data/vulndb, refreshed at runtime) Source slug: bitnami Schedule: Runs daily at 05:30 UTC (cron(30 5 * * ? *)), 256 CPU / 512 MB, expected_duration_minutes = 45.

Overview

Bitnami publishes BIT-<component>-<year>-<n> advisories in OSV format for the components it packages into its container images and Helm charts. The value is the mapping from an upstream CVE to the Bitnami-packaged component and its fixed version — the boundary an image scanner needs and which the CVE record itself does not express.

The binary is a thin main over the shared pipeline (internal/processor.Run) with the shared OSV mapper (internal/osv.MapAdvisory):

processor.PullOrClone refreshes /data/vulndb and resolves HEAD. The Containerfile data stage bakes a depth-1 clone plus a .baked-sha file so the FROM scratch image can resolve a SHA without a git binary.
HEAD is compared to BulkDataDumpTracker.sha256 for source bitnami_advisory. Unchanged and no --force → NoWork, exit 0.
processor.DetectChangedFiles narrows the walk to files touched in the last 3 days; full scan on first run (no tracker) or --force.
data/**/BIT-*.json is walked, parsed as OSV JSON, and skipped when CVEMetadata.sourceFileHash equals the file’s SHA-1.
Batches of 200 files per transaction, one SAVEPOINT sp_file per file.

cveId is the first CVE-YYYY-NNNN in aliases[]; the BIT-… id lands in sourceAdvisoryRef. When no CVE alias exists the BIT-… id becomes the primary cveId.

Records produced

Condition	Records
Every stored advisory	`CVEMetadata` (`source="bitnami"`, `sourceAdvisoryRef` = BIT id, `sourceFileHash` = SHA-1 of the file)
Advisory has `summary`/`details`	`CVEDescription`
Advisory has `references[]`	`CVEMetadataReferences` (`referenceSource="Bitnami"`)
Advisory has `severity[]`	`CVEMetric` (Bitnami ships CVSS vectors on most records); plus one derived `containerType="vulnetix"` `cvssV4_0` metric from the description text
Advisory has CWE ids	`CVEProblemType`
Advisory has `affected[]`	`CVEAffected` + `CVEAffectedVersion`
Always (even with an empty alias list)	`CVEAlias` via `db.InsertAliases` — the BIT↔CVE edge plus the same-`cveId` cross-source edges
Affected package resolves to an ecosystem	`Dependency`, `DependencyRegistry`, `PackageVersion`, `PackageVersionCVE`, and `GitHubRepoDependency` when the advisory names a known GitHub repo (`db.EnrichAffectedWithDependency`)
Clean run (no file errors, not cancelled)	`BulkDataDumpTracker` row `bitnami_advisory` — `sha256` = git HEAD
`PIX_INFERENCE_ENABLED` set (ECS only)	`CVEAttackTechnique` / `CVEAttackMitigation` / `CVEAttackDetection` / `CVEAttackD3fend`, `CVEProblemType` (`derivedBy="vulnetix"`), `CVETreeSitter`, `PixLog` via `internal/aienrich`

Resume

Repository level — BulkDataDumpTracker.sha256 (source bitnami_advisory) holds the last cleanly processed HEAD. It is not advanced when any file errored or the run was cancelled.
File level — CVEMetadata.sourceFileHash (SHA-1) loaded once via db.LoadProcessedHashes (5-minute budget) as a skip set.

Failure modes

Symptom	Cause / guard
`repository unchanged, skipping processing`	HEAD matches the tracker. Expected on most runs.
`failed to load resume hashes, processing all files`	Read replica slower than the 5-minute budget; run degrades to a full re-store.
`file errors present, skipping tracker update to retry next run`	Tracker deliberately left stale so the next run retries the failed files.
First-run cost	A fresh database has to store the whole catalogue (~5k advisories); the 45-minute budget assumes an already-populated table plus a 3-day delta.
`soft deadline reached, stopping early`	Budget nearly spent; partial progress commits, tracker not advanced.

Flags

Flag	Default	Purpose
`--force`	`false`	Ignore the tracker SHA and the per-file hash skip set.
`--batch-size`	`200`	Files per transaction.
`--repo`	`/data/vulndb`	Clone path.
`--data-dir`	`<repo>/data`	Override the walked directory.

Local: just go-bitnami-git-backfill (unsets EXPECTED_DURATION_MINUTES, and the AI Gateway vars unless AIENRICH=true).

S3 Persistence

Archive path: bitnami/files/{sha256}/{filename} ✓
Quarantine path: failed-feeds/bitnami-git-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
Failure reasons emitted: parse-error, store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[bitnami-git-processor] PROC -->|success| ARCHIVE[("S3: bitnami/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/bitnami-git-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.