almalinux-git-processor

Status: Live Source: AlmaLinux/osv-database (advisories/ALSA-*.json) Type: git (repository baked into the container image at /data/alma-vulndb, refreshed at runtime) Source slug: almalinux Schedule: Runs daily at 03:15 UTC (cron(15 3 * * ? *)), 256 CPU / 512 MB, expected_duration_minutes = 45.

Overview

AlmaLinux publishes its errata (ALSA advisories) as OSV-format JSON. Each advisory names the RPM packages and fixed versions for one or more upstream CVEs, which is the granularity a scanner needs to answer “is this AlmaLinux host affected” — the upstream CVE record alone does not carry AlmaLinux package/version boundaries.

The processor is a thin main around the shared pipeline (internal/processor.Run) and the shared OSV mapper (internal/osv.MapAdvisory):

processor.PullOrClone refreshes /data/alma-vulndb and resolves HEAD. The Containerfile data stage bakes a depth-1 clone plus a .baked-sha file, so the FROM scratch image can still resolve a SHA when git/go-git cannot reach the remote.
The HEAD SHA is compared to BulkDataDumpTracker.sha256 for source almalinux_advisory. Unchanged and no --force → NoWork, exit 0.
processor.DetectChangedFiles narrows the walk to files touched in the last 3 days. A full scan happens on the first run (no tracker) or with --force.
advisories/ALSA-*.json is walked, parsed as OSV JSON, and skipped when CVEMetadata.sourceFileHash already equals the file’s SHA-1.
Batches of 200 files commit in one transaction, one SAVEPOINT sp_file per file so a single bad advisory cannot poison the batch.

cveId is the first CVE-YYYY-NNNN found in aliases[]; when the advisory carries none, the ALSA-… id is used verbatim. Advisories the shared mapper classifies as malicious-package records are skipped (see internal/osv.IsMaliciousPackage).

Records produced

Condition	Records
Every stored advisory	`CVEMetadata` (`source="almalinux"`, `sourceAdvisoryRef` = ALSA id, `sourceFileHash` = SHA-1 of the file)
Advisory has `summary`/`details`	`CVEDescription`
Advisory has `references[]`	`CVEMetadataReferences` (`referenceSource="AlmaLinux"`)
Advisory has `severity[]`	`CVEMetric`; plus one derived `containerType="vulnetix"` `cvssV4_0` metric from the description text
Advisory has CWE ids	`CVEProblemType`
Advisory has `affected[]`	`CVEAffected` + `CVEAffectedVersion`
Always (even with an empty alias list)	`CVEAlias` via `db.InsertAliases` — the ALSA↔CVE edge plus the same-`cveId` cross-source edges
Affected package resolves to an ecosystem	`Dependency`, `DependencyRegistry`, `PackageVersion`, `PackageVersionCVE`, and `GitHubRepoDependency` when the advisory names a known GitHub repo (`db.EnrichAffectedWithDependency`)
Clean run (no file errors, not cancelled)	`BulkDataDumpTracker` row `almalinux_advisory` — `sha256` = git HEAD
`PIX_INFERENCE_ENABLED` set (ECS only)	`CVEAttackTechnique` / `CVEAttackMitigation` / `CVEAttackDetection` / `CVEAttackD3fend`, `CVEProblemType` (`derivedBy="vulnetix"`), `CVETreeSitter`, `PixLog` via `internal/aienrich`

Resume

Two independent resume layers:

Repository level — BulkDataDumpTracker.sha256 holds the last successfully processed HEAD. The tracker is not advanced when any file errored, or when the run was cancelled by the overtime watcher, so the next run retries.
File level — CVEMetadata.sourceFileHash (SHA-1 of the raw file) is loaded once per run via db.LoadProcessedHashes (5-minute budget) and used as a skip set. A resume-load failure is non-fatal: the run degrades to reprocessing every file.

Failure modes

Symptom	Cause / guard
`repository unchanged, skipping processing`	HEAD matches the tracker. Expected on most runs; `--force` overrides.
`failed to load resume hashes, processing all files`	Read replica slower than the 5-minute budget. Non-fatal, but the run does full work.
`file errors present, skipping tracker update to retry next run`	One or more advisories failed to parse or store; the tracker is deliberately left stale. Exit code is 0 unless `notify.Finalize` judged the error rate systemic.
`soft deadline reached, stopping early`	`EXPECTED_DURATION_MINUTES` budget nearly spent. Partial progress is committed; the tracker is not advanced.
Advisories stop moving in ECS	The image bakes a depth-1 clone; a runtime pull that cannot fast-forward falls back to the baked `.baked-sha`. A pinned SHA means data is only as fresh as the last image build.

Flags

Flag	Default	Purpose
`--force`	`false`	Ignore the tracker SHA and the per-file hash skip set; full re-scan.
`--batch-size`	`200`	Files per transaction.
`--repo`	`/data/alma-vulndb`	Clone path.
`--data-dir`	`<repo>/advisories`	Override the walked directory.

Local: just go-almalinux-git-backfill. The recipe unsets EXPECTED_DURATION_MINUTES so a backfill runs to completion, and unsets the AI Gateway vars unless AIENRICH=true.

S3 Persistence

Archive path: almalinux/files/{sha256}/{filename} ✓
Quarantine path: failed-feeds/almalinux-git-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
Failure reasons emitted: parse-error, store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[almalinux-git-processor] PROC -->|success| ARCHIVE[("S3: almalinux/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/almalinux-git-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.