rocky-linux-json-processor

Status: Live Source: Apollo Errata Management API (/api/v3/osv/ + /api/v3/advisories/) Type: json (two paginated JSON feeds from one public API) Source slugs: rocky_linux_osv and rocky_linux_advisory Schedule: Runs daily at 14:30 UTC (cron(30 14 * * ? *)).

Overview

Rocky Linux is a RHEL rebuild, so its security value is not the CVE itself — Red Hat and NVD already carry that — but which Rocky RPM, at which NEVRA, fixes it. That mapping exists nowhere else, and it is what turns “this CVE affects openssl” into “your Rocky 9 host needs openssl-3.2.2-6.el9_5”. The Apollo Errata Management API is the only machine-readable source for it.

Apollo exposes the same corpus through two endpoints with different schemas, and the processor ingests both under separate source slugs so each format’s provenance stays intact:

FeedEndpointSource slugMapperShape
OSV/api/v3/osv/rocky_linux_osvshared osv.MapAdvisorystandard OSV 1.7.0. Rocky puts CVE ids in upstream, not aliases
Advisory/api/v3/advisories/rocky_linux_advisoryrocky.MapAdvisoryproprietary: cves[], affected_products[], packages[] (RPM NEVRAs), fixes[] (Bugzilla tickets)

The two are not redundant. The OSV feed is keyed by the RLSA identifier and carries the machine-readable version ranges; the advisory feed carries the RPM package lists, the affected-product matrix and the Bugzilla fix trail, and emits one CVEMetadata record per CVE listed in the advisory, with the RLSA-… name stored as an alias. Merging them under one slug would make it impossible to tell which representation a given field came from.

A health check against /_/healthz runs first and the process exits non-zero if Apollo is down, so a bad upstream is a visible failure rather than a silent no-op run.

Records produced

ConditionRecords
Every mapped OSV advisoryCVEMetadata (source="rocky_linux_osv", id = RLSA-YYYY:NNNNN) plus CVEDescription / CVEMetadataReferences / CVEMetric / CVEProblemType / CVEAffected / CVEAffectedVersion / CVEAlias via the central store path
Every CVE in a mapped advisoryCVEMetadata (source="rocky_linux_advisory", id = the CVE) plus the same child rows; the RLSA-… advisory name is written as an alias, which materialises an alias-target placeholder row under the same source
S3 upload succeededArtifact (bomFormat = the source slug) + Link (PLAIN_JSON) + a CVEMetadata.fileLinkId back-reference
Every feedone BulkDataDumpTracker row per source, sha256 = hash of (last_updated_at + total)

Resume

Two layers:

  1. Per-advisory content hash — SHA256 of the marshalled advisory JSON, stored in CVEMetadata.sourceFileHash. db.LoadProcessedHashes(source) loads it as cveId → hash; an advisory whose hash is unchanged is skipped.
  2. Feed-level freshness — BulkDataDumpTracker.sha256 over Apollo’s own last_updated_at plus the total count.

--force bypasses the per-advisory hash check.

Flags

FlagDefaultDescription
--forcefalseReprocess advisories even when the content hash is unchanged
--limit0Max advisories to process per feed (0 = unlimited)
--workers5Concurrent advisory-processing workers (advisory feed only)
--feedallWhich feed to run: osv, advisory, or all

Failure modes

  • Apollo down — the health check fails and the run exits 1 before touching the database.
  • Per-record write contention — storeItem retries each record up to 3 times, with UTF-8 sanitisation on SQLSTATE 22021 and a backoff on connection errors. Records that still fail are reported through notifier.RecordError and the run continues. Live runs do hit this: the 2026-08-05 14:30 run logged SQLSTATE 40P01 (deadlock) and 55P03 (lock timeout) on CVEProblemType and on the dependency-enrichment path, and two CVEs failed after all retries.
  • Pagination stops early — both feeds treat “page shorter than the requested limit” as the last page, but Apollo caps page size at 50 while the processor requests 200. The advisory feed therefore stops after page 2 and sees only the newest ~100 of 9,452 advisories per run; the OSV feed stops after page 1 because Apollo reports total == size. This is a known defect, tracked in the processor’s efficacy record — do not read the row counts for these two sources as upstream coverage.
  • Deadline applies even to backfills — the soft deadline is set unconditionally (85 minutes when EXPECTED_DURATION_MINUTES is unset), which contradicts the AGENTS.md “backfill must not have a deadline” rule. A local just go-rocky-linux-json-backfill will stop after ~75 minutes of working time.

S3 Persistence

  • Archive path: rocky-linux/osv/{sha256}/{cveId}.json and rocky-linux/advisory/{sha256}/{cveId}.json — note this deviates from the {source}/files/{sha256}/{filename} contract; the prefix is built at cmd/rocky-linux-json-processor/main.go:566
  • Quarantine path: failed-feeds/rocky-linux-json-processor/{YYYY-MM-DD}/{reason}/{filename} ✓ (cmd/rocky-linux-json-processor/s3.go:29)
  • Failure reasons emitted: parse-error, schema-violation

Uses s3client.NewUploader’s PutBool through a local closure (cmd/rocky-linux-json-processor/s3.go:41-47) rather than the shared Archive / Quarantine helpers. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Apollo API] --> PROC[rocky-linux-json-processor] PROC -->|success| ARCHIVE[("S3: rocky-linux/{feed}/{sha256}/{cveId}.json")] PROC -->|failure| Q[("S3: failed-feeds/rocky-linux-json-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.