Red Hat CSAF Processor — Design Document

1. Overview

Purpose: Fetch Red Hat CSAF 2.0 documents from both of Red Hat’s distribution trees and store them into the standard CVEMetadata pipeline with zero data loss.

Source: redhat

Advisory categories ingested: RHSA-* (security), RHEA-* (enhancement), RHBA-* (bug-fix). Red Hat publishes all three in the /data/csaf/v2/advisories/ tree; the processor keys each on its tracking ID verbatim and treats them identically downstream (alias linking, VEX enrichment, S3 archival).

Schedule: Daily at 13:30 UTC — cron(30 13 * * ? *)

Timeout: 90 minutes daily / 240 minutes backfill

Resources: 256 CPU units, 512 MB memory

Binary / cmd directory: redhat-csaf-processor

2. Data Sources

Red Hat publishes one CSAF file per RHSA (multi-CVE bundle) in the advisories/ tree, and one VEX file per CVE (portfolio-wide per-product disposition) in the vex/ tree. The archive zip endpoints (/archive_latest.zip) return 404 and are not used.

3. Run Modes

The -tree flag selects which tree(s) to process in a run: both (default), advisories, or vex. Phase A (advisories) runs before Phase B (VEX) so the VEX enrichment pass can find RHSA rows just written.

4. Primary-key Modelling (new pattern)

This is the first processor to use a non-CVE-ID primary key in CVEMetadata.cveId.

• An advisory CSAF writes one CVEMetadata row keyed on the advisory ID (e.g. RHSA-2026:6404, RHEA-2023:7327, RHBA-2024:1234) with source='redhat'.
• Each CVE listed in the advisory is written as a CVEAlias row via data.Aliases — InsertAliases canonicalises the direction. Callers resolve “what Red Hat advisories cover CVE-X?” by querying CVEAlias where aliasCveId='CVE-X' AND primarySource='redhat', then following primaryCveId.
• All multi-vuln aggregation happens in MapAdvisory: descriptions, metrics, references, CWEs, and affected products from every vulnerabilities[] entry union into the one advisory row.
• Affected-product dedup includes version: osv.CalculateAffectedHash(vendor, name, version, pkgName) — Red Hat encodes per-architecture / per-edition variants in product_version branch names like "8 x86_64" and "8 s390x", so omitting version from the key would collapse architectures into a single row.

The composite PK (cveId, source) and the absence of cveId format validation make this safe. The VDB API must consult CVEAlias whenever a user asks about a plain CVE-* identifier.

VEX fallback rows

A VEX document whose CVE has no matching advisory aliases (RHSA, RHEA, or RHBA) creates a standalone CVEMetadata row keyed on the CVE ID (CVE-YYYY-NNNNN, source=redhat) populated from the VEX itself. This captures Red Hat’s “not affected” verdicts for CVEs they don’t ship a fix for. Result: source='redhat' can hold both advisory-keyed and CVE-*-keyed rows. Consumer queries must be aware of this.

Orphan reconciliation: when a later VEX run finds matching advisory rows for a CVE that previously generated a fallback (e.g. Phase A failed transiently and recovered on a subsequent day), the fallback row is deleted in the same transaction before enrichment runs (DeleteVEXFallbackRow). This prevents stale CVE-keyed rows from outliving the advisory ingestion that supersedes them.

5. VEX Enrichment (Phase B)

For every VEX document the processor:

1. Uploads the raw VEX JSON to S3 (redhat-archive/{YYYY-MM-DD}/{cve-id-lowercase}.vex.json) — always.
2. Looks up CVEAlias for the VEX’s CVE ID.
3. If matching RHSAs exist: calls ApplyVexStatusUpdates to set CVEAffected.defaultStatus for every RHSA row’s affected product that matches a VEX product (by name). VEX flag labels themselves (vulnerable_code_not_present, component_not_present, vulnerable_code_not_in_execute_path) all resolve to not_affected — the raw label is preserved only in the S3 VEX artifact.
4. Otherwise: maps the VEX into a full CVESourceData with cveId=CVE-YYYY-NNNNN and stores it via processor.StoreCVESourceData.

Flag → defaultStatus resolution table

6. Product Hierarchy Detection

Red Hat uses the same CSAF branch hierarchy as MSRC:

branches[category=vendor]
  └── branches[category=product_family]    ← family detection (e.g. "Red Hat Enterprise Linux")
        └── branches[category=product_name] ← specific product
              └── branches[category=architecture | product_version | product_version_range]
                    └── product (leaf)      ← leaf product ID, referenced by status/remediations

relationships[] composes leaf products with platforms to form full product names (e.g. openshift-builds-controller-rhel9@sha256:... × Red Hat OpenShift Builds 1.7.1). The processor walks this tree to build:

• productMap: productID → full name
• familyMap: productID → family name (from nearest category=product_family ancestor or via relationship inheritance)
• versionMap: productID → version string (from product_version / product_version_range branches)

No product family names are hardcoded. New Red Hat product families are detected automatically from the tree structure.

7. Database Tables Written

No saas/prisma/models/redhat.prisma — existing tables absorb all Red Hat data. The raw CSAF and VEX JSON is preserved byte-for-byte in S3, so any field not projected into relational columns is always recoverable.

7.1 Container origin attribution (redhat-hardened)

Every successfully stored RHSA (Phase A) and every VEX-fallback CVE row (Phase B) emits a ContainerOriginAdvisory row keyed by the (cveId, source='redhat') composite that points at the Red Hat hardened registry:

The write is delegated to db.UpsertContainerOriginAdvisory (scripts/go-processors/internal/db/container_origin.go), wrapped in a SAVEPOINT inside the parent transaction so a single COA failure never rolls back the CVEMetadata/CVEAffected writes. Failure is logged + counted, not propagated.

These rows surface through the public API at:

• GET /api/vdb/v1/alpine/registries — contributes to the redhat-hardened registry’s advisoryCount, last30Days, last90Days, last365Days.
• GET /api/vdb/v1/alpine/archives — contributes to the monthly sourceBreakdown["redhat-hardened"] count.
• GET /api/vdb/v1/alpine/archive/{YYYY-MM} — advisories with kind: "redhat-hardened" and product lists scoped by loadAlpineAffected (which permits vendor='Red Hat').

No filtering is applied: every advisory the processor ingests writes a row, matching the policy used by alpine-json-processor, chainguard-json-processor, and wolfi-json-processor for their respective registries.

8. Environment Variables

9. Local Invocation

# Daily mode, both trees (default)
just go-redhat-csaf-daily

# Daily mode against production
just go-redhat-csaf-daily TARGET=prod

# Full backfill (all RHSAs + all VEX files — big!)
just go-redhat-csaf-backfill

# Advisories tree only (RHSA CSAFs, no VEX enrichment)
just go-redhat-csaf-daily TREE=advisories

# VEX tree only (enrichment + fallback rows)
just go-redhat-csaf-daily TREE=vex

# Smoke test — first 5 items per tree
just go-redhat-csaf-backfill LIMIT=5

# Force-reprocess even when hashes match
just go-redhat-csaf-daily FORCE=true

10. Concurrency

• 5 worker goroutines per phase (configurable via -workers)
• Shared rate limiter: 300 ms per request ≈ 3 req/sec
• DB writes are single-threaded (one consumer goroutine per phase)
• Soft deadline: softDeadline = now + (EXPECTED_DURATION_MINUTES − 10) (main.go:96), and the workers then stop dispatching at softDeadline − 10 min (main.go:293, main.go:554) — so the effective work window is EXPECTED_DURATION_MINUTES − 20, i.e. 70 minutes on the 90-minute daily budget. -mode backfill sets no soft deadline at all and runs to completion.

11. Idempotency

• All inserts use INSERT ... ON CONFLICT DO UPDATE
• Per-advisory resume uses sourceFileHash (SHA256 of CSAF bytes) via db.LoadProcessedHashes
• Phase B never resume-skips: VEX is always re-applied even when its SHA matches a prior run. Skipping by hash would hide the window where a matching advisory was ingested between two identical VEX revisions, leaving the new advisory’s CVEAffected.defaultStatus unfolded. Enrichment is a cheap idempotent UPDATE; fallback storage uses ON CONFLICT.
• Running the processor twice against the same data produces identical DB state
• Phase A and Phase B each advance an independent BulkDataDumpTracker so one tree can catch up without the other being blocked

12. S3 Key Format

redhat-archive/{YYYY-MM-DD}/{rhsa-id-with-colons-as-underscores}.csaf.json   ← RHSA CSAF
redhat-archive/{YYYY-MM-DD}/{cve-id-lowercase}.vex.json                       ← VEX document

• {YYYY-MM-DD} is the current_release_date (RHSA) or initial_release_date (VEX) — both written to the same dated directory so every artifact related to a particular release day sits together.
• RHSA IDs contain a colon (RHSA-2026:6404); the colon is replaced with _ in the S3 key because S3 tolerates colons but many HTTP clients and URL parsers do not.
• The redhat-archive/* prefix is granted public read via terraform/s3.tf so the files are directly addressable (same scheme as msrc-archive/*).

13. AI Enrichment (aienrich)

The processor initialises aienrich.NewFromEnv once in main() and stores the result in a package-level enricher var consumed by both phases. Wire-in: cmd/redhat-csaf-processor/main.go.

Phase A (RHSA advisories): after storeAdvisoryItem returns true, enricher.RunBatch fires with a single-target payload for the primary CVE: CveID = r.cveData.CveID, Source = redhat.Source ("redhat"), Aliases = r.cveData.Aliases (all sibling CVEs from the same RHSA bundle).

Phase B (VEX): the enriched path only updates status on existing RHSA rows — the primary processor already fired aienrich for the cveID during Phase A, so re-running here would be wasteful. Only the fallback outcome (no matching RHSA → a CVE-keyed row is written) fires enricher.RunBatch. The cveID is extracted from doc.Vulnerabilities[0].CVE (with a doc.Document.Tracking.ID fallback).

Four passes fire per target (no ghsapoc — self-gated to source = "github"):

RunBatch is invoked outside the per-record transaction. See the aienrich overview for the full activity contract, environment variables, and operator runbook.

S3 Persistence

• Archive path: redhat/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/redhat-csaf-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: parse-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.