CISA Vulnrichment Git Processor

Overview

Processes CISA Vulnrichment CVE Record Format v5 JSON files into CVEMetadata and related tables. The vulnrichment repository contains ~136K files with both original CNA data and CISA’s ADP enrichments (SSVC decision points, CWE, CVSS).

Data Source

  • Repository: https://github.com/cisagov/vulnrichment.git
  • Format: CVE Record Format v5 JSON (same as cvelistV5)
  • Structure: {YYYY}/{N}xxx/CVE-{YYYY}-{NNNNN}.json
  • Volume: 169,450 files as of 2026-08-06 (the repo is not baked into the container image — it is cloned at runtime on every run)

ID Scheme

Records are stored with source="cisa_adp" and remapped IDs to avoid collision with cve.org records:

  • Stored cveId: CISA-{YYYY}-{NNNNN} (e.g., CISA-2024-0001)
  • Alias: Original CVE-{YYYY}-{NNNNN} stored in CVEAlias table
  • Rationale: Same CVE ID exists in both cvelistV5 (source=cve.org) and vulnrichment (source=cisa_adp) with different enrichment data

Processing Flow

  1. Clone/pull the vulnrichment repository via processor.PullOrClone() (depth processor.IncrementalDepth in ECS, 1 locally)
  2. Check BulkDataDumpTracker (source="vulnrichment_advisory") — skip with NoWork if HEAD SHA unchanged
  3. Walk repository root for CVE-*.json files (year-based subdirectories)
  4. Apply the git delta window (processor.DetectChangedFiles + processor.FilterPaths) — skipped on the first run and under --force
  5. Load existing sourceFileHash map for resume, keyed by the CISA id
  6. Local runs only: resume from the .repo statefile checkpoint, which is rewritten after every batch and deleted on a clean finish
  7. Batch-process files (100 per transaction, with a savepoint around each file):
    • Parse via vulnrichment.ParseRecord() (a thin wrapper over cvelistv5)
    • Remap CVE ID to CISA ID (vulnrichment.CveIDToVVD)
    • Store via vulnrichment.StoreRecord(), which calls cvelistv5.StoreRecordWithSource(..., source="cisa_adp", cveID=CISA-…) and then db.InsertAliases with the original CVE id
  8. Stop before the 60-minute runtime budget expires. If the hard budget cancels an in-flight transaction, roll it back without classifying its files as bad input or counting its writes as committed.
  9. Update the tracker with HEAD SHA only after a complete repository walk. A truncated run leaves the tracker unchanged so the next run continues catching up through the per-file hash resume set.

Key Reuse

  • internal/vulnrichment: CveIDToVVD, StoreRecord, ParseRecord, FileHash, WalkCVEFiles — a wrapper package around internal/cvelistv5 that pins source="cisa_adp" and the CISA id remap
  • internal/cvelistv5: ParseRecord, StoreRecordWithSource, WalkCVEFiles, FileHash, UTF-8 sanitisation
  • internal/processor: PullOrClone, DetectChangedFiles, statefile, IsRunningInECS
  • internal/db: LoadProcessedHashes, WithTx, GetTracker, UpsertTracker, InsertAliases

Failure modes

  • Runtime cancellation: The repository clone takes about 6.5 minutes. A 15-minute budget left little processing time and could expire inside a 500-file transaction. The task now has a 60-minute budget and uses 100-file transactions. Context cancellation rolls the current transaction back cleanly and leaves the tracker unchanged.
  • Catch-up runs are expected: The source still clones at runtime. When a run reaches its soft deadline before scanning the full repository, the next daily run retries the same HEAD and skips files whose hashes are already current.
  • No AI enrichment: no aienrich.Enricher is constructed, so the CWE / ATT&CK / TreeSitter passes never run for cisa_adp records.
  • Per-file SQL failures: Each file has a savepoint. A PostgreSQL error rolls back that file while allowing the rest of the transaction to continue.

Data Mapping

All CNA + ADP containers are stored, including:

  • Descriptions (CVEDescription)
  • References (CVEReference)
  • Metrics (CVEMetric) — CVSS v2/v3.0/v3.1/v4.0 + SSVC “other” metrics
  • Problem Types (CVEProblemType) — CWE IDs from CISA ADP
  • Affected (CVEAffected + CVEAffectedVersion) — CPE applicability

Deployment

  • ECS: go-vulnrichment-git-processor task, daily at 06:00 UTC (cron(0 6 * * ? *))
  • Resources: 256 CPU / 1024 MB (large repo, processed across multiple resume runs)
  • expected_duration_minutes: 60 (scripts/task-manager.toml), which reaches the container as EXPECTED_DURATION_MINUTES; rundeadline.Soft reserves six minutes for the final transaction and shutdown
  • Local: just go-vulnrichment-git-backfill (the recipe unsets EXPECTED_DURATION_MINUTES, so a backfill runs to completion)

S3 path deviations

The generated section below normalises the source slug to cisa-adp; the real prefix is the CVEMetadata.source value cisa_adp (underscore), and the {sha256} slot holds cvelistv5.FileHash, which is SHA-1 (internal/cvelistv5/mapper.go:26-28). Real keys therefore look like cisa_adp/files/{sha1}/2024/0xxx/CVE-2024-0001.json. Quarantine reasons are parse-error, store-error and tx-rollback (main.go:279, 295, 311).

S3 Persistence

  • Archive path: cisa-adp/files/{sha256}/{filename} ✓
  • Quarantine path: failed-feeds/vulnrichment-git-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
  • Failure reasons emitted: parse-error, store-error, tx-rollback

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[vulnrichment-git-processor] PROC -->|success| ARCHIVE[("S3: cisa-adp/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/vulnrichment-git-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.