cvelistv5-json-backfill

Status: Live (local-only, on-demand — no ECS task definition, no EventBridge schedule) Source: local clone of CVEProject/cvelistV5 (<repo>/cves/**/CVE-*.json) Type: json (CVE Record Format 5.x JSON files on disk) Source slug: cve.org Schedule: none — invoked by hand via just go-cvelistv5-json-backfill.

Overview

One-shot bulk importer for the entire CVE Program record corpus. The CVE Program publishes every CVE record as a CVE Record Format 5.x JSON file in the cvelistV5 git repository; this tool walks a local clone of that repository and stores each record under CVEMetadata.source = "cve.org" using the shared internal/cvelistv5 mapper.

It exists purely to seed history. The ongoing incremental writer for the same slug is mitre-cve-json-processor, which polls the CVE Services API on a schedule. Both write source="cve.org", so a backfill run and a scheduled run are interchangeable for any given record — the backfill is simply the only practical way to load ~300k historical records without hammering the API.

Because the repository is large (hundreds of thousands of files) the tool is deliberately local-only: there is no Containerfile target, no ECS task definition, and no EventBridge schedule. It carries no soft deadline — the justfile recipe unsets EXPECTED_DURATION_MINUTES and the binary never constructs a deadline of its own, so a run always goes to completion (see Backfill must not have a deadline).

Records produced

ConditionRecords
Every changed recordCVEMetadata (source="cve.org", sourceFileHash = SHA of the file) plus the children written by cvelistv5.StoreRecord: CVEDescription, CVEMetadataReferences, CVEMetric, CVEProblemType, CVEAffected, CVEAffectedVersion
Every changed recordCVEAlias — written by cvelistv5.StoreRecord through db.InsertAliases, so an empty alias list still triggers the same-cveId cross-source backfill
Every stored recordS3 archive at cve.org/files/{sha256}/{CVE-YYYY-NNNN}.json
Parse failureS3 quarantine at failed-feeds/cvelistv5-json-backfill/{YYYY-MM-DD}/parse-error/{file}
Store failure or batch rollbackS3 quarantine under .../store-error/{file}

No BulkDataDumpTracker row is written — resume is per-file, not per-run.

source='cvelistv5' is not this processor. The slug is cve.org (main.go:24,26, internal/cvelistv5/mapper.go:21), which in production holds 375,412 rows, 355,966 of them titled — that is the cvelistV5 corpus, co-owned with the scheduled mitre-cve-json-processor (which calls the same cvelistv5.StoreRecord and therefore also writes cve.org). The 360 rows that carry source='cvelistv5' are legacy alias-FK placeholders — dataVersion='5.1', state='PUBLISHED', datePublished=0, no title, no payload, all last touched 2026-04-27 — and no Go processor in the repo writes that string. They are 100% alias-justified shells (see ORCH-02) and must be excluded from any coverage reading.

Resume

Two independent mechanisms, both active by default:

  1. DB-driven per-file resume. db.LoadProcessedHashes("cve.org") returns {cveId → sourceFileHash}. A file whose content hash matches the stored hash is counted as unchanged and skipped. This is the authoritative mechanism and works from any starting point.
  2. Local statefile checkpoint. After each 500-file batch the last processed path is written to {--state-dir}/cvelistv5.state. On restart the walk resumes at that path’s index. The statefile is only consulted when not running in ECS and --force is unset, and it is deleted on a clean run with zero errors.

--force bypasses both.

Batching

Files are processed in batches of 500 inside a single transaction per batch (db.WithTx). Read failures, parse failures and store failures are counted and logged per file; the batch continues.

Known limitation — the per-file loop has no SAVEPOINT around cvelistv5.StoreRecord. PostgreSQL aborts the whole transaction on the first statement error, so one failing record inside a batch causes every subsequent record in that batch to fail too, and the batch commit then rolls the batch back. Sibling processors (cisa-kev-json-processor, gitlab-json-processor, internal/apksecdb) wrap each record in its own savepoint; this one does not yet. A rolled-back batch is quarantined to S3 in full and retried on the next run, so no data is lost — but a single malformed file can cost up to 499 otherwise-good records per pass.

Failure modes

SymptomCause
data directory not found--repo points somewhere without a cves/ subdirectory (or pass --data-dir explicitly)
failed to load resume hashes, processing all filesThe read-replica hash load timed out; the run continues without resume and re-upserts everything (slow but correct)
Entire batch reported as erroredOne record failed mid-transaction — see the savepoint limitation above
%d files errored + exit 1Any per-file failure; the statefile is preserved so the next run resumes

Flags

FlagDefaultDescription
--repo/data/cvelistV5Path to the cvelistV5 clone. The justfile recipe resolves REPO to an absolute path before cd-ing into scripts/go-processors.
--data-dir""Override the data directory (default {repo}/cves)
--state-dir.repoDirectory holding the statefile checkpoint
--forcefalseReprocess every file, ignoring both the sourceFileHash skip and the statefile

Local execution

just go-cvelistv5-json-backfill                 # local DB, pulls the clone first
just go-cvelistv5-json-backfill prod            # .env.production
just go-cvelistv5-json-backfill local true      # NO_PULL=true — skip the git pull

S3 Persistence

  • Archive path: cve.org/files/{sha256}/{filename} ✓
  • Quarantine path: failed-feeds/cvelistv5-json-backfill/{YYYY-MM-DD}/{reason}/{filename} ✓
  • Failure reasons emitted: parse-error, store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[cvelistv5-json-backfill] PROC -->|success| ARCHIVE[("S3: cve.org/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/cvelistv5-json-backfill/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.