vulncheck-nvd-json-processor

Status: Live, but not current — see Failure modes Source: VulnCheck commercial backup API — https://api.vulncheck.com/v3/backup/nist-nvd2 (Bearer token; the returned download URL is pre-signed, so it must be fetched without an Authorization header) Type: json (multi-GB ZIP of .json.gz NVD 2.0 chunks) Source slug: vulncheck-nvd Schedule: Runs daily at 08:30 UTC (cron(30 8 * * ? *)), 1024 CPU / 3072 MB, expected_duration_minutes = 300.

Overview

VulnCheck’s NVD2 backup is a mirror of NVD with VulnCheck’s own corrections and backfilled CVSS/CPE data on records NVD left thin. It is the second opinion against nist-nvd: where NVD publishes a CVE with no vector and no CWE, VulnCheck often has one. Holding both sources lets the CVE detail page show a disagreement instead of silently picking one, and internal/db/cvemetadata_lookup.go ranks vulncheck-nvd fourth in the preferred-source chain (cve.org → mitre-cve → nist-nvd → vulncheck-nvd → github) for any processor that needs to attach data to “the” record for a CVE.

Each run:

  1. GET /v3/backup/nist-nvd2 with a Bearer token, take data[0] (newest).
  2. Download the pre-signed ZIP into memory (30-minute HTTP budget).
  3. SHA256 the whole archive. If it matches the tracker, exit via NoWork — this is content-addressed, so a republished backup with a new filename but identical bytes is correctly skipped.
  4. Walk the inner *.json.gz / *.json chunks in archive order, decompress, parse the NVD 2.0 records, and process them in batches of 100.

Records produced

ConditionRecords
Every changed CVECVEMetadata (source="vulncheck-nvd") plus the child rows written by processor.StoreCVESourceDataDeferred — CVEDescription, CVEMetadataReferences, CVEMetric, CVEProblemType, CVEAffected, CVEAffectedVersion, CVEAlias
S3 upload succeededArtifact (bomFormat = "vulncheck-nvd") + Link (PLAIN_JSON, url …/nist-nvd2#{cveId}) + a CVEMetadata.fileLinkId back-reference
Every inner chunkVulnCheckNvdFile — (archiveSha256, filename) with status processing → completed (with cveCount) or failed (with errorMessage)
Deferred, post-commitaffected-package dependency enrichment via processor.RunDeferredEnrichment (separate short transactions, to avoid cross-transaction deadlocks)
Deferred, post-commitAI enrichment via aienrich.RunBatch — affected routines, ATT&CK mapping, CWE inference and TreeSitter queries, plus one PixLog row per pass
End of a completed runBulkDataDumpTracker (source = "vulncheck_nvd", sha256 = archive content hash)

Three-tier resume

TierKeyEffect
ArchiveBulkDataDumpTracker.sha256 = SHA256 of the downloaded ZIPidentical archive ⇒ whole run is a NoWork no-op
FileVulnCheckNvdFile(archiveSha256, filename) + fileSha256a chunk already completed with the same content hash is skipped. Note the key includes the archive hash, so a new archive invalidates every file row and the walk restarts from chunk 0
CVECVEMetadata.sourceFileHash (SHA1 of the record bytes), loaded up-front by db.LoadProcessedHashesan unchanged CVE is skipped without a write. The in-memory set is updated as the run proceeds so later batches skip records already handled

The tracker is only written when the run was not interrupted (ctx.Err() == nil), so an overtime cancel deliberately leaves the archive hash unchanged and the next run re-enters.

Failure modes

  • The run never reaches recent CVEs. The chunks are walked in archive order, which is ascending CVE id — chunk 0 is 1999. AI enrichment is invoked from inside the per-batch function (processAdvisories → enricher.RunBatch), which is exactly the pattern AGENTS.md forbids (“never call RunBatch inside the per-batch store loop — that is what made enrichment starve ingest”). The measured consequence on the 2026-08-05 08:30 run: 4 h 51 m of a 5 h budget spent, processed=600 skipped=19397 errored=3, and "soft deadline reached, stopping early" filesRemaining=177. The 2026-08-06 run was still on chunk 5 (nvdcve-2.0-005.json.gz) after 9 minutes, enriching CVE-2004-xxxx records. Because the file-level resume is keyed by the archive hash and the archive changes daily, every run restarts at chunk 0, so the tail of the archive — every recent CVE — is never examined.
  • Observable effect on coverage. Production holds 354,943 vulncheck-nvd rows, but the newest datePublished is in July 2026 and there are zero August 2026 records. For CVEs published since mid-July, vulncheck-nvd has 1,310 rows against nist-nvd’s 10,500.
  • Hardcoded deadline margin. The soft deadline is EXPECTED_DURATION_MINUTES − 10 rather than internal/rundeadline.Soft. It is correctly gated on the env var, so local backfills are unbounded.
  • Memory. The whole archive and each decompressed chunk are held in memory; the task is sized 3072 MB for that reason. An OOM produces no summary line.

S3 Persistence

  • Archive path: vulncheck-nvd/files/{sha256}/{cveId}.json ✓ — one file per CVE record, uploaded before the DB transaction so no connection is held during network I/O (cmd/vulncheck-nvd-json-processor/main.go:374-381)
  • Quarantine path: failed-feeds/vulncheck-nvd-json-processor/{YYYY-MM-DD}/{reason}/{filename} ✓ (cmd/vulncheck-nvd-json-processor/s3.go)
  • Failure reasons emitted: schema-violation (chunk failed to parse, main.go:271), store-error (per-CVE transaction failure, main.go:465)

Uses s3client.NewUploader’s PutBool through a local closure rather than the shared Archive / Quarantine helpers, which is why the archive key is built by hand. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[api.vulncheck.com /v3/backup/nist-nvd2] --> PROC[vulncheck-nvd-json-processor] PROC -->|success| ARCHIVE[("S3: vulncheck-nvd/files/{sha256}/{cveId}.json")] PROC -->|failure| Q[("S3: failed-feeds/vulncheck-nvd-json-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.

Processing flow

flowchart TD START([Start]) --> ENV{DATABASE_URL +
VULNCHECK_TOKEN set?} ENV -->|no| FAIL([Exit 1]) ENV -->|yes| RESUME[LoadProcessedHashes vulncheck-nvd
5 min budget] RESUME --> INDEX[GET /v3/backup/nist-nvd2] INDEX -->|error| FAIL INDEX -->|empty| NW([NoWork]) INDEX --> DL[Download pre-signed ZIP, no auth header] DL -->|error| FAIL DL --> HASH[SHA256 archive] HASH --> FRESH{tracker.sha256 == archive hash?} FRESH -->|yes| NW2([NoWork — backup unchanged]) FRESH -->|no| LOOP[For each inner .json.gz chunk, in archive order] LOOP -->|soft deadline / overtime cancel| STOP[Stop dispatching, skip tracker update] LOOP --> DEC[Decompress + SHA256 chunk] DEC -->|error| FF1[markFileFailed + RecordError] DEC --> FSKIP{VulnCheckNvdFile completed
with same fileSha256?} FSKIP -->|yes| LOOP FSKIP -->|no| PARSE[parseNVDJSON] PARSE -->|error| FF2[Quarantine schema-violation + markFileFailed] PARSE --> BATCH[Batches of 100 → processAdvisories] BATCH --> P1[Phase 1: extract cveId, per-CVE hash check, S3 upload] P1 --> P2[Phase 2: one tx, SAVEPOINT per record,
StoreCVESourceDataDeferred + Artifact/Link] P2 --> P3[Phase 3: RunDeferredEnrichment] P3 --> P4[Phase 4: aienrich RunBatch] P4 --> BATCH BATCH --> MARK[Mark VulnCheckNvdFile completed] MARK --> LOOP LOOP -->|all chunks done| TRACK[UpsertTracker vulncheck_nvd = archive SHA256] STOP --> FIN TRACK --> FIN[notifier.Finalize]

Data model

erDiagram BulkDataDumpTracker { string source PK "vulncheck_nvd" string sha256 "SHA256 of the archive bytes" int totalCVEs "records processed this run" } VulnCheckNvdFile { string uuid PK string archiveSha256 UK string filename UK int fileIndex string fileSha256 string status "processing | completed | failed" int cveCount string errorMessage bigint processedAt } CVEMetadata { string cveId PK string source PK "vulncheck-nvd" string sourceFileHash "SHA1 of the record bytes" int fileLinkId } Artifact { string uuid PK string bomFormat "vulncheck-nvd" string r2Key } Link { int id PK string url "…/nist-nvd2#{cveId}" string artifactUuid FK } Artifact ||--o{ Link : referenced_by Link ||--o{ CVEMetadata : fileLinkId