vulncheck-kev-json-processor

Status: Live Source: VulnCheck commercial API — https://api.vulncheck.com/v3/index/vulncheck-kev (Bearer token) Type: json (page-paginated index endpoint, page= 1-based, 1000 records per page) Source slug: none in CVEMetadata — this processor writes the VulnCheck* tables only. vulncheck is used solely as the S3 archive prefix. Schedule: Runs every 6 hours (cron(0 */6 * * ? *)).

Overview

CISA’s KEV catalog is the well-known one, but it is deliberately narrow: it lists only vulnerabilities CISA has confirmed exploited and that affect US federal systems. VulnCheck’s KEV is several times larger and, crucially, carries the evidence — the XDB exploit entries, the reported-exploitation URLs, and the canary telemetry count — for CVEs CISA never adds. That evidence is what turns “high CVSS” into “being exploited right now”, which is the single strongest input to SSVC and to remediation prioritisation.

Each run pages the index endpoint with limit=1000 and page=1..total_pages (_meta.total_pages, falling back to ceil(total_documents / 1000)), accumulates the records in memory, then upserts each one. It requires VULNCHECK_TOKEN and exits 1 without it.

History. Until 2026-09 the loop sent offset=1000, 2000, …. The index does not accept offset (its pagination parameters are page, limit, cursor and start_cursor), so every request returned page 1 again: a run logging records=6000 for a 5,099-document index had seen the same thousand records six times, and the other four thousand were never fetched at all.

Field names

The feed spells its evidence fields in snake_case and the CISA-shaped fields in camelCase. The record struct reads the feed’s spellings and still accepts the camelCase variants it used to expect:

History. Until 2026-09 the struct read dateAdded, requiredAction, xdb, reportedExploitation and reportedExploitedByVulnCheckCanaries. None of those keys exist in the feed, so every one decoded to its zero value: every row’s dateAdded was the run clock, requiredAction was always NULL, no XDB or reported-exploitation row was ever written by this code, and the canary count was always zero.

The unit of identity upstream is not a CVE. A KEV record is keyed by (vendorProject, product, vulnerabilityName) and may reference zero, one, or several CVEs — so the CVE link is a junction table, not a column.

Records produced

Why NULL keys mattered. Postgres treats NULLs as distinct in a unique index. Writing NULL for an empty vulnerabilityName (common in the feed) meant ON CONFLICT never matched that record and every six-hourly run inserted it again: production held ~19k VulnCheckKEV rows for a ~5k-record feed. scripts/dedupe-vulncheck-kev.sql collapses the duplicates; run it once after deploying the fixed processor.

No CVEMetadata, CVEAlias, or Kev rows are written. The CISA Kev table belongs to cisa-kev-json-processor; this processor’s data lives entirely in the VulnCheck* namespace and is joined out to CVEs through VulnCheckKEVCVE.

Legacy rows under source='vulncheck'. Production holds 1,667 CVEMetadata rows with source='vulncheck' and dataVersion='vulncheck-kev-v1', last written 2026-02-28. No current Go code writes them — they are the output of a retired earlier implementation. Do not attribute them to this processor.

CVE existence check

Before inserting a VulnCheckKEVCVE row the processor asks the read replica whether the CVE exists at all:

SELECT "source" FROM "CVEMetadata" WHERE "cveId" = $1 ORDER BY "source" LIMIT 1

If the CVE is unknown to the database, the junction row is silently skipped — so a KEV record for a CVE we have not ingested contributes exploitation evidence that nothing can reach. The ORDER BY makes the recorded source stable; without it the same (kevUuid, cveId) collected one junction row per source across runs and the per-vuln endpoint listed the KEV entry once for each.

Date parsing

parseDateMs tries RFC3339 (with or without fractional seconds), then 2006-01-02T15:04:05, then 2006-01-02, and returns nil on failure. A nil date_added keeps the stored dateAdded on conflict and only falls back to the run clock on first insert. It must never overwrite with the run clock: that is how every row came to carry a fetch timestamp, and how the KEV Watch trend chart showed a thousand VulnCheck “additions” in a single week. XDB and reported-exploitation rows are insert-only, so their fallback to the run clock is harmless.

Freshness

The tracker hash is a digest of the fetched records, sorted, so the same set hashes the same regardless of arrival order. It used to digest _meta.timestamp, which is the time the response was generated and therefore differs on every call: the “index unchanged” skip had never once fired. Only a complete fetch is compared or checkpointed; a run that lost a page or stopped at the soft deadline writes an empty hash.

Failure modes

• No token — exits 1 immediately.
• Index fetch failure — the first page is fatal: notifier.Errored then exit 1.
• Page fetch failure — each page is retried by internal/retry (5-minute elapsed budget, 30 s max interval, fresh 60 s request context per attempt, because the VulnCheck endpoint is HTTP/2 and intermittently stalls before sending headers). After 3 consecutive page failures the run aborts with exit 1.
• Per-record upsert failure — recorded through notifier.RecordError, the record is quarantined to S3, and the run continues; a non-zero error count makes the run exit 1 at the end.
• Lost page — a page that fails after its retries is skipped, the run is marked incomplete, and the tracker hash is left empty so the next run re-fetches everything. The records that did arrive are still upserted.
• Soft deadline — applies to scheduled runs only (EXPECTED_DURATION_MINUTES set); a local backfill has no deadline and runs to completion. A deadline hit during pagination or upsert also leaves the run incomplete, as above.

S3 Persistence

• Archive path: vulncheck/files/{sha256}/{filename} ✓ — one canonical JSON file per KEV record after a successful upsert (cmd/vulncheck-kev-json-processor/main.go:238). {filename} is {firstCVE}.json, falling back to a sanitised {vendorProject}_{product}_{vulnerabilityName}.json (main.go:266-285), so the same record always produces the same key.
• Quarantine path: failed-feeds/vulncheck-kev-json-processor/{YYYY-MM-DD}/{reason}/{filename} ✓ (main.go:228)
• Failure reasons emitted: store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.

Processing flow

Data model