enrich-nuclei Design

Enriches CVE records with Nuclei template and ExploitDB exploit data.

Overview

• Batch size: 1000 CVEs per run
• Schedule: Every hour (cron(0 * * * ? *)), 512 CPU / 1024 MB, EXPECTED_DURATION_MINUTES=45; soft deadline is budget − 10 minutes (60 min when unset, i.e. local runs)
• Concurrency: Lock via BulkDataDumpTracker (source=enrich_nuclei); exits if < 8 min since last run; lock refreshed every 5 min via heartbeat goroutine during long runs
• State: Per-CVE rows in BulkDataDumpTracker (source=enrich_nuclei:{cveId})
• Allowed prefixes: CVE-, GHSA- (configured via allowedPrefixes var in main.go)

Template path history (why this mattered)

nuclei-templates reorganised its CVE tree from a flat cves/{year}/ into per-protocol directories (http/cves/{year}/, network/cves/{year}/, javascript/cves/{year}/). The old flat path returns 404 for every CVE in existence, and because a 404 is a legitimate “no template” answer the breakage was completely silent: the task kept reporting nucleiFound=0 run after run. Production evidence at the time of the fix — 943,751 CVEs marked processed under the enrich_nuclei: prefix, and zero Exploit rows with source='nuclei', zero CVEMetadataReferences with referenceSource='NUCLEI'.

Because phase 1 permanently claims a CVE the first time it is seen, the ~944k already-claimed CVEs recover only via the phase-2 reprocess pass (oldest-first, 1000/run, hourly), which needs roughly 39 days for a full sweep. If a faster recovery is wanted, clear the enrich_nuclei: tracker rows.

Retry/Backoff

HTTP requests to raw.githubusercontent.com (Nuclei templates) and exploit-db.com (ExploitDB) use exponential backoff:

• Up to 4 retries per request
• Backoff starts at 1s, doubles each attempt, capped at 60s, with up to half-backoff random jitter
• Retries on HTTP 429 (rate limited) and network errors
• Request function rebuilt fresh each attempt (new http.NewRequestWithContext call)

GHSA ID Handling

For GHSA- prefixed IDs, extractYear returns "" (GHSA IDs have no 4-digit year at position 1), so CheckNucleiTemplate is skipped. FetchExploitDBByCVE and LinkExistingExploits still run normally for GHSA IDs.

Processing Flow

1. Check lock (enrich_nuclei) — exit 0 if recent
2. Acquire lock
3. Start heartbeat goroutine (refreshes lock every 5 min)
4. Find next batch (1000 CVEs, CVE- and GHSA- prefixes only): unprocessed first, then oldest reprocess
5. Claim batch (upsert per-CVE tracker rows)
6. For each CVE:
   • CheckNucleiTemplate: HEAD raw.githubusercontent.com/.../nuclei-templates/main/{protocol}/cves/{year}/{cveId}.yaml, probing protocol in http, network, javascript until one returns 200 (skipped entirely for non-CVE IDs, where extractYear yields "")
     • If 200: GET the YAML, upload to nuclei-templates/{cveId}.yaml in S3, then upsert CVEMetadataReferences (type=exploit, referenceSource=NUCLEI) + Exploit (source=nuclei, with r2Bucket/r2Key) + ExploitCVE
     • An error is returned only when every probe failed at the transport level; an honest set of 404s means “no template exists” and is not a failure
   • FetchExploitDBByCVE: GET exploit-db.com/search?cve={cveId} (JSON)
     • For each result: fetch raw file, parse header, upsert CVEMetadataReferences + Exploit + ExploitCVE
   • LinkExistingExploits: query Exploit WHERE cveIds @> [cveId], insert missing ExploitCVE junctions
   • Update CVEMetadata.lastEnriched

Tables Written

• CVEMetadataReferences — inserted (ON CONFLICT DO NOTHING) with type=exploit and referenceSource of NUCLEI or EXPLOIT_DB. The source column carries the CVE’s first known source (MIN(source) from CVEMetadata), so the reference is attached to whichever authority sorts first alphabetically — not to every source that knows the CVE.
• Exploit — exploit records (source=nuclei or source=exploit-db), with r2Bucket/r2Key when S3 is configured
• ExploitCVE — CVE-to-exploit junction
• CVEMetadata.lastEnriched — enrichment timestamp
• BulkDataDumpTracker — lock + per-CVE state (lock row refreshed every 5 min by heartbeat)

S3 content archive

Skipped when S3_BUCKET_NAME is unset. Nothing is quarantined — a fetch failure is logged and the CVE is left for the next reprocess pass.

S3 Persistence

• Archive: ⚠ Not yet implemented — requires record reconstruction (DB row → canonical JSON).
• Quarantine: ⚠ Not yet implemented — same reason.
• Likely reasons when implemented: fetch-error, enrich-error

This is an enrichment processor; it reads from CVEMetadata rather than ingesting raw feeds, so there is no original payload to archive verbatim. See S3 Persistence Contract § Processors whose unit-of-work is not a file.