enrich-references Design

Categorizes and enriches CVE reference URLs by fetching metadata from ExploitDB, GitHub, VulnerabilityLab, and other sources.

Overview

  • Batch size: 50 CVEs per run
  • Schedule: Every 15 minutes (cron(*/15 * * * ? *)), 256 CPU / 512 MB, EXPECTED_DURATION_MINUTES=45
  • Soft deadline: EXPECTED_DURATION_MINUTES - 10 minutes (60 min when unset, i.e. local runs); on hit the loop stops between CVEs and the run still reports Completed
  • Concurrency: Lock via BulkDataDumpTracker (source=enrich_references); exits if < 8 min since last run. There is no heartbeat, so a run longer than 8 min can be joined by the next invocation — the per-CVE claim rows make that idempotent rather than duplicative.
  • State: Per-CVE rows in BulkDataDumpTracker (source=enrich_references:{cveId}), claimed up front so a crash mid-batch does not re-serve the same CVEs
  • Allowed prefixes: CVE-, GHSA-, MAL-, GO-, PYSEC-, RUSTSEC-, GSD-, ALBA-, ALSA-, DSA-, DLA-, USN-, MGASA- (configured via allowedPrefixes var in main.go)

Processing Flow

  1. Check lock (enrich_references) — exit 0 if recent
  2. Acquire lock
  3. Find next batch (50 CVEs): unprocessed first, then oldest reprocess
  4. Claim batch (upsert per-CVE tracker rows)
  5. For each CVE:
    • Load all CVEMetadataReferences rows
    • For each reference URL: CategorizeURL → dispatch by subcategory:
SubcategoryReference type setExploit rowS3 upload
exploit-dbexploit (+ referenceSource=EXPLOIT_DB)full Exploit + ExploitCVE with author / date / platform / categoryraw exploit body → exploit-db/{exploitId}.txt, recorded as Exploit.r2Bucket/r2Key/bodyContentHash/fileSize
vulnerability-labexploit (+ referenceSource=VULNERABILITY_LAB, title backfilled)Exploit + ExploitCVE with author / date—
github (exploit/poc)exploitminimal Exploit (source=github), title backfilled from the Gist API when the URL is a gistraw PoC bytes → github/refs/{sha256}-{basename}, then Exploit.r2Key/bodyContentHash/fileSize updated
github (fix)fix—— (commit / PR data is fetched but only warms the GitHub tables)
github (discussion)discussion——
metasploitexploit (+ referenceSource=METASPLOIT)minimal Exploit + ExploitCVE—
packetstormexploit (+ referenceSource=PACKETSTORM)minimal Exploit + ExploitCVE—
attackerkb, vulners, 0day-today, seebugexploitminimal Exploit + ExploitCVE—
shadowserver, sans-isc, shodan, greynoisesightingminimal Exploit + ExploitCVE—
crowdsecsighting (+ referenceSource=CROWDSEC)minimal Exploit + ExploitCVE—
cert, cisa, microsoft, apple, gentoo, redhat, ubuntu, debianadvisory——
  • Update CVEMetadata.lastEnriched

The “minimal Exploit” shape is exploitId = "{subcategory}:{url}" truncated to 255 chars, a synthesised title, originalUrl, and a single-element cveIds JSON array. Sighting-class subcategories get an Exploit row too — the row is the join target for ExploitCVE, and the type=sighting reference is what distinguishes an observation from working exploit code.

URL Categorization

CategorizeURL mirrors url-categorizer.ts: 36 domain patterns + ordered GitHub path patterns. GitHub path patterns (order matters — specific before generic):

  1. projectdiscovery/nuclei-templates → exploit
  2. nomi-sec/PoC-in-GitHub → poc
  3. Repo name contains exploit/poc/cve → exploit
  4. /commit/{sha} → fix
  5. /pull/{n} → fix
  6. /issues/{n} → discussion
  7. /releases/tag/ → fix
  8. /blob/ with code extension → poc
  9. gist.github.com → poc

Tables Written

  • CVEMetadataReferences — type and referenceSource updated in place (and title for vulnerability-lab). Rows are never inserted; this task only classifies references other processors already stored.
  • Exploit — exploit / sighting records, including r2Bucket/r2Key/bodyContentHash/fileSize for the two sources whose content is archived
  • ExploitCVE — CVE-to-exploit junction, tagged with the reference row’s own source
  • CVEMetadata.lastEnriched — enrichment timestamp
  • BulkDataDumpTracker — lock + per-CVE state

Environment Variables

VariableRequiredDescription
DATABASE_URLYesWrite connection
DATABASE_URL_READNoRead replica
GITHUB_PATNoGitHub API token for commit / PR / gist enrichment. Without it the github branch still classifies the reference and creates the Exploit row, but skips the API lookups and the gist title. Supplied in ECS via local.go_task_secrets_github.
S3_BUCKET_NAMENoEnables the exploit-db and github PoC body uploads

S3 content archive

This task has no feed payload — it reads CVEMetadataReferences rows other processors wrote — but it does archive the exploit content it fetches:

PathWhen
exploit-db/{exploitId}.txtAn exploit-db reference resolved to raw exploit content
github/refs/{sha256}-{basename}A github exploit/poc reference resolved to raw file or gist bytes

Both are recorded on the Exploit row (r2Bucket, r2Key, bodyContentHash, fileSize) so the body can be retrieved for offline static analysis. Fetch failures are logged and the reference is left unclassified for the next reprocess pass — nothing is quarantined.

S3 Persistence

  • Archive: ⚠ Not yet implemented — requires record reconstruction (DB row → canonical JSON).
  • Quarantine: ⚠ Not yet implemented — same reason.
  • Likely reasons when implemented: fetch-error, enrich-error

This is an enrichment processor; it reads from CVEMetadata rather than ingesting raw feeds, so there is no original payload to archive verbatim. See S3 Persistence Contract § Processors whose unit-of-work is not a file.