enrich-nse-fetch Design

Searches GitHub for Nmap NSE scripts that reference CVE/GHSA identifiers, stores them as first-class Exploit records, and creates ExploitCVE junctions linking to all matching CVEMetadata rows including aliases.

Data Source

GitHub Code Search API. The API has a hard 1000-result cap per query (REST docs; the 2023 Blackbird engine did not raise it and no pagination or cursor trick bypasses it), so every configured prefix is uniformly year-sharded. Default prefix set: CVE-,GHSA-,CNVD-,BDU-.

• For each prefix P, emit extension:nse <P>YYYY- for YYYY in 1999 .. current+1, ordered newest-first.
• 4 prefixes × 29 years = 116 shards on the default set. Prefixes whose IDs don’t contain a year (e.g. GHSA-) return 0 results per shard — that’s acceptable overhead; the only scaling constraint is the GitHub PAT search rate limit (30 req/min), which a 40-min soft deadline comfortably absorbs.
• No per-prefix special-casing — adding a new prefix is a one-token config change.

Excludes files that mention “vulners” verbatim (noise source with bulk CVE references, not targeted exploits).

Processing Flow

1. Acquire BulkDataDumpTracker lock (source = "enrich_nse", 8-min TTL) and start a heartbeat goroutine that re-stamps it every 5 minutes, so a 40-minute run cannot expire its own lock
2. Build sharded query list from --prefix values (see Data Source)
3. For each shard (ordered newest-year-first for CVE): issue GitHub Code Search with sort=indexed&order=desc so freshly indexed files come first. Paginate up to 10 pages × 100 results. Break out of the shard on the first already-processed file (SHA match) since everything indexed earlier was seen in a prior run. Also stops early on soft deadline or --limit.
4. Per file:
   • Compare git blob SHA to stored BulkDataDumpTracker entry → skip if unchanged
   • Fetch full file content via GitHub REST API
   • Extract vuln IDs via the configurable --prefix flag (default CVE-,GHSA-,CNVD-,BDU-). CVE- and GHSA- use strict format regexes; any other prefix falls back to PREFIX + 2+ chars + at least one more hyphenated segment, which rejects bare words like CVE-SEARCH.
   • Strip Lua comments → SHA256 hash for content-based dedup (Exploit.bodyContentHash)
   • Get first commit date (≤ 2 API calls: most-recent + last-page). Always fetched, even when the content hash matches an existing Exploit, because first-commit date is per-file — every fork or copy has its own git history.
   • Upload raw (un-stripped) content to S3 at nmap-nse/{owner}/{repo}/{path} (the .nse suffix is trimmed)
   • Upsert Exploit record (source=nmap-nse, platform=network, category=exploit)
   • Resolve CVE IDs + aliases → insert ExploitCVE junctions
   • Mark the file processed in BulkDataDumpTracker under enrich_nse_file:{owner}/{repo}/{path} with the blob SHA
5. Release lock

Files that mention vulners verbatim, and files from which no vuln ID could be extracted, are still marked processed — so they are not re-fetched — but produce no Exploit row.

Deduplication

• Blob SHA (via BulkDataDumpTracker.sha256): skips unchanged files entirely, zero API calls
• Content hash (Exploit.bodyContentHash): SHA256 of comment-stripped content; a match means the file is not counted as a new exploit, though the Exploit row is still upserted with the current path’s metadata

Rate Limiting

• Code Search: 30 req/min authenticated → soft cap at 10 pages, sleep-to-reset if remaining < 5
• REST API (content + commits): 5000 req/hour → max 3 calls per new file (content + up to 2 commit-list pages); existing files skipped
• RateLimitError and AbuseRateLimitError both sleep to the reset instant and retry the same page

Tables Written

• Exploit — one row per distinct NSE script (source=nmap-nse), with bodyContentHash, fileSize, r2Bucket/r2Key, originalUrl (GitHub HTML URL) and sourceArchiveUrl (raw URL)
• ExploitCVE — one junction per (cveId, source) pair resolved from CVEMetadata and its CVEAlias neighbours, so an NSE script naming a GHSA also links the aliased CVE
• BulkDataDumpTracker — the enrich_nse master lock plus one enrich_nse_file:{owner}/{repo}/{path} row per seen file

No CVEMetadata, CVEMetadataReferences or CVEAlias rows are written — this task only reads the alias graph.

Flags

Environment Variables

S3 Persistence

• Archive: ⚠ Not yet implemented — requires record reconstruction (DB row → canonical JSON).
• Quarantine: ⚠ Not yet implemented — same reason.
• Likely reasons when implemented: fetch-error, decode-error

This is an enrichment processor; it reads from CVEMetadata rather than ingesting raw feeds, so there is no original payload to archive verbatim. See S3 Persistence Contract § Processors whose unit-of-work is not a file.