GitHub PoC Processor — design

Why

PoC discovery used to live in saas/src/services/vdb/vulnProcessor.ts:

• ~Line 1638 — fetch nomi-sec/PoC-in-GitHub raw JSON per CVE
• ~Line 1967 — search projectdiscovery/nuclei-templates per CVE via the GitHub Commits Search API

Both tasks ran inline with every /process call, fetched only URLs (not content), wrote only to CVEMetadataReferences, had no retry/pagination, and were hardcoded to two repos. They could not surface CVE-keyed PoCs sitting in arbitrary repos — academic supplementary code, security researchers’ personal repos, etc.

This processor moves PoC discovery into vdb-manager, archives content to S3, and writes first-class Exploit + ExploitCVE rows so the article surfaces real PoC counts via the existing CveLookup widget.

Subsystems

The binary runs four subsystems sequentially in one process so they share rate-limit budget and the soft deadline.

Highest-signal sources run first so the rate budget is spent on the bulk code search last.

Code search sharding

GitHub Code Search caps each query at 1,000 results with no pagination workaround. A single CVE-2025- query against ecosystems like npm or PyPI saturates that ceiling instantly. Every shard combines three axes so each individual query stays under the cap:

shard = (year, prefix, extension)
years    : current_year+1 → 2015                  (~12)
prefixes : ["CVE-", "GHSA-"]                       (2)
ext      : ~37 entries (scripting + native + markup)
total    : ≈ 888 shards/run

Per-page limit: 10 pages × 100 results = 1,000 hits per shard. Combined with the “stop on first already-processed file” short-circuit, steady-state runs touch only net-new files since the prior run.

Authenticated GitHub search rate limit is 30 req/min. With short-circuit running, a steady-state pass finishes well inside the 90-min window.

Tracker keys

• github_poc — top-level processor lock (8-min concurrency window)
• github_poc_file:{fileKey} — per-file blob-SHA used to skip unchanged files between runs

S3 layout (S3 only — not R2)

github-poc/
  nomi-sec/{cveId}.json             # raw nomi-sec/PoC-in-GitHub blob per CVE
  nuclei-templates/{cveId}.yaml     # raw projectdiscovery template
  code-search/{owner}/{repo}/{path} # raw file content from arbitrary repos

Repo-search hits don’t archive content; the repo URL itself is the artifact.

Source enum values written to Exploit.source

Required environment

• DATABASE_URL (write)
• DATABASE_URL_READ
• GITHUB_PAT
• S3_BUCKET_NAME
• standard AWS credentials (instance role on ECS)
• optional EXPECTED_DURATION_MINUTES (default 90)

Schedule

Daily 04:00 UTC via EventBridge (cron(0 4 * * ? *)). Runs after the NSE processor (03:00) so they don’t compete for GitHub search budget.

Saas deprecation roadmap

1. Deploy this processor; let it backfill (one heavy run, then daily steady state).
2. Replace the two GitHub-fetching tasks in saas/src/services/vdb/vulnProcessor.ts with prisma.exploit.count(...) reads against the new sources. Mark // DEPRECATED: pointing here.
3. After 30 days of green runs, remove the saas tasks entirely.

Verification

just go-github-poc-fetch prod

A successful run prints aggregated counts per subsystem and exits 0. That’s the gate.

S3 Persistence

• Archive: ⚠ Not yet implemented — requires record reconstruction (DB row → canonical JSON).
• Quarantine: ⚠ Not yet implemented — same reason.
• Likely reasons when implemented: fetch-error, enrich-error

This is an enrichment processor; it reads from CVEMetadata rather than ingesting raw feeds, so there is no original payload to archive verbatim. See S3 Persistence Contract § Processors whose unit-of-work is not a file.