ghsa-git-processor

Status: Live Source: github/advisory-database (git) Type: git (runtime clone/pull via pure-Go go-git — no baked data stage) Source slug: github Schedule: Runs daily at 04:30 UTC (cron(30 4 * * ? *)). CPU 512 / memory 2048 MB / expected_duration_minutes = 60.

Overview

The GitHub Advisory Database is the reviewed, ecosystem-keyed advisory corpus behind Dependabot. It is the single richest source of package-level affected ranges (npm, PyPI, Maven, Go, RubyGems, NuGet, Packagist, crates.io, Pub, Hex, Composer, Swift, GitHub Actions) and it is where a large fraction of non-CVE-numbered ecosystem vulnerabilities live. Losing this feed would leave lockfile matching dependent on OSV mirrors and would break every GHSA-* lookup, the GHSA PoC enrichment pass, and the alias bridge between GHSA and CVE namespaces.

Note on the source slug. This processor writes source='github', not 'ghsa'. Rows that appear under source='ghsa' in CVEMetadata are alias-shell placeholders minted by the CVEAlias foreign key when some other processor links a GHSA alias — they are not written here.

Each run:

1. Refreshes the repo. processor.PullOrClone clones (or pulls) /data/advisory-database at depth 1. The final image stage is scratch and carries no baked advisory data, so the clone runs through go-git.
2. Short-circuits on an unchanged HEAD against BulkDataDumpTracker.sha256 for source ghsa_advisory; a match without --force reports task.nowork and exits 0. --start-batch also bypasses this check (an operator resuming an interrupted run).
3. Narrows to changed files with processor.DetectChangedFiles (3-day window) when a tracker exists. --force, a first run, or --start-batch forces a full scan.
4. Walks advisories/ recursively for GHSA-*.json (both reviewed/ and unreviewed/ trees), sorted deterministically so --start-batch is stable.
5. Maps and stores in transactions of --batch-size (default 200), one SAVEPOINT per file.

The mapper override is osv.MapAdvisoryKeepID — the GHSA-xxxx-xxxx-xxxx ID is always the cveId, and any CVE-* alias is written as a CVEAlias edge rather than absorbing the record into a CVE row. Advisories that OSV flags as malicious-package records (including the ossf/malicious-packages mirrors GitHub republishes) are skipped — oss-malware-git-processor owns those.

Records produced

Resume

Three layers, because the repo is ~350k files:

• Repo-level — the ghsa_advisory tracker SHA.
• Per-file — db.LoadProcessedHashes("github") builds a {cveId → sourceFileHash} skip set from the read replica. On production this is the largest such scan in the fleet and has a 5-minute budget; on timeout the run logs failed to load resume hashes, processing all files and re-upserts.
• Batch-level — --start-batch N resumes from the Nth batch of the deterministically sorted path list, bypassing the tracker SHA check.

The tracker write is skipped when the run context was cancelled mid-way, so an overtime run re-scans next time.

Throughput — plan a cold load in advance

Measured on a production backfill (2026-08-06): ~1.1 s per advisory, i.e. 3.8 minutes per 200-file batch, 1,774 batches, ~112 hours for a full pass over 354,736 advisories.

The cost is serial per-record round trips to the RDS write proxy, not CPU. Per stored advisory the store path issues, in order: the CVEMetadata upsert plus six statements for its datePublished cross-source self-heal, an UpdateSourceFileHash, one multi-row insert each for descriptions / references / metrics / problem types, then — per affected entry — an affected upsert, a versions insert, and three single-row upserts (Dependency, PackageVersion, PackageVersionCVE) for each version boundary, plus two savepoint statements. An advisory spanning several ecosystems with many introduced/fixed pairs therefore costs tens of round trips.

Practical consequences:

• With expected_duration_minutes = 60 the soft deadline is 50 minutes ≈ 13 batches. A scheduled run cannot cold-load the catalogue; it only keeps up once the per-file resume skip set is healthy (a skip costs a file read and a SHA1, no round trip).
• A cold or forced load should be driven manually in windows with --start-batch, not left to the schedule.
• If db.LoadProcessedHashes fails, the run logs failed to load resume hashes, processing all files, continues with an empty skip set, and re-upserts everything — turning a routine daily run into the 112-hour path. Check that line before concluding a run is merely slow. The known cause is a missing or invalid CVEMetadata(source, "sourceFileHash") index in the target database.

Alerting

Individual file failures in a 350k-file repo are expected, so this processor does not fail on the first error. After the walk it computes successRate = 1 − errored/processed and pages (task.errored, exit 1) only below 95 %; otherwise it reports task.completed with the rate attached.

AI enrichment

Wired to aienrich. Beyond the shared per-record passes (affected routines, ATT&CK, CWE inference, TreeSitter), GHSA is the only source that triggers the GHSA-specific PoC pass (RunGHSAPoC, its own 180 s per-record budget), which is gated internally on this source slug. Inference is time-boxed against EXPECTED_DURATION_MINUTES, so on a large catch-up run not every stored record is enriched — unreached records keep their CVE row and are simply un-enriched. Local just runs are inference-free unless AIENRICH=true.

Failure modes

Flags

Local run: just go-ghsa-git-backfill (append prod for production; pass START_BATCH= to resume).

S3 Persistence

• Archive path: github/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/ghsa-git-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: parse-error, store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.