Snyk Fetch Processor — Design

Overview

Scrapes the Snyk vulnerability database (security.snyk.io) to ingest SNYK-prefixed vulnerability records. Listing pages are paginated to discover identifiers, then individual detail pages are fetched and parsed via the embedded __NUXT_DATA__ JSON array.

Records are stored under SNYK identifiers (source=snyk). CVE and GHSA identifiers found in each record are stored as aliases.

Feed

Page Structure

Listing Pages

Each listing page at /disclosed-vulnerabilities/{n} contains:

• Anchor tags with href="/vuln/SNYK-*" throughout the SSR HTML (full-page scan, no container regex needed)
• Empty pages (beyond last) return 200 OK with 0 SNYK hrefs, terminating pagination

Detail Pages

Each vulnerability page at /vuln/{SNYK-ID} contains:

• <script id="__NUXT_DATA__"> with a JSON array
• Index 4 of the array is a field map: {"title": 6, "description": 7, ...}
• Indexed values resolve to other array positions holding actual data
• String values throughout the array contain CVE/GHSA aliases, CWE IDs, CVSS vectors, version ranges, and lesson URLs

Parsing

NUXT_DATA Field Map (index 4)

Array Scan (prefix-matched strings)

Storage

No new tables or columns. All tables already exist.

Malware Threat-Actor Enrichment

A Snyk record is treated as a malicious package when its __NUXT_DATA__ malicious boolean is set, or its title equals “Malicious Package”. Such records (isMaliciousPackage=true) qualify for threat-actor attribution under source='snyk'. After a record’s store transaction commits, a first-time-only post-commit pass invokes the shared attribution engine (internal/actorintel), which resolves the malware author from package/repo identity plus upstream registry, GitHub, and Docker Hub lookups. Results land in the generic, shared MalwareThreatActor edge (cveId, cveSource='snyk' → ThreatActor) and MalwareAttribution status tables — not the OSM-only OsmThreat* tables. A (cveId, source) already carrying a MalwareAttribution row is never re-enriched.

The existing backlog of malicious records is cleared by the one-time cmd/malware-actor-backfill (just go-malware-actor-backfill), which selects isMaliciousPackage=true AND source<>'osm' across all malware sources.

Incremental Strategy

On startup, load all cveId values from CVEMetadata where source='snyk' into a map[string]bool. Per listing page: for each SNYK ID, skip if already known and --all/--force is false. Because the skip set is checked before the detail fetch, re-walking an already-ingested listing page costs one listing request and no detail requests.

Page frontier resume. The highest listing page reached is persisted as BulkDataDumpTracker.totalCVEs under source snyk_fetch_progress, and the next run starts at frontier - 5 and paginates forward until a page yields no SNYK hrefs (main.go:149-154).

⚠ Known defect. New Snyk disclosures appear at the front of /disclosed-vulnerabilities (the code comment at main.go:145-147 acknowledges that “new vulns shift older pages forward”). Starting at frontier - 5 therefore skips pages 1 … frontier-6, which is where all new content lands. Production evidence: the frontier row was last updated 2026-08-01, but max(lastFetchedAt) across source='snyk' rows is 2026-03-30 and max(datePublished) is 2026-03-22 — no new record has been stored in four months of weekly runs. Until this is fixed, treat the Snyk corpus as frozen at 3,081 records.

Rate limit: 600-900ms random delay between detail page fetches.

Locking: Uses BulkDataDumpTracker with key snyk_fetch_lock plus a 5-minute heartbeat to prevent concurrent instances. enrichment.CheckLock treats a lock row touched within the last 8 minutes as held.

Flags

Soft deadline

softDeadlineDuration defaults to 110 minutes and is applied unconditionally (main.go:74-78); EXPECTED_DURATION_MINUTES only overrides the value, it does not disable the deadline. The just go-snyk-fetch-backfill recipe unsets that env var, which means a local full-archive backfill is still cut off after 110 minutes rather than running to completion — contrary to the “backfill must not have a deadline” rule in scripts/go-processors/AGENTS.md.

Malware attribution post-pass

malwareactor.PostPass(ctx, pool, "snyk", …) runs after the lock is released (main.go:283), i.e. outside the ingest loop and outside any transaction. See the section below for what it writes.

ECS Schedule

Runs weekly on Saturdays at 04:00 UTC (cron(0 4 ? * SAT *)). CPU: 256, Memory: 512 MB, Expected duration: 120 minutes.

Key Files

Not wired

• S3 archive / quarantine — no s3client.Uploader is constructed, so no raw payload is persisted at any point. This is the only processor in this batch that keeps nothing, and it is recorded as non-compliant in the compliance matrix.
• internal/aienrich — no enricher is constructed, so the CWE / ATT&CK / TreeSitter passes never run for snyk records. Snyk publishes its own CWE and CVSS, so the gap is narrower here than for the vendor-bulletin scrapers.

S3 Persistence

Not used. This processor does not currently archive payloads or quarantine failures to S3. Per the S3 Persistence Contract this is non-compliant — see the compliance matrix for the implementation roadmap.

Expected paths when implemented:

• Archive: snyk/files/{sha256}/{filename}
• Quarantine: failed-feeds/snyk-fetch-processor/{YYYY-MM-DD}/{reason}/{filename}
• Likely reasons: fetch-error, parse-error, store-error