snowflake-fetch-processor — Design

1. Overview

• Purpose: Ingest Snowflake Security Bulletins from HTML pages into CVEMetadata
• Source URL: https://www.snowflake.com/en/why-snowflake/snowflake-security-hub/security-bulletins/
• Owner: Snowflake Inc.
• Licence: Public security disclosures
• Schedule cadence: Runs weekly on Mondays at 06:00 UTC (cron(0 6 ? * MON *)). Source is low-volume, infrequent.
• ECS resources: cpu_units=256, memory_mb=512, expected_duration_minutes=15
• Reads: External HTML listing + detail pages, BulkDataDumpTracker, LoadProcessedHashes
• Writes: CVEMetadata, CVEAlias, CVEDescription, CVEMetadataReferences, CVEProblemType, CVEAffected, CVEAffectedVersion, CVEMetric (only the derived containerType="vulnetix" cvssV4_0 row — Snowflake publishes no vector), Dependency / DependencyRegistry / PackageVersion / PackageVersionCVE (via the shared pipeline’s affected-enrichment), BulkDataDumpTracker, CritRecord (via the CRIT drain), S3 archive + quarantine
• Not wired: internal/aienrich — this producer does not construct an enricher, so the CWE / ATT&CK / TreeSitter passes never run for snowflake records.

2. Source contract (from Phase 0 recon)

2.1 Reachability

• curl -sSI returns 200 OK, no Cloudflare challenge, no bot detection
• Standard browser User-Agent sufficient
• No auth or rate-limiting observed

2.2 Format

• HTML-only — no JSON API, no RSS/Atom feed, no structured data (JSON-LD)
• Listing page: server-rendered HTML with <a> links to individual bulletins
• Individual page: HTML with structured sections using <h4> + <ul>/<li> pattern
• AEM (Adobe Experience Manager) site — __INITIAL_STATE__ contains page structure JSON only, not bulletin content

2.3 Fields per bulletin (from detail page HTML)

2.4 Pagination

• Listing page has pagination controls (2 pages, 14 total results currently)
• No data-page attributes — pagination links are standard <a> tags
• No Last-Modified or ETag freshness signal
• No incremental “since” parameter

2.5 Cadence, volume, identifier, backfillability

2.6 Anti-bot / auth / rate-limit profile

• None observed. Standard HTTP/2 client with browser headers sufficient.
• No Chromium/headless browser required.

3. Architecture diagram

4. Source → DB field mapping

5. Identifier policy

• No bundle suppression needed (each bulletin = 1 CVE)
• No minted IDs needed
• Call db.InsertAliases(ctx, tx, cveID, source, nil, logger) for every record so same-cveId cross-source backfill runs

6. CRIT / VEX

CRIT staging is implemented and is enabled in production: the ECS command is ["/app/snowflake-fetch-processor", "--emit-crit=true"] (terraform/go-schedules.tf:4254), and cmd/snowflake-fetch-processor/crit_mapper.go maps each bulletin to a (snowflake, service, resource_type) candidate resolved against the spec + extended dictionaries. Staged envelopes are registered and drained in-process by critpublisher.DrainKeys (4 workers) at end of run, which upserts CritRecord and writes the CycloneDX VEX sidecar for vex_status=fixed. Bulletins the deterministic map cannot attribute are offered to the CRIT inference queue via critprep.Hook when a dictionary token is still present in the text.

Caveat worth keeping in mind when reading the output: Snowflake bulletins describe client-side connectors/drivers (C/C++, Node.js, Go, .NET, JDBC, Python, PHP PDO), not managed cloud resources, so the resolved service triples are coarse by nature.

7. S3 / source-file archive layout

• Bucket: $S3_BUCKET_NAME (skipped entirely when unset)
• Archive key: snowflake/files/{sha256}/{cveId}.html — content-addressed by the SHA-256 of the raw detail-page HTML, which is the same value stored in CVEMetadata.sourceFileHash
• Payload: raw HTML of the individual bulletin detail page
• Quarantine key: failed-feeds/snowflake-fetch-processor/{YYYY-MM-DD}/{reason}/{cveId}.html
• Quarantine reasons: store-error only. Fetch and parse failures are counted and logged but not quarantined — snowflake.FetchDetail returns an error rather than the response bytes, so there is no payload to persist at those two failure points.

8. Error handling & Slack

• Per-record failure semantics: continue on individual fetch/parse/store errors
• Retry profile: none at the HTTP layer — fetchHTML performs a single GET per URL and a failure increments failed and moves on. Listing page 1 failing is fatal; later listing pages failing terminates pagination silently.
• Fatal: listing page 1 unreachable, or DATABASE_URL unset
• Slack events: Started, Completed, Errored, NoWork
• Stats dict: {"fetched": N, "stored": N, "skipped": N, "failed": N, "critStaged": N}
• Exit code: non-zero when any row failed, so a single transient 5xx from snowflake.com fails the whole scheduled run.

9. Performance

• Concurrency: 1 worker (very small volume, polite to source)
• Per-request timeout: 30s
• Rate-limit pacing: 1s delay between detail page fetches
• Freshness gate: before fetching anything, the run compares now - BulkDataDumpTracker.lastProcessedAt for source snowflake against that row’s frequency (seconds) and exits NoWork when still fresh. --force bypasses it.
• Soft deadline: EXPECTED_DURATION_MINUTES - 10min (main.go:138); unset by the justfile recipe so backfill runs to completion
• Resume strategy: db.LoadProcessedHashes at startup, keyed by cveId → sourceFileHash (SHA-256 of the raw HTML)
• Tracker sha256: written as the empty string. listingHash is declared and never computed (main.go:182), so the tracker carries no content fingerprint — only lastProcessedAt is meaningful.

10. Backfill

No separate backfill binary warranted. Full archive is ~14 pages, accessible via paginated listing. The scheduled processor with --force covers backfill.

11. Implementation notes

11.1 HTML parsing strategy

The source is AEM-rendered HTML with no JSON API. We use regex-based parsing on both listing and detail pages. This is simpler than the pwno processor’s RSC payload extraction and does not require a headless browser (unlike cert-il).

11.2 Listing page parsing

Extract all href="/en/why-snowflake/snowflake-security-hub/security-bulletins/CVE-YYYY-NNNN/" patterns. Deduplicate URLs. Follow pagination links if present.

11.3 Detail page parsing

Extract fields using targeted regex patterns against the raw HTML:

• cveLinkRe: href="(/en/why-snowflake/snowflake-security-hub/security-bulletins/CVE-\d{4}-\d+)/"
• titleRe: <title>([^<]+)</title>
• pubDateRe: Publication date: (\d{4}-\d{2}-\d{2})
• descriptionRe: <li>(CVE-\d{4}-\d+)&nbsp;-&nbsp;([^<]+)</li>
• cweRe: <li>(CWE-\d+)[^<]*</li>
• cpeRe: <li><b>(cpe:2\.3:[^<]+)</b></li>
• resolutionRe: <h4[^>]*>Resolution</h4>\s*<ul>\s*<li>([^<]+)</li>

11.4 Version range parsing

The CPE line includes version constraints in plain text: (versions >= 0.5.0, < 2.2.0)

Parse this into AffectedVersionData with:

• Version = "0.5.0", Status = "affected", VersionType = "semver", LessThan = "2.2.0"

S3 Persistence

• Archive path: snowflake/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/snowflake-fetch-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.