Fastly Fetch Processor — Design

Status: Live (commentary enricher) Source: Fastly Security Advisories — alias of /blog/category/security/ Type: fetch (HTML scraping, paginated listing) Source slug: fastly (reference + S3 prefix; no CVEMetadata rows) Schedule: Runs weekly on Tuesdays at 06:00 UTC (cron(0 6 ? * TUE *)).

Overview

Fastly has no CVE namespace and no first-party advisory feed; the security blog category is where it says what a widely-exploited CVE means for the traffic it fronts and which of its own services were affected. This processor is therefore a third-party-analysis reference enricher, not an advisory producer — the precedent digitalocean-fetch-processor was modelled on.

CVEMetadataReferences rows that link existing CVE records to Fastly’s commentary posts. The reference attaches to every (cveId, source) tuple in CVEMetadata and is tagged referenceSource="fastly", type="third-party-analysis". CVEs not already present are skipped (counted as cvesUnknown) — a commentary post is too weak a signal to seed a new CVE record on its own.
CRIT candidates with vex_status="not_affected" when a post names a Fastly product that mitigates the upstream CVE (NGWAF, Compute@Edge, Edge Cloud, etc.). Pure commentary posts produce no CRIT.

Without it, the “what did the CDN in front of us do about this CVE” evidence disappears from the reference set, and Fastly coverage vanishes from the CRIT corpus (53 CritRecord rows for provider='fastly' in production — the largest of the three commentary processors).

Data Source

Property	Value
URL	`https://www.fastly.com/security-advisories/`
Pagination	`/blog/category/security/{n}` from page 1 through the latest
Auth	None — fully public
Format	Server-rendered HTML (no RSS, no JSON API)
Identifier shape	None Fastly-issued; CVEs extracted with a `CVE-YYYY-NNNN+` regex
Cadence	A handful of CVE-mentioning posts per year

internal/fastly owns FetchListing / FetchDetail / ParseDetail + ContentHash. ParseDetail returns ok=false for a post with no CVE references — those are counted as postsSkipped and never stored.

Records produced

Table	Rows
`CVEMetadataReferences`	One per `(cveId, CVEMetadata.source)` pair per post — `url` = post URL, `type='third-party-analysis'`, `referenceSource='fastly'`, `title` = post title
`CritRecord`	One per `(cveId, provider=fastly, service, resourceType)` after the in-process drain — only for posts naming a service the CRIT dictionary resolves
`BulkDataDumpTracker`	Freshness row `fastly` (`totalCVEs` = references inserted this run)
`CVEMetadata` / `CVEAlias`	none

Incremental strategy

Freshness gate: the BulkDataDumpTracker row fastly — the run exits NoWork when now - lastProcessedAt < frequency. --force bypasses it.
Soft deadline: when EXPECTED_DURATION_MINUTES is set (ECS only), the post loop stops at expected − 10 minutes. Local backfills unset the variable and run to completion.
Pacing: 1 second between detail fetches.

⚠ Known defect (audit 2026-08-06) — reference inserts are not idempotent. insertCommentaryRefs (main.go:357-368) inserts with a bare ON CONFLICT DO NOTHING, but CVEMetadataReferences has no unique constraint covering (cveId, source, url, referenceSource) — only plain indexes — so the conflict clause can never fire, and inserted is counted without checking RowsAffected. Every weekly run therefore re-inserts every reference: production holds 17,102 rows for 712 distinct (cveId, source, url) triples under referenceSource='fastly' (~24 duplicates each), and the refsInserted stat is inflated by the same factor. digitalocean-fetch-processor solved this with an explicit WHERE NOT EXISTS (…) guard plus tag.RowsAffected() (digitalocean-fetch-processor/main.go:352-369) — port that here, then de-duplicate the existing rows.

CRIT staging

Gated on --emit-crit (false on the CLI, true in the ECS command). The post text is matched against the Fastly product map in crit_mapper.go; a match is resolved against the CRIT spec + extended dictionaries, and one envelope is staged per (cveId, CVEMetadata-source, service-match) with vex_status="not_affected" (the provider sits upstream of the vulnerable component). Because the feed is Fastly-scoped by identity the matcher runs with ImplicitProviders: ["fastly"], so short service tokens (compute, waf) need no in-text “Fastly”.

After the post loop critpublisher.DrainKeys (4 workers, skippable with CRIT_DISABLE_INPROCESS_DRAIN=1) reads the staged envelopes back and upserts CritRecord rows; reviewer-owned columns (approvedBy/approvedAt) are never touched. When the deterministic map staged nothing for a post, offerToCritInference hands it to the critprep inference queue in S3 instead — provided critprep still found a dictionary token in the text. That path writes no database row.

Failure modes

Condition	Behaviour
Listing fetch fails	fatal — `notifier.Errored`, exit 1
Detail fetch / source lookup / insert fails	`RecordError`, `postsFailed++`, loop continues; exit 1 at the end
Post has no CVEs	`postsSkipped++`, nothing written
All posts skipped and nothing inserted	`notifier.NoWork`
CRIT drain errors	`RecordError` (non-fatal)

Flags

Flag	Default	Description
`--force`	`false`	Bypass the tracker freshness gate
`--limit`	`0` (all)	Process at most N posts
`--emit-crit`	`false` (CLI) / `true` (ECS command)	Stage CRIT envelopes and drain them into `CritRecord`

ECS Schedule

Property	Value
Family	`go-fastly-fetch-processor`
Cron	`cron(0 6 ? * TUE *)` — weekly, Tuesdays 06:00 UTC
CPU / Memory	256 / 512 MB
`expected_duration_minutes`	15
Command	`["/app/fastly-fetch-processor", "--emit-crit=true"]`

CloudWatch logs retain 3 days, then archive to S3 (Glacier Deep Archive).

S3 Persistence

Archive path: fastly/files/{sha256}/{filename} ✓
Quarantine path: failed-feeds/fastly-fetch-processor/{YYYY-MM-DD}/{reason}/{filename} ⚠ not yet wired
Failure reasons emitted: fetch-error, parse-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[fastly-fetch-processor] PROC -->|success| ARCHIVE[("S3: fastly/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/fastly-fetch-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.