vercel-security-bulletins-fetch-processor — Design

1. Overview

Purpose: Ingest Vercel security bulletins from https://vercel.com/kb/bulletin into the VDB pipeline.
Source URL: https://vercel.com/kb/bulletin (listing), https://vercel.com/kb/bulletin/{slug} (individual)
Owner: Vercel, Inc.
Licence: Publicly available security disclosures
Schedule cadence: Runs weekly on Mondays at 05:00 UTC (cron(0 5 ? * MON *)). Low volume (currently 3 bulletins) but incident-driven.
ECS resources: cpu_units=256, memory_mb=512, expected_duration_minutes=15
Reads: External HTML pages, db.LoadProcessedHashes (keyed by the primary cveId), db.LoadCanonicalDatePublished (CRIT only)
Writes: CVEMetadata, CVEAlias, CVEDescription, CVEMetadataReferences, BulkDataDumpTracker, S3QueueObject + CritRecord + VEX (CRIT path only), S3 archive + quarantine
Not wired: internal/aienrich; no CVEMetric, CVEProblemType, CVEAffected or PackageVersion rows

⚠ CRIT staging is inert in production. --emit-crit defaults to false (main.go:79) and the ECS command in terraform/go-schedules.tf:4034 passes no flags, so §6 below only describes what a local just go-vercel-security-bulletins-fetch-backfill (whose recipe defaults EMIT_CRIT=true) does. scripts/task-manager.toml declares emit-crit default true, which does not reach the container.

⚠ No freshness gate exists. Despite §2 and §8 below, the binary never calls db.GetTracker; it only writes the tracker at the end of a run (main.go:237). The NoWork Slack event for “tracker is fresh” can never fire. That write also passes the literal 86400 as totalCVEs, so the tracker’s record count is meaningless.

2. Source contract (from Phase 0 recon)

Cadence: Irregular / incident-driven
Volume: Very low (3 bulletins total as of 2026-05-07)
Identifier shape: URL slugs (vercel-april-2026-security-incident) + CVE IDs when applicable (CVE-2025-55184)
Backfill feasibility: Full historical archive is the listing page itself (no pagination, no archive). A single run fetches everything.
Anti-bot / auth / rate-limit: None detected with standard browser User-Agent. Vercel edge network serves HTML directly.
Pagination / freshness: No pagination. No Last-Modified or ETag of value. Tracker-based freshness check with 24h frequency.

3. Architecture diagram

graph LR Source[Vercel KB /bulletin] --> HTTP[httpclient.New] --> Parse[regex extract slugs] Parse --> PerSlug[per-slug: fetch + parse HTML] PerSlug --> Map[map to CVEMetadata] Map --> Tx[(pgx Tx)] --> Store[db.UpsertCVEMetadata] Store --> CVEAlias[(db.InsertAliases)] Store --> CVEDescription & CVEReference Tx -.commit.-> S3[s3client.ArchiveRecord] Map --> CRIT[critutil.NewEnvelope -> StageCandidate] --> Drain[critpublisher.DrainKeys] --> CritRecord & VEX

4. Source → DB field mapping

Source field	Target table / column	Notes
slug	`CVEMetadata.cveId` (when no CVE)	Primary identifier
CVE-ID from title/body	`CVEMetadata.cveId` (when present)	Canonical identifier
slug	`CVEAlias.aliasCveId`	When CVE is primary
`<title>` (minus suffix)	`CVEMetadata.title`	“Security Bulletin: CVE-…”
meta description	`CVEDescription.value`	Preferred description
body text	`CVEDescription.value` (fallback)	Truncated to 2000 chars
page URL	`CVEMetadata.sourceAdvisoryRef`, `CVEMetadataReferences.url`	Type = “advisory”
canonical JSON SHA-256	`CVEMetadata.sourceFileHash`	Resume / dedup
—	`CVEMetadata.datePublished`	Hardcoded `0` (`main.go:487`, commented “no reliable date from source”). The Next.js payload’s `lastUpdated` date is extracted (`extractLastUpdated`) and is already used as the CRIT `provider_fix_date`, so a usable date is in hand and simply not persisted. All five production `source='vercel'` rows have `datePublished = 0`.

5. Identifier policy

Case	`CVEMetadata.cveId`	`CVEAlias` rows
Bulletin contains CVE ID(s)	First CVE-ID (e.g. `CVE-2025-55184`)	Slug + remaining CVEs
Bulletin contains NO CVE ID	Slug verbatim (e.g. `vercel-april-2026-security-incident`)	`nil` (cross-source backfill still runs)

When CVE is primary, the slug is stored as an alias so reverse lookups from the Vercel URL still resolve.

6. CRIT / VEX

Service triples

Provider	Service	ResourceType	Trigger hint
vercel	`vercel_platform`	`project`	“Vercel”, “deployment”, “project”
vercel	`nextjs`	`application`	“Next.js”, “Nextjs”, “React”
vercel	`vercel_edge`	`function`	“edge”, “middleware”
vercel	`vercel_kv`	`database`	“KV”, “Redis”
vercel	`vercel_postgres`	`database`	“Postgres”, “PostgreSQL”

resolveServices matches lowercase substrings against title + meta description + the whole tag-stripped page body. The hints are broad — “vercel”, “project” and “deployment” all appear in the page chrome of every bulletin, so vercel_platform/project matches unconditionally, and kv / edge / react match as substrings of unrelated words. Expect over-matching rather than under-matching.

Extended dictionary

internal/critutil/dictionaries/extended/vercel.json

VEX status derivation

“patched” / “fixed” / “resolved” in body → fixed
“mitigated” / “workaround” → not_affected
Otherwise → under_investigation

7. S3 / source-file archive layout

Bucket: $S3_BUCKET_NAME
Archive key: vercel/files/{sha256}/{slug}.json
Payload schema: Canonical JSON of vercelBulletin struct
Quarantine reasons: store-error (DB transaction failure)

8. Error handling & Slack

Per-record vs batch: Per-bulletin failure is logged and recorded via notifier.RecordError; processing continues.
Retry profile: 3 attempts, exponential backoff (500ms × attempt), on DB tx failures.
Slack events:
- Started — at process start
- Completed — with stats {"fetched": N, "stored": M}
- Errored — when listing fetch fails or >0 per-record errors
- NoWork — when tracker is fresh and --force not set

9. Performance

Concurrency: Sequential per-bulletin fetches (low volume, polite to source).
Per-request timeout: 30s (httpclient default).
Rate-limit pacing: 200ms delay between individual bulletin fetches.
Soft deadline: ⚠ broken in ECS. softDeadline = start + (EXPECTED_DURATION_MINUTES - 10) (main.go:97) and the per-bulletin loop breaks when now > softDeadline - 5min (main.go:179). With expected_duration_minutes = 15 that reduces to now > start, which is true by the time the pool has connected and the listing has been fetched — so every scheduled run logs approaching soft deadline; stopping and stores zero bulletins. Local backfills are unaffected because the recipe unsets the env var (leaving softDeadline zero, which disables the check).
Resume strategy: db.LoadProcessedHashes keyed by the primary cveId → sourceFileHash. (main.go:222 writes the in-memory cache under slug instead, which never matches the cveID lookup at main.go:199; the DB-loaded map is what actually makes resume work across runs.)

10. Backfill

No separate cmd/vercel-security-bulletins-backfill/ needed. The scheduled binary fetches the full listing every run (no pagination, low volume). The justfile recipe go-vercel-security-bulletins-fetch-backfill runs the same binary with unset EXPECTED_DURATION_MINUTES.

S3 Persistence

Archive path: vercel/files/{sha256}/{filename} ✓
Quarantine path: failed-feeds/vercel-security-bulletins-fetch-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
Failure reasons emitted: store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[vercel-security-bulletins-fetch-processor] PROC -->|success| ARCHIVE[("S3: vercel/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/vercel-security-bulletins-fetch-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.