MSRC CSAF Processor — Design Document

1. Overview

Purpose: Fetch Microsoft Security Response Center (MSRC) per-CVE CSAF 2.0 advisories, merge companion VEX documents when available, and store the result into:

• Standard CVEMetadata pipeline tables (source = 'msrc')
• Microsoft-specific enrichment tables: MsrcPatchTuesday, MsrcAdvisory, MsrcProductFamily, MsrcAffectedProduct, MsrcAffectedVersion

Source: msrc

Schedule: three EventBridge schedules all fire the same ECS task definition (module.msrc_csaf_processor, terraform/go-schedules.tf:1900), so there is one binary, one log group (/ecs/vdb-scheduler/msrc-csaf-processor) and one task-manager.toml family — the extra two only change how often it wakes up:

The extra wake-ups are cheap because of the changes.csv SHA256 gate below: a run with nothing new logs changes.csv unchanged, nothing to do, reports Outcome=no_work and exits in about a second.

Timeout: 90 minutes (expected_duration_minutes = 90)

Resources: 256 CPU units, 512 MB memory

2. Data Sources

Microsoft publishes one CSAF file per CVE identifier, unlike other vendors that publish one file per advisory bulletin.

3. Run Modes

Both modes share the same worker pool and per-advisory resume logic. Only daily carries a soft deadline — backfill deliberately runs to completion (cmd/msrc-csaf-processor/main.go:74), per the “backfill must not have a deadline” rule.

msrc-csaf-backfill in scripts/task-manager.toml is not a separate binary — there is no cmd/msrc-csaf-backfill. It is the same cmd/msrc-csaf-processor invoked with --mode=backfill by the go-msrc-csaf-backfill justfile recipe, and it is local-only (no ECS task definition, no schedule). The backfill recipe also defaults EMIT_CRIT=true, which the scheduled runs do not set.

Known defect (resume is only half-live). The resume lookup is keyed on the CSAF vulnerabilities[0].cve value (main.go:317), but the row it is looking for is stored under document.tracking.id (internal/msrc/mapper.go:44). For every advisory whose tracking id is not the bare CVE — which is all of them today, MSRC ships msrc_CVE-YYYY-NNNNN — the lookup misses, so the advisory is re-fetched, re-upserted and its MsrcAffectedVersion rows deleted and re-inserted on every run. Production shows the signature directly: tracking-id-keyed rows average fetchCount 3.9 (max 54) while the legacy bare-CVE rows sit at exactly 1.

4. Patch Tuesday Detection

Grouping is anchored on document.tracking.initial_release_date, falling back to current_release_date only when the initial date is missing or unparseable (internal/msrc/mapper.go:114-125). MSRC routinely revises advisories after first publication, which moves current_release_date; the Patch Tuesday an advisory shipped under is fixed at its initial ship date. When the anchor date falls on the second Tuesday of the month (UTC day-of-month in [8, 14] and weekday = Tuesday), the advisory is part of a Patch Tuesday release.

func isPatchTuesday(t time.Time) bool {
    t = t.UTC()
    return t.Weekday() == time.Tuesday && t.Day() >= 8 && t.Day() <= 14
}

Patch Tuesday CVEs are grouped under an MsrcPatchTuesday record. The title is formatted as "Month Day, Year" (e.g. "April 14, 2026").

5. VEX Merging Strategy

For every CSAF advisory fetched, the processor attempts to retrieve the companion VEX document at the corresponding /csaf/vex/ URL. A 404 response is expected for many CVEs and is silently ignored.

When a VEX document exists:

1. VEX product status declarations override any conflicting CSAF product status for the same product IDs
2. VEX product tree branches are merged into the CSAF product tree (expands name resolution)
3. VEX remediations are appended to CSAF remediations (may add additional KB articles)

Both the raw CSAF JSON and raw VEX JSON are stored in MsrcAdvisory.csafRawJson / MsrcAdvisory.vexRawJson for zero data loss.

6. Product Hierarchy Detection

The CSAF product tree uses a hierarchical branch structure:

branches[category=vendor]
  └── branches[category=product_family]    ← family detection (e.g. "Microsoft Office")
        └── branches[category=product_name] ← specific product (e.g. "Office 2024 Retail")
              └── branches[category=architecture|product_version]
                    └── product (leaf)     ← MsrcAffectedProduct.csafProductId

relationships[] links base products to platforms to form full-product-name leaf nodes. The processor walks this tree recursively to build:

• productMap: productID → full name
• familyMap: productID → family name (derived from category=product_family ancestor)

No family names are hardcoded. New product families released by Microsoft will be detected automatically from the tree structure.

7. Database Tables Written

Vulnerability identifier prefixes

The MSRC tracking id is the primary identifier — CVEMetadata.cveId is set to doc.Document.Tracking.ID verbatim (internal/msrc/mapper.go). Examples:

The embedded CVE id (vuln.CVE) is always passed into CVESourceData.Aliases so the central pipeline writes a CVEAlias row linking the primary id to the public CVE. db.InsertAliases (per AGENTS.md — Alias Writes) handles bundle suppression and self-link guarding when the primary and the alias happen to be the same string.

This mirrors the AWS pattern (ALAS- primary, CVE- alias) and the Cloudflare/GCP patterns documented in the sibling design docs.

Migration debt. Before commit 84db2d0 (2026-05-05) the primary id was the bare CVE, not the tracking id. That change did not migrate the rows already written, so source='msrc' currently holds two rows for the same advisory for 4,776 advisories: a live msrc_CVE-… row and a frozen CVE-… twin whose lastFetchedAt is stuck at 2026-05-05 with fetchCount = 1. 10,590 rows represent 5,814 advisories. MsrcAdvisory is duplicated identically (4,776 + 5,814). Consumers counting MSRC coverage or joining on the bare CVE will see stale content.

Tables

--emit-crit is off on all three ECS schedules (terraform passes only ["/app/msrc-csaf-processor"], terraform/go-schedules.tf:1905) and on by default in the go-msrc-csaf-backfill justfile recipe, so the two CRIT tables are written by local backfills only.

8. Environment Variables

9. Local Invocation

# Daily mode (checks changes.csv)
just go-msrc-csaf-daily

# Daily mode against production
just go-msrc-csaf-daily TARGET=prod

# Full backfill (all advisories from index.txt)
just go-msrc-csaf-backfill

# Backfill, first 10 advisories only (smoke test)
just go-msrc-csaf-backfill LIMIT=10

# Force-reprocess even if hashes match
just go-msrc-csaf-daily FORCE=true

10. Concurrency

• 5 worker goroutines (configurable via -workers flag)
• Rate limit: 300ms per request ≈ 3 req/sec shared across workers
• Each worker fetches one CSAF + attempts one VEX fetch per advisory
• Database writes are single-threaded (one consumer goroutine)
• Soft deadline: two hardcoded 10-minute margins compound, so workers stop taking new advisories 20 minutes before EXPECTED_DURATION_MINUTES (main.go:79 sets start + (mins − 10); main.go:285 then tests now > softDeadline − 10min). At the shipped 90-minute budget that leaves 70 minutes of fetching. This is the hardcoded-margin pattern internal/rundeadline.Soft exists to replace.

11. Idempotency

• All writes use INSERT ... ON CONFLICT DO UPDATE (UPSERT)
• MsrcAffectedVersion rows are deleted and re-inserted on re-processing (no natural unique key)
• Running the processor twice on the same data produces identical database state
• sourceFileHash (SHA256 of CSAF bytes) is used for per-CVE resume — unchanged advisories are skipped

12. S3 Key Format

msrc-archive/{dateSlug}/{trackingId}.csaf.json   ← CSAF document
msrc-archive/{dateSlug}/{trackingId}.vex.json    ← VEX document (when present)

Written at cmd/msrc-csaf-processor/main.go:524 and :534. dateSlug is the Patch Tuesday date for PT advisories, otherwise current_release_date, both YYYY-MM-DD (main.go:675). The keys are not content-addressed: the bucket policy grants public read on msrc-archive/* so vdb-api can serve https://vulnetix.com/msrc-archive/{dateSlug}/{trackingId}.csaf.json directly, and a revised advisory overwrites its own key.

Quarantine uploads for parse-error and schema-violation use the standard failed-feeds/… prefix (main.go:306, main.go:313).

AI Enrichment (aienrich)

After each item’s storeItem returns true (the per-item transaction committed), aienrich.RunBatch fires with a single-target payload for that CVE. Wire-in: cmd/msrc-csaf-processor/main.go, immediately after storeItem returns in the main result loop. The target carries CveID = result.cveData.CveID, Source = msrc.Source ("msrc"), Aliases = result.cveData.Aliases.

Four passes fire (no ghsapoc — self-gated to source = "github" only):

MSRC tracking ids frequently arrive without a CWE assignment — the CWE pass is the gap-filler here. RunBatch is invoked outside the parent transaction. See the aienrich overview for the full activity contract.

S3 Persistence

• Archive path: msrc-archive/{dateSlug}/{trackingId}.{csaf|vex}.json ✓ — see §12. (The generator’s default {source}/files/{sha256}/{filename} template does not apply here; corrected by hand, cf. ORCH-09.)
• Quarantine path: failed-feeds/msrc-csaf-processor/{YYYY-MM-DD}/{reason}/{filename} ✓ (main.go:306, main.go:313)
• Failure reasons emitted: parse-error, schema-violation

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.