broadcom-vmware-security-fetch-processor — Design

Purpose

Ingest VMware Security Advisories (VMSA) and Tanzu (TKG / TAS / Ops Manager / Harbor / CF buildpack / BOSH stemcell) enrichment data into vdb-manager via four unauthenticated, structured public sources. Source identity: broadcom-vmware.

Why this architecture (and not headless scraping)

Broadcom retired the VMware security advisory portal into a Liferay SPA (support.broadcom.com/web/ecx/security-advisory) and moved the public-facing list to a React SPA at broadcom.com/support/vmware-security-advisories. Both surfaces resist headless rendering:

• broadcom.com redirects headless Chrome to /browser-not-supported immediately, even with disable-blink-features=AutomationControlled, real-Chrome UA, no-zygote and the full cert-il-fetch-processor stealth set. Probe artefacts: cmd/broadcom-vmware-security-fetch-processor/probe.go (build-tag probe), snapshots dumped to /tmp/vmw-probe-*.html.
• support.broadcom.com Liferay portlet renders the SPA shell but does not fetch advisory data without an ECX session cookie.

Tested upstream alternatives that DO work without a browser:

Two-phase main flow

main()
  ├─ Phase A: VMSA ingestion
  │    1. FetchAdvisoryRepo() — single GitHub tarball GET, stream-extract all
  │       security-advisories/vmsa-*/README.md
  │    2. parseAdvisoryMarkdown() — extract VMSA-id (filename), CVE-IDs (regex),
  │       date (changelog ISO line OR "On Month D, YYYY" body line), Title,
  │       Description (Introduction section, ATX or setext)
  │    3. MapAdvisory() — one CVESourceData per CVE in the advisory
  │       (R-VMW-1: VMSA on aliases, never primary)
  │    4. processor.StoreCVESourceData per CVE in a transaction
  │    5. S3 archive vmsa-{id}.json on success, quarantine on parse error
  │    6. stageCRITForAdvisory() — emit envelopes for resolved (CVE × ServiceMatch)
  │
  ├─ Phase B: Tanzu enrichment
  │    1. FetchUSN(since) — paginate ubuntu.com/security/notices.json (limit=20)
  │    2. FetchBoshStemcellReleases() — fetch up to 200 BOSH stemcell releases
  │    3. FetchBuildpackReleases() — iterate the 10 CF buildpack repos
  │    4. Build USN-id → []CVE-id map
  │    5. For each (stemcell, USN-ref) pair: lookup CVE-IDs and call
  │       upsertEnrichmentRow, product='Tanzu BOSH Stemcell ({distro} {tag})'
  │    6. For each (buildpack, CVE) pair: upsertEnrichmentRow,
  │       product='Cloud Foundry {lang}-buildpack'
  │    R-VMW-2: never creates CVEMetadata; CVE-existence enforced implicitly
  │    by FK constraint on CVEAffected (logged-skip on miss)
  │
  └─ Phase C: critpublisher.DrainKeys(Workers: 4) (only if --emit-crit)

Phases are independent — failure in one does not block the other.

Source identity & naming

No backfill recipe — both phases pull rolling/newest-first feeds; manual one-shots use aws ecs run-task or go run ./cmd/... directly.

Business rules

Phase A — extraction details

The repo’s markdown is unstructured Q&A. The only reliably extractable fields:

• VMSA ID: from the directory path security-advisories/vmsa-YYYY-NNNN/.
• CVE IDs: regex CVE-\d{4}-\d{4,7} against the entire body.
• Date: prefer leading ISO date in the changelog (^YYYY-MM-DD, line), fall back to body-text “On Month D, YYYY” phrase used in modern ATX-headed Q&A pages. ~75% of advisories yield a date; older 2021/2022 entries with no Introduction section are date-less (acceptable — NVD owns datePublished).
• Description: prefer Introduction section (ATX ## Introduction or setext); fall back to the first non-blockquote/non-heading paragraph >30 chars. Capped to 1500 chars.

Coverage: ~20 advisories (cherry-picked vSphere/VCF-focused). Not the full VMSA corpus; full coverage requires NVD references[] cross-walk, out of scope for v1.

Phase B — enrichment details

The Ubuntu USN feed is paginated newest-first with a server-enforced limit ≤ 20 per page (HTTP 422 above that). The processor uses cves_ids not cves because the API encodes cves as [{id,notices_ids},...] rather than [string,...].

release_packages is polymorphic (object or array per entry) — decoded defensively into release → []package map; downstream code does not currently consume this field but it’s preserved for future tagging.

Stemcell parsing extracts distro + release-tag from the BOSH release title (e.g. "ubuntu jammy v1.1193" → {Distro: ubuntu, ReleaseTag: jammy, Family: tanzu-stemcell-ubuntu}).

Buildpack CVE extraction is best-effort regex against release body; many buildpack releases don’t enumerate CVEs explicitly. Sparse output is expected.

CRIT emission

• provider = "vmware" (upstreamed in ietf-crit-spec commit d60eb0b).
• template_format = "vmware_url" (also in d60eb0b).
• Service tuples sourced from internal/critutil/dictionaries/extended/vmware.json (12 entries: vsphere/vcenter/esxi/nsx/vsan/cloud_foundation/aria_automation/tkg/tas/ ops_manager/harbor/cf_buildpack/bosh_stemcell).
• fix_propagation:
  • cf_buildpack → rolling_replace (apps must restage)
  • bosh_stemcell → rebuild_and_redeploy (VM recreate)
  • everything else → version_update
• resource_lifecycle: tkg/tas/ops_manager/aria_automation → stateful_managed; cf_buildpack → ephemeral; bosh_stemcell + on-prem → stateful_customer.
• Confidence ceiling: medium (advisory bodies live behind the Liferay portal we cannot fetch).

Verification

# Phase A — VMSA ingestion
set -a; source .env.production; set +a
unset EXPECTED_DURATION_MINUTES
cd scripts/go-processors && go run ./cmd/broadcom-vmware-security-fetch-processor \
    --phase=a --emit-crit=false 2>&1 | tee /tmp/vmw-a.log

# Phase B — Tanzu enrichment
go run ./cmd/broadcom-vmware-security-fetch-processor \
    --phase=b --usn-since-days=180 --emit-crit=false 2>&1 | tee /tmp/vmw-b.log

# Both phases + CRIT (matches ECS daily cron)
go run ./cmd/broadcom-vmware-security-fetch-processor \
    --phase=both --force=true --emit-crit=true 2>&1 | tee /tmp/vmw.log
grep '"msg":"crit drain complete"' /tmp/vmw.log

-- VMSA aliases (CVEAlias columns are primaryCveId/primarySource/aliasCveId/aliasSource)
SELECT count(*) FROM "CVEAlias"
 WHERE "aliasCveId" LIKE 'VMSA-%' AND "discoveredFrom" = 'broadcom-vmware';

-- Tanzu affected rows (no ecosystem column — group by the product label)
SELECT "product", count(*) FROM "CVEAffected"
 WHERE source = 'broadcom-vmware'
 GROUP BY 1 ORDER BY 2 DESC;

-- CRIT records
SELECT service, count(*) FROM "CritRecord"
 WHERE provider='vmware' GROUP BY 1 ORDER BY 2 DESC;

-- S3 queue empty after drain
SELECT "processingStatus", count(*) FROM "S3QueueObject"
 WHERE source='broadcom-vmware-security-fetch-processor' GROUP BY 1;

Risk surface

Definition of done

• ✅ go build ./... and go vet ./... clean
• ✅ Spec-lib vmware / vmware_url upstreamed (commit d60eb0b) and vendored copies synced
• ✅ Phase A live-fetches 19 advisories from the github tarball with date parsing for all 2024+ entries
• ✅ Phase B live-fetches 100+ USNs paginating cleanly, 50 stemcell releases with 164 USN refs, 10 buildpack repos enumerated
• ✅ Production run with DB writes — 109 CVEMetadata rows under source='broadcom-vmware' (89 real advisories + 20 VMSA-* alias placeholder rows minted by db.InsertAliases); newest fetch 2026-07-31
• ✅ ECS task family + EventBridge schedule deployed (weekly, Sundays 03:00 UTC, --emit-crit=true)
• ✅ Containerfile build via post-push-ecr.sh hook

Open items found by the 2026-08-06 audit

• Phase A CRIT staging can only ever run on first ingest. The processedHashes[adv.BodyHash] resume check continues before stageCRITForAdvisory (main.go:251-254, main.go:293), so once an advisory is stored its CRIT envelope is never re-attempted — a later dictionary addition or a staging failure is never picked up. Production has 0 CritRecord rows for provider='vmware' despite --emit-crit=true on the schedule. Move the CRIT block outside the resume continue, or gate it on “no CritRecord for this CVE yet”.
• 30 of the 89 real advisory rows carry datePublished = 0 (plus the 20 alias placeholders). The column is NOT NULL int4, so an unparsed date lands as 0 rather than NULL; the extraction notes already call the date-less 2021/2022 entries out, but zero is indistinguishable from 1970 to consumers.

S3 Persistence

• Archive path: broadcom-vmware/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/broadcom-vmware-security-fetch-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.