Cloudflare Advisories Processor — Design Document

Overview

Ingests Cloudflare’s published security advisories from three complementary public surfaces and writes one CVEMetadata row per advisory under source = "cloudflare". Where an advisory carries CVE identifiers, those are linked as aliases via db.InsertAliases. Raw payloads are archived to S3. When --emit-crit is set, CRIT candidate envelopes are staged to S3 per (vulnID × service-match) after a WAF-mitigation guard accepts the advisory as Cloudflare-service-affected.

No API key is strictly required. GITHUB_PAT is honoured (when present) to raise the GHSA REST rate limit. No new database tables or migrations.

Sources

Sources are processed in the listed order. Each source honours the soft deadline derived from EXPECTED_DURATION_MINUTES (when set) and the --limit cap. Once any source signals stop, the remaining sources are skipped.

Per the project rule, backfill mode (no EXPECTED_DURATION_MINUTES) runs to completion regardless of duration.

Vulnerability Identifier Prefixes

Cloudflare advisories never define their own CVE-style namespace, so this processor uses a layered identifier strategy:

All Cloudflare-emitted rows carry CVEMetadata.source = "cloudflare" regardless of which sub-source produced them. Sub-source provenance is preserved via CVEMetadataReferences.referenceSource (one of ghsa, blog, changelog-application-security, changelog-security-overview, changelog-waf).

This mirrors the AWS pattern (ALAS-* for cveId, CVE-* for aliases) described in AGENTS.md.

Architecture

Data Flow

Source-to-DB Mapping

CVEMetadata

Nothing else is persisted. writeAdvisory (main.go:575-612) writes only CVEMetadata + sourceFileHash + one reference + aliases — no CVEDescription, no CVEMetric, no CVEAffected, no rawDataJSON. The GHSA REST payload does carry description, severity, a CVSS vector, the affected package list and patched_versions, and those are used for CRIT resolution only, so every Cloudflare row in CVEMetadata is title-only (103 rows, 0 with a vector, 0 with rawDataJSON as of the 2026-08-06 audit). Persisting them is open work.

SourceFileHash

CVEMetadataReferences (one row per advisory)

CVEAlias (zero or more rows per advisory)

Written via db.InsertAliases(ctx, tx, primaryID, "cloudflare", cveIDs, …). Aliases are CVE ids extracted from:

• GHSA path: structured cve_id field
• Blog path: regex CVE-\d{4}-\d+ over title + description
• Changelog path: regex CVE-\d{4}-\d+ over title + body (only when the entry also matches \b(vulnerability|security|CVE|fixed|patch)\b)

Cross-source backfill is performed automatically by db.InsertAliases.

S3QueueObject (only when --emit-crit)

One row per staged CRIT candidate, processingStatus = "pending".

S3 Layout

Service / Resource-Type Resolution

Resolve(hints) walks the synonyms table (longer/more-specific patterns first) and returns deduplicated ServiceMatch{Service, ResourceType, HintMatched}. Source of hints:

• GHSA: vulnerabilities[].package.name (preferred), falls back to summary.
• Blog: regex over title + description: (?i)(Cloudflare WARP[^"<>]*|Cloudflare Workers|Cloudflare Pages|Cloudflare R2|Cloudflare D1|Cloudflare Access|Cloudflare Gateway|Cloudflare Zone|Cloudflare WAF|Pingora|cloudflared|WARP Connector|Cloudflare One).
• Changelog: same regex (without Cloudflare One), restricted to entries with both a CVE id and a security-context word.

Recognised products with no spec or extended dictionary entry (Magic Transit / WAN / Firewall, Workers AI, Vectorize, Queues, Durable Objects, Hyperdrive) are matched against skipHints so they are logged as unmatched-skipped rather than silently dropped.

Business Rules (from code)

These rules are encoded as conditions in the source. Treat the code as authoritative when the doc lags.

B1 — De-duplication across sources

• Per-source: skip if seen[id] already set (seen keyed by GHSA id, CVE id, or synthetic CFADVISORY-*).
• GHSA → blog: a blog post is dropped when every CVE it carries was already seen via GHSA (allDup check in handleBlog).
• GHSA/blog → changelog: same rule for changelog entries.
• This makes GHSA the canonical source whenever it covers the CVE.

B2 — Synthetic id minting

• Blog: id = "CFADVISORY-" + post.PubDate.Year() + "-" + slugify(title). When the title slug is empty, fall back to "blog-" + pubDate.Unix().
• Changelog: id = "CFADVISORY-CL-" + feedSlug + "-" + slugify(title). Fallback "clog-" + updated.Unix().
• slugify lowercases, keeps [a-z0-9], collapses [ \-_] to single -, truncates at 32 chars, trims trailing -.

B3 — Changelog quality filter

A changelog entry is processed only when its title + body text matches both:

• CVE-\d{4}-\d+ (a CVE id is mentioned), AND
• (?i)\b(vulnerability|security|CVE|fixed|patch)\b (security context).

This avoids ingesting feature/release-note entries.

B4 — Idempotency hashes

The hash is persisted via db.UpdateSourceFileHash. (Resume-by-hash for this processor is reserved — force is accepted by the binary but not yet wired to short-circuit reads, because volume is small.)

B5 — WAF-mitigation guard (IsCloudflareServiceAffected)

Before staging any CRIT candidate, the processor must affirm that the advisory describes a vulnerability in a Cloudflare service, not merely “Cloudflare WAF blocks CVE-X for customers”. Decision tree:

1. If package.name matches a synonym → affected (true).
2. Else inspect lowercased summary + description + body:
   • Positive signals (vulnerability in cloudflare, vulnerability in pingora, cloudflare … was affected, issue in cloudflare workers/ pages/r2/d1/warp/access/gateway, …) → affected.
   • WAF-only signals (blocked by cloudflare waf, cloudflare waf blocks, waf rule deployed, cloudflare protects customers) without any positive signal → not affected.
3. Else if any free-text hint resolves to a synonym → affected.
4. Else → not affected.

Positive signals beat WAF signals (defence-in-depth case).

B6 — fix_propagation inference (CRIT only)

inferFixPropagation(service, body, patchedVersion):

• Text overrides (highest priority):
  • "no customer action" / "no action required" / "deployed across" / "applied across" / "rolled out" → automatic
  • "configuration change" / "apply this configuration" → config_change
  • "upgrade" + "version" → version_update
• Service defaults:
  • warp → version_update if patchedVersion else config_change
  • pingora_proxy, worker, pages, r2, d1, zone → automatic
  • access, gateway → config_change
  • default → version_update

Whether the value came from text or defaults is recorded as the CRIT provenance confidence (high from text, medium from defaults).

B7 — resourceLifecycle per service

warp, access, gateway → stateful_customer; pingora_proxy → global_control_plane; worker, pages, r2, d1 → ephemeral; zone → config_only; fallback → stateful_managed.

B8 — sharedResponsibility and “existing remain vulnerable”

• shared_responsibility = "provider_only" iff fix_propagation == "automatic", else "customer_action_required".
• existing_deployments_remain_vulnerable = !(provider_only && automatic).

B9 — CVE publish-date canonicalisation

For CRIT temporal block:

• If vulnID starts with CVE-, look up CVEMetadata.datePublished for the canonical row and use it as vuln_published_date.
• Otherwise (or when not found) fall back to the advisory’s own pubDate.
• provider_fix_date always uses the advisory pubDate.
• service_available_date comes from serviceavail.Date("cloudflare", svc), defaulting to 2010-09-27 (Cloudflare launch).

B10 — CRIT envelope cardinality

For each accepted advisory the processor emits: len(matches) × max(len(cveIDs), 1) candidate envelopes. When no CVE id is associated, the bulletin id is used as the vulnID.

B11 — Retry & quarantine

storeAdvisory retries the transaction up to 3 times with linear back-off (attempt × 500 ms). On final failure the raw payload is written to quarantine/cloudflare-advisories-fetch-processor/store-error/ and notifier.RecordError is called.

B12 — Soft deadline & limit

• EXPECTED_DURATION_MINUTES minus 10 minutes establishes the soft deadline (main.go:73), and checkStop halts 5 minutes before that (main.go:151) — so a run does useful work for EXPECTED_DURATION_MINUTES − 15 (15 minutes on the configured 30-minute budget).
• --limit N caps the total processed across all sources.
• Backfill mode (no EXPECTED_DURATION_MINUTES) runs to completion.

B14 — Error accounting (audit 2026-08-06)

Per-record store failures call notifier.RecordError, but the run always ends in notifier.Completed (main.go:230) — HasErrors()/Errored() are never consulted, and a GHSA fetch failure is only logged (main.go:161-163). A run in which every advisory failed to store still reports success and exits 0. Fix by finalising through notifier.Finalize, as alpine-apk/homebrew do.

B13 — Inter-record pacing

A 200 ms sleep is inserted between records inside each source loop to remain courteous to upstream APIs.

Flags

ECS Schedule

• Cron: cron(0 12 * * ? *) — daily at 12:00 UTC
• CPU: 256 units
• Memory: 512 MB
• Expected duration: 30 minutes
• Container image: go-processors:go-cloudflare-advisories-fetch-processor-{tag}

Verification Queries

-- Cloudflare CVEMetadata rows
SELECT
    CASE
        WHEN "cveId" LIKE 'GHSA-%' THEN 'ghsa'
        WHEN "cveId" LIKE 'CFADVISORY-CL-%' THEN 'changelog'
        WHEN "cveId" LIKE 'CFADVISORY-%' THEN 'blog'
        ELSE 'other'
    END AS path,
    COUNT(*)
FROM "CVEMetadata"
WHERE source = 'cloudflare'
GROUP BY 1
ORDER BY 1;

-- Aliased CVE coverage (CVEAlias columns: primaryCveId/primarySource/aliasCveId/aliasSource)
SELECT COUNT(DISTINCT a."aliasCveId")
FROM "CVEAlias" a
WHERE a."primarySource" = 'cloudflare' AND a."aliasCveId" LIKE 'CVE-%';

-- Sub-source breakdown via references
SELECT "referenceSource", COUNT(*)
FROM "CVEMetadataReferences"
WHERE "cveId" IN (SELECT "cveId" FROM "CVEMetadata" WHERE source = 'cloudflare')
GROUP BY "referenceSource";

-- Pending CRIT candidates
SELECT COUNT(*)
FROM "S3QueueObject"
WHERE source = 'cloudflare-advisories-fetch-processor'
  AND "processingStatus" = 'pending';

S3 Persistence

• Archive path: cloudflare/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/cloudflare-advisories-fetch-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.