GCP Security Bulletins Processor — Design Document

Overview

Single Go binary that ingests Google Cloud security bulletins from three public Atom feeds and writes one CVEMetadata row per bulletin under source = "google-cloud". Every CVE id mentioned in the bulletin body is linked as an alias via db.InsertAliases, every GHSA id likewise. URLs referenced in the bulletin body are stored as CVEMetadataReferences rows. Raw payloads are archived to S3. When --emit-crit is set, CRIT candidate envelopes are staged per (vulnID × service-match).

The GKE feed is the highest-fidelity input for CRIT generation (explicit node-version remediation, well-typed cluster service mapping) and is processed last so its bulletins see the largest dedup-set from earlier feeds.

No API key required. All feeds are public Atom XML.

Sources

The processor pulls three Atom feeds, enumerated in main.go:57-61:

Host equivalence — https://docs.cloud.google.com/feeds/<file> and https://cloud.google.com/feeds/<file> return byte-identical content (verified: 50 683 / 112 030 / 193 335 bytes for the three feeds). The cloud.google.com host is the canonical one; the docs. host is a documentation alias and is not fetched (a duplicate fetch would double load with no new data).

Cross-feed dedup is enforced by seenBulletinIDs (main.go:171-174): once a bulletin id is processed in one feed it is skipped in subsequent feeds. The feed loop order is: unified → compute-engine → kubernetes-engine.

Backfill mode (no EXPECTED_DURATION_MINUTES) runs to completion regardless of duration.

Vulnerability Identifier Prefixes

All three feeds share CVEMetadata.source = "google-cloud". Sub-feed provenance is preserved via reference rows whose referenceSource records which feed emitted them. This mirrors the AWS pattern (ALAS-* for cveId, CVE-* for aliases) described in AGENTS.md.

Architecture

Data Flow

Source-to-DB Mapping

CVEMetadata (main.go:334-342)

SourceFileHash (main.go:346-348)

CVEMetadataReferences (main.go:349-358)

One row for the bulletin’s own URL plus zero or more rows per HTTPS URL extracted from the body via anchorRe = href="(https?://[^"]+)", de-duplicated and excluding the self-link.

CVEAlias (main.go:360)

Written via db.InsertAliases(ctx, tx, bulletinID, "google-cloud", aliases, …). The aliases slice contains every distinct CVE-… id and every distinct GHSA-… id extracted from the body. Cross-source backfill is performed automatically by db.InsertAliases.

S3QueueObject (only when --emit-crit)

One row per staged CRIT candidate, processingStatus = "pending".

S3 Layout

Key reconstruction for CRIT provenance: gcpS3Key(sha256, filename) = "google-cloud/files/{sha256}/{filename}" (crit_mapper.go:21-23).

Service / Resource-Type Resolution

Resolve(hints) (service_map.go:107-131) walks the synonyms table (longer / more specific patterns first) and returns deduplicated ServiceMatch{Service, ResourceType, HintMatched}. Hints come from two sources merged in parser.normalize (parser.go:127-140):

1. Anchor slugs via productLinkRe = docs\.cloud\.google\.com/([a-z0-9-]+)/security-bulletins — gives kubernetes-engine, compute-engine, cloud-run, etc.
2. Free-text via serviceTextRe — (?i)(GKE|Kubernetes Engine|Container-Optimized OS|Compute Engine|GCE|Cloud Run|…|Filestore|Firestore).

Recognised products with no spec dictionary entry today (App Engine, GDC software, GKE-on-AWS / on-Azure / GDCBM, GDC VMware) match skipHints (service_map.go:93-102) so they are logged as unmatched-skipped rather than silently dropped.

Business Rules (from code)

These rules are encoded as conditions in the source. Treat the code as authoritative when the doc lags.

B1 — Cross-feed dedup

A bulletin id processed in one feed is skipped in any later feed (main.go:171-174). The unified feed runs first, so per-product feeds mostly contribute net-new bulletins not already in unified.

B2 — Resume by hash

db.LoadProcessedHashes(read, "google-cloud") is called once per feed (main.go:155). An entry is skipped when !--force && resumeSet[id] == bulletinHash(entry) (main.go:178-180). Hash is SHA-1 of BulletinID|Updated(RFC3339)|Body (main.go:386-394), so a body edit re-triggers processing.

B3 — Severity normalisation

severityRe = (?i)<td>\s*(Critical|High|Medium|Low|Important)\s*</td> (parser.go:83). The first match wins and is title-cased (parser.go:111-113, 147-153). The severity is embedded in the title and passed to CRIT mapping but no CVEMetric row is written (GCP feeds don’t carry CVSS vectors per CVE).

B4 — Service hint extraction

Two regexes feed Resolve:

• Product-link slugs from any anchor whose href contains docs.cloud.google.com/{slug}/security-bulletins (parser.go:85).
• Free-text mentions in the bulletin body (parser.go:89). Both are lower-cased and de-duplicated before being passed to service_map.Resolve (parser.go:127-140).

B5 — GKE high-fidelity signals

The GKE feed is processed last and benefits from:

• Explicit kubernetes_engine/cluster mapping (service_map.go:24-26).
• “Upgrade to GKE node version X.Y.Z” body text triggers inferFixPropagation returning version_update from the explicit text branch (crit_mapper.go:40-41), tagged fromText=true → CRIT confidence high rather than medium.
• resourceLifecycle("kubernetes_engine") = "stateful_customer" (crit_mapper.go:64-65) reflecting that node versions are customer controlled.
• sharedResponsibility = "customer_action_required" (since fix_propagation is version_update, not automatic).

This combination produces CRIT candidates that pass strict validation without dictionary fall-through, which is why GKE bulletins are the processor’s primary CRIT generator.

B6 — CVE & GHSA alias extraction

• cveRe = CVE-\d{4}-\d+ over the body (parser.go:80,117-122), case-insensitive dedup, written to CVEAlias.
• ghsaRe = GHSA-[0-9a-z]{4}(?:-[0-9a-z]{4}){2} over the body, same dedup, also written to CVEAlias. Both id types travel through db.InsertAliases so canonical edge direction and bundle suppression rules are honoured (AGENTS.md — Alias Writes section).

B7 — Reference URL extraction

For each bulletin the processor writes:

1. The bulletin’s own <link rel="alternate"> URL with type="advisory".
2. Every distinct https://… URL captured by anchorRe = href="(https?://[^"]+)" over the body, with type="web", dedup case-sensitive, excluding the self-link.

All references carry referenceSource = "google-cloud".

B8 — Soft deadline & limit

• EXPECTED_DURATION_MINUTES minus 10 minutes establishes the soft deadline; the loop stops 10 minutes before that point (both at the feed boundary and at the per-entry boundary — main.go:136-139, 167-169).
• --limit N caps the total processed across all feeds (main.go:163-166).
• Backfill mode (no env var) runs to completion.

B9 — Retry & quarantine

storeBulletin retries the transaction up to 3 times with linear back-off (attempt × 500 ms, main.go:302-318). On final failure the raw payload is written to quarantine/gcp-security-bulletins-rss-processor/store-error/ and notifier.RecordError is called.

B10 — Inter-record pacing

A 200 ms sleep is inserted between processed records inside each feed loop (requestDelay, main.go:212).

B11 — fix_propagation inference (CRIT only)

inferFixPropagation(body, service) (crit_mapper.go:29-53):

• Text overrides (highest priority):
  • "no customer action" / "no action required" → automatic
  • "google has already applied" / "google applied this fix" → automatic
  • "rebuild your image" / "rebuild based on" → rebuild_and_redeploy
  • "apply this configuration" / "configuration change" → config_change
  • "upgrade" + "version" → version_update
• Service defaults when no clear signal:
  • kubernetes_engine, compute, cloud_sql, spanner, bigquery, cloud_storage, filestore, memorystore, bigtable, firestore → version_update
  • cloud_run, cloud_functions, cloud_build, dataflow, composer, dataproc, vertex_ai → automatic
  • default → version_update

Whether the value came from text or defaults is recorded as the CRIT provenance confidence (high from text, medium from defaults).

B12 — resourceLifecycle per service (crit_mapper.go:59-75)

• Stateful managed (compute, cloud_sql, spanner, bigquery, cloud_storage, filestore, memorystore, bigtable, firestore).
• Stateful customer (kubernetes_engine — node versions customer-managed).
• Ephemeral (cloud_run, cloud_functions, cloud_build, dataflow, composer, dataproc, vertex_ai).
• Config-only (cloud_iam, cloud_kms, cloud_armor, cloud_dns, cloud_logging, cloud_monitoring, secret_manager, artifact_registry).
• Default → stateful_managed.

B13 — CRIT envelope cardinality

For each accepted bulletin the processor emits len(matches) × max(len(cveIDs), 1) candidate envelopes (main.go:252-289). When no CVE id is associated, the bulletin id itself is used as the CRIT vulnID.

B14 — CVE publish-date canonicalisation

For CRIT temporal block: if vulnID starts with CVE-, look up CVEMetadata.datePublished for the canonical row via db.LoadCanonicalDatePublished and use it as vuln_published_date. Otherwise fall back to the bulletin’s own published date. provider_fix_date always uses the bulletin’s Published field. service_available_date comes from serviceavail.Date("google-cloud", svc) with a sensible default when absent.

Flags

ECS Schedule

• Cron: Runs daily at 12:30 UTC (cron(30 12 * * ? *)).
• CPU: 256 units
• Memory: 512 MB
• Expected duration: 30 minutes
• Container image: go-processors:go-gcp-security-bulletins-rss-processor-{tag}

Verification Queries

-- GCP CVEMetadata rows
SELECT COUNT(*) FROM "CVEMetadata" WHERE source = 'google-cloud';

-- Aliased CVE coverage
SELECT
    CASE WHEN a.alias LIKE 'CVE-%'  THEN 'CVE'
         WHEN a.alias LIKE 'GHSA-%' THEN 'GHSA'
         ELSE 'other' END AS alias_kind,
    COUNT(*)
FROM "CVEAlias" a
JOIN "CVEMetadata" m ON m."cveId" = a."cveId"
WHERE m.source = 'google-cloud'
GROUP BY 1;

-- References per bulletin (advisory + extracted body URLs)
SELECT m."cveId", COUNT(r.*) AS ref_count
FROM "CVEMetadata" m
LEFT JOIN "CVEMetadataReferences" r ON r."cveId" = m."cveId"
WHERE m.source = 'google-cloud'
GROUP BY m."cveId"
ORDER BY ref_count DESC
LIMIT 20;

-- Pending CRIT candidates from this processor
SELECT COUNT(*)
FROM "S3QueueObject"
WHERE source = 'gcp-security-bulletins-rss-processor'
  AND "processingStatus" = 'pending';

S3 Persistence

Not used. This processor does not currently archive payloads or quarantine failures to S3. Per the S3 Persistence Contract this is non-compliant — see the compliance matrix for the implementation roadmap.

⚠ Not in the compliance matrix — status needs verification.

Expected paths when implemented:

• Archive: gcp-security-bulletins/files/{sha256}/{filename}
• Quarantine: failed-feeds/gcp-security-bulletins-rss-processor/{YYYY-MM-DD}/{reason}/{filename}
• Likely reasons: (none documented)