elastic-rss-processor

Status: Live Source: Elastic Security Announcements (Discourse) Type: rss (single Discourse RSS 2.0 fetch, 30 latest topics) Source slug: elastic Schedule: Daily at 09:00 UTC (cron(0 9 * * ? *)), expected_duration_minutes = 15, 256 CPU / 512 MB.

Records produced

Condition	Records
Every ESA item	`CVEMetadata` (`source="elastic"`, primary id = first CVE in the body), `CVEDescription`, `CVEMetadataReferences` (type `advisory`), `CVEAlias` via `db.InsertAliases`
Any non-CVE alias not already in the DB (the `ESA-` id itself, `GHSA-` refs)	a placeholder `CVEMetadata` row under `source="elastic"` with `datePublished = 0` and no title, minted by `db.InsertAliases` → `db.EnsureMinimalCVEMetadata` (`internal/db/cvealias.go:138`) so the alias edge has a target
ESA with no CVE reference	minted `ESA-YYYY-NNNN` primary id + `GcveIssuance` (`GCVE-110-ESA-…`) + `GcveAlias` rows (`main.go:337-368`)
`--emit-crit` and a service match	staged CRIT candidate envelopes in S3, drained to `CritRecord` by `critpublisher.DrainKeys` (`main.go:279`)
Every run	`BulkDataDumpTracker` row for `elastic` (feed sha256 + stored count, `main.go:300`)

Roughly half of the source='elastic' rows in the database are the placeholder class above (65 ESA-* + 1 GHSA-* against 64 real CVE-* rows at the time of writing). They are alias-justified, not parse failures — exclude them from any coverage reading.

1. Overview

Purpose: ingest Elastic Security Announcements from the public Discourse RSS feed and emit CVEMetadata rows under source="elastic" plus dual CRIT envelopes (Elastic Cloud SaaS + self-managed Stack).
Source URL: https://discuss.elastic.co/c/announcements/security-announcements/31.rss
Owner: Elastic Security (CNA assignerShortName="Elastic" for issues also published on cve.org).
Schedule: cron(0 9 * * ? *) daily 09:00 UTC. 1h offset from Phase 1.16 GitLab’s 08:00 slot.
ECS resources: cpu_units=256, memory_mb=512, expected_duration_minutes=15.
Reads: Elastic Discourse RSS (one HTTP fetch), BulkDataDumpTracker.elastic for freshness, LoadProcessedHashes for per-record resume.
Writes: CVEMetadata, CVEDescription, CVEMetadataReferences, CVEAlias (ESA-* alias), BulkDataDumpTracker, S3 archive, optional CritRecord. GCVE issuance + alias only for items that lack a CVE-* alias.

2. Source contract (Phase 0 recon)

Question	Answer	Evidence
Cadence	Hourly+ (Discourse)	RSS lastBuildDate ticks per post
Volume	30 latest items / pull (Discourse default)	`<item>` count in captured fixture
Identifier	`ESA-YYYY-NN` (Elastic mint) + CVE refs in body	Title format `<Product> X.Y.Z Security Update (ESA-YYYY-NN)`; CVE refs scraped from `<description>` HTML
Backfillability	RSS = 30 latest only; full archive via Discourse JSON API (out of scope this round)	RSS truncates at 30
Anti-bot	None	Public read, no auth
Pagination	n/a (live 30)	RSS feed not paginated
Freshness signal	RSS body sha256 → `BulkDataDumpTracker` row `source='elastic'`	`db.GetTracker(…, "elastic")` at `main.go:72`, `db.UpsertTracker` at `main.go:300`
Schema	Discourse RSS 2.0	`<rss version="2.0" xmlns:dc=…>`

3. Architecture

graph LR Source[discuss.elastic.co
RSS feed] --> HTTP[httpclient.New] HTTP --> ParseRSS[elastic.ParseFeed] ParseRSS --> ParseItem[elastic.ParseItem] ParseItem --> Extract[CVE refs + ESA id + product] Extract --> Tx[(pgx Tx)] Tx --> Pipeline[processOneCVE
db.UpsertCVEMetadata] Pipeline --> CVEMetadata Pipeline --> CVEAlias[(db.InsertAliases)] Tx -.commit.-> S3[s3client.ArchiveRecord] Extract --> CRIT[mapElasticToCRITs
elastic_cloud + per-product] CRIT --> Stage[critutil.StageCandidate] Stage --> Drain[critpublisher.DrainKeys] Drain --> CritRecord

4. Source → DB field mapping

Source field	Target
`<title>`	`CVEMetadata.title` (sans ESA-* parenthetical)
`ESA-YYYY-NN`	minted-id GCVE alias OR direct alias when CVE-* present
First `CVE-YYYY-NNNN` in body	`CVEMetadata.cveId` (preferred)
`<pubDate>`	`CVEMetadata.datePublished` (Unix sec)
`<link>`	`CVEMetadata.sourceAdvisoryRef`
`<description>` HTML, stripped	`CVEDescription.value`
Detail page URL on discuss.elastic.co	`CVEMetadataReferences.url` (type=advisory)
Product token in title (Kibana / Elasticsearch / Logstash / Beats / Fleet / OTel Java / Package Registry)	`CVEMetadata.affectedProduct` + CRIT service inference

5. Identifier policy

CVEMetadata.cveId: prefer first CVE-YYYY-NNNN extracted from description body. When absent, mint ESA-YYYY-NNNN via db.LoadMaxGcveSequence("ESA", year) (twilio pattern).
CVEAlias: ESA-YYYY-NN always written as alias of the cveId (canonical alias-write contract via db.InsertAliases).
Bundle suppression does apply. db.InsertAliases drops every CVE-prefixed alias whenever the primary id is a CVE and the alias list holds more than one CVE (internal/db/cvealias.go:65-86). Most ESAs carry one or two CVEs, so the sibling edge survives (8 cross-CVE discoveredFrom='elastic' edges exist in production); an ESA that bundles three or more CVEs loses its CVE↔CVE sibling links and keeps only the ESA-*/GHSA-* edges.
GHSA refs in title (e.g. (ESA-2026-22 / GHSA-xw7x-h9fj-p2c7)): also written as alias.

6. CRIT / VEX

Default emission: dual envelopes per CVE — elastic_cloud/deployment (provider_only/automatic) + matching self-managed service per title keyword (elasticsearch, kibana, logstash, beats, fleet, package_registry).
VEX status: "fixed" — Elastic only publishes ESAs against shipped releases.
temporal.service_available_date: 2012-02-01 (Elastic launch).
temporal.provider_fix_date: RSS pubDate (when fixed releases ship).

7. S3 layout

Archive on success: elastic/files/{sha256}/{ESA-YYYY-NN}.json
Quarantine on failure: failed-feeds/elastic-rss-processor/{date}/{reason}/{filename}
Reasons: parse-error (RSS unmarshal), store-error (DB tx)

8. Error handling & Slack

Per-record failures non-fatal; quarantine + count.
RSS fetch failure fatal (Errored).
Slack: Started, Errored, NoWork, Completed.

9. Performance

Sequential walk of 30 items; per-record DB upsert ~10 ms.
Soft deadline: only when EXPECTED_DURATION_MINUTES is set (main.go:135-138), so a local backfill runs unbounded. The margin is hardcoded as minutes − 10 rather than internal/rundeadline.Soft, which with expected_duration_minutes = 15 leaves 5 minutes of working time — enough for a 30-item feed, but see ORCH-08.
Resume does not currently work. main.go:117 loads db.LoadProcessedHashes("elastic") and main.go:195 compares seen[adv.EsaID] against the item’s content hash, but two things break it: db.UpsertCVEMetadata does not write the sourceFileHash column at all (internal/db/cvemetadata.go:50-77), and the row that would hold the hash is keyed by the CVE id, not the ESA id. All 130 source='elastic' rows have a null sourceFileHash, and the CVE rows show fetchCount up to 38. What actually limits work is the 24-hour tracker freshness gate at main.go:70-85.

9a. Failure modes

Symptom	Cause
`data is fresh, skipping` + NoWork	Less than `frequencySecs = 86400` since the tracker’s `lastProcessedAt` (`main.go:76`). Because the schedule period is also exactly 24h, a run that fires a few seconds early no-ops and the feed is not read that day.
Every advisory re-stored each run (`fetchCount` climbing)	Dead resume, above.
`soft deadline reached, stopping cleanly`	5-minute headroom exhausted (ORCH-08).
Fatal exit on feed fetch/parse	`main.go:98-114` treats a feed fetch or parse failure as fatal after quarantining the payload.

10. Backfill

Same binary; justfile recipe unsets EXPECTED_DURATION_MINUTES.
Full historical archive via Discourse JSON API: separate effort.

S3 Persistence

Archive path: elastic/files/{sha256}/{ESA-YYYY-NN}.json ✓ — uploader.ArchiveRecord at main.go:240
Quarantine path: failed-feeds/elastic-rss-processor/{YYYY-MM-DD}/{reason}/{filename} ✓ — uploader.Quarantine at main.go:109, uploader.QuarantineRecord at main.go:230
Failure reasons emitted: parse-error (feed unmarshal), store-error (record transaction)

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.