twilio-fetch-processor — Design

1. Overview

• Purpose: Ingest Twilio security advisories from the Twilio Trust Center and Changelog RSS feed, creating CVEMetadata records and CRIT (cloud-resource) candidates.
• Source URLs:
  • Trust Center: https://security.twilio.com/ (HTML scraping)
  • Changelog RSS: https://www.twilio.com/en-us/changelog.feed.xml (RSS 2.0)
• Owner: Vulnetix
• Licence: Public / vendor-published security advisories
• Schedule cadence: Runs weekly on Tuesdays at 05:00 UTC (cron(0 5 ? * TUE *)).
• ECS resources: cpu_units=256, memory_mb=512, expected_duration_minutes=15
• Reads: Twilio Trust Center HTML, Twilio Changelog RSS, BulkDataDumpTracker (freshness), db.LoadProcessedAdvisoryRefs (resume, keyed by sourceAdvisoryRef = advisory URL), db.LoadMaxGcveSequence
• Writes: CVEMetadata, CVEDescription, CVEMetadataReferences, CVEAlias, GcveIssuance, GcveAlias, BulkDataDumpTracker, CritRecord (via CRIT drain), S3 archive + quarantine
• Not wired: internal/aienrich; no CVEMetric, CVEProblemType, CVEAffected or PackageVersion rows (the producer calls db.Upsert* directly rather than going through processor.StoreCVESourceData)

2. Source contract (from Phase 0 recon)

Trust Center (security.twilio.com)

• Format: HTML (SafeBase/Drata portal), no API, no RSS
• Volume: Extremely low (3 updates total as of 2024-07)
• Identifiers: Mixed — one item has CVE-2024-39891, two have no CVE
• Fields per update: Title, date (sparse), category tag (General/Incidents/Vulnerabilities), free-text body
• Anti-bot: None observed; standard browser headers sufficient
• Pagination: None
• Freshness: No Last-Modified/ETag; the implementation uses a combined item-count hash (see §9)

Changelog RSS (twilio.com/en-us/changelog.feed.xml)

• Format: RSS 2.0 XML
• Volume: ~30 entries in feed, mostly product announcements
• Identifiers: No CVEs in current feed; security-adjacent entries are certificate changes, compliance announcements, library updates
• Fields: title, link, pubDate, guid, description (no category element)
• Anti-bot: None
• Pagination: None (single feed)
• Freshness: Feed-level; use combined content SHA

Combined

• Cadence: Daily is sufficient; source changes infrequently
• Volume: 0-5 new advisories per run
• Identifier: CVE-prefixed when available, source-prefixed (TWILIO-YYYY-NNNN) otherwise
• Backfillability: Full historical archive is small enough to reprocess on every run; resume set (LoadProcessedHashes) skips unchanged items

3. Architecture diagram

4. Source → DB field mapping

5. Identifier policy

• Minted IDs follow the GCVE pattern: TWILIO-YYYY-NNNN with GCVE-110-TWILIO-YYYY-NNNN issuance records
• Sequence loaded via db.LoadMaxGcveSequence at startup, incremented in-memory, persisted via db.UpsertGcveIssuance
• Bundle suppression: N/A (Twilio advisories are 1:1 with CVEs or have no CVEs)
• Because non-CVE changelog items are keyed by their URL in adv.ID, the db.InsertAliases call also mints a placeholder CVEMetadata row whose cveId is the changelog URL. Those rows are alias shells (no title, no datePublished) and must be excluded from any field-coverage measurement.

⚠ Known defect — --force re-mints instead of reusing. The mint branch allocates a fresh sequence number whenever the URL resume set is bypassed (main.go:253 guards on processedURLs[adv.URL] && !*force, and main.go:267-274 mints unconditionally on the else branch). There is no lookup of an already-issued TWILIO-… id for a known sourceAdvisoryRef, so a forced run duplicates every non-CVE advisory under a new id. Production evidence: 4 of the 15 minted rows are duplicates — TWILIO-2026-0006/0015, 0007/0014, 0008/0013, 0009/0012 each pair share one sourceAdvisoryRef.

6. CRIT / VEX

Provider/Service/ResourceType triples

Extended dictionary

internal/critutil/dictionaries/extended/twilio.json — 13 entries covering all Twilio services.

VEX status derivation

• “fixed” → description contains “fixed”, “update”, “patched”
• “not_affected” → description contains “no action”, “not affected”, “no impact”
• “under_investigation” → default

Fix propagation derivation

• “automatic” → “no action required”, “automatically”
• “version_update” → “update”, “upgrade” (default)
• “config_change” → “reconfigure”, “configuration”
• “credential_rotation” → “rotate” + “credential”

7. S3 / source-file archive layout

• Archive: {source}/files/{sha256}/{cveId}.json
• Quarantine: failed-feeds/{processorName}/{date}/{reason}/{filename}
• CRIT pending: crit-candidates/pending/{YYYY}/{MM}/{DD}/{cveId}/{provider}/{service}/{resource_type}/{sha}.json

8. Error handling & Slack

• Per-record failure: Log warning, increment failed count, continue to next item
• Retry profile: 3 attempts, exponential backoff (1s, 2s), transient errors retried
• Slack events:
  • Started — at processor startup
  • Completed — with stats: fetched, stored, skipped, failed, critStaged
  • Errored — when any record fails or fatal error occurs
  • NoWork — when tracker SHA matches (data unchanged)

9. Performance

• Concurrency: Sequential per-item processing (low volume, no need for fan-out)
• Per-request timeout: 30s HTTP timeout (httpclient.New(30 * time.Second)), 2-minute DB transaction timeout
• Rate-limit pacing: None required (only 2 HTTP requests per run)
• Soft deadline: EXPECTED_DURATION_MINUTES - 10min, but the default is a hardcoded 15 minutes applied unconditionally (main.go:68-72), so the justfile recipe unsetting the env var does not remove the cap.
• Resume strategy: db.LoadProcessedAdvisoryRefs — the set of sourceAdvisoryRef values already stored under source='twilio', matched against adv.URL. Not LoadProcessedHashes.
• Global freshness gate: BulkDataDumpTracker.sha256 for source twilio. ⚠ The value compared is sha256("tc:<count>|rss:<count>") (main.go:146) — a hash of the two item counts, not of the content. Any change that leaves both counts unchanged (an edited advisory, or one added and one removed) is reported as fresh and the entire run is skipped.

10. Backfill

• No separate backfill binary needed; the scheduled processor can process the full historical archive in seconds
• just go-twilio-fetch-backfill TARGET=prod FORCE=true LIMIT=0 runs the full archive
• Expected runtime on prod data: <1 minute

S3 Persistence

• Archive path: twilio/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/twilio-fetch-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: parse-error, store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.