enrich-coalition-cess Design

Enriches CVE records with Coalition ESS (Exploit Scoring System) data.

Overview

  • Batch size: 50 CVEs per run
  • Schedule: Every 30 minutes (cron(*/30 * * * ? *)), expected_duration_minutes = 15
  • Concurrency: Lock via BulkDataDumpTracker (source=enrich_coalition_cess); exits if < 8 min since last run
  • State: Per-CVE rows in BulkDataDumpTracker (source=enrich_coalition_cess:{cveId})
  • Allowed prefixes: CVE- (configured via allowedPrefixes var in main.go; uses shared enrichment.FindNextBatch)
  • Minted source slug: coalition_cess (underscore — internal/coalitioncess/mapper.go:14), record ids CESS-{YYYY}-{N}
  • Soft deadline: now + (EXPECTED_DURATION_MINUTES − 10), i.e. 5 minutes on the ECS budget, defaulting to now + 60 min when the env var is unset. The loop breaks at that instant, so a run can stop part-way through its 50-CVE batch; unreached CVEs keep their claim row and are retried on a later run.

Processing Flow

  1. Check lock (enrich_coalition_cess) — exit 0 if recent
  2. Acquire lock
  3. Find next batch (50 CVEs, CVE- prefix only via enrichment.FindNextBatch): unprocessed first, then oldest reprocess
  4. Claim batch (upsert per-CVE tracker rows)
  5. For each CVE:
    • FetchCVE: GET ess-api.coalitioninc.com/cve/{cveId}
      • If 404: mark enriched, skip
    • Upload raw JSON to S3 at coalition-cess/{cveId}.json — a bare PutObject, not the contracted content-addressed {source}/files/{sha256}/{filename} archive path, and there is no quarantine path
    • Create Artifact + Link records for the S3 object (Link.url is the S3 key, not a resolvable URL)
    • Upsert CessScore (probability_exploit_usage, score, variation, model version) — one row per (cve, dateString), so the table is a daily time series, not one row per CVE
    • Build cvelistv5.Record from Coalition data (CVSS → metrics, CESS → OtherMetric, exploits → references)
    • Store as CESS record via StoreRecordWithSource() (source=coalition_cess, id=CESS-{YYYY}-{N})
    • Insert CVEAlias linking CESS-* ↔ original CVE-* via db.InsertAliases
    • Update CVEMetadata.lastEnriched — note MarkCVEEnriched (internal/enrichment/tracker.go:215) filters on cveId only, so it stamps the row of every source carrying that CVE, not just the CESS row
  6. Release lock

A CVE the Coalition API does not know (HTTP 404 → info == nil) is marked enriched and skipped, so it is not retried on the next run.

Tables Written

  • CessScore — exploit probability scores (one row per CVE per day)
  • CVEMetadata — CESS records (source=coalition_cess), plus a lastEnriched stamp on all sources’ rows for the CVE
  • CVEAlias — CESS ↔ CVE alias mappings
  • Artifact — S3 artifact metadata
  • Link — artifact download links
  • CVEDescription — enriched descriptions
  • CVEMetadataReferences — exploit references
  • CVEMetric — CVSS + CESS metrics
  • CVEProblemType, CVEAffected — whatever cvelistv5.storeContainer derives from the synthesised record
  • BulkDataDumpTracker — run lock (enrich_coalition_cess) + per-CVE claim rows (enrich_coalition_cess:{cveId})

API

  • Endpoint: GET https://ess-api.coalitioninc.com/cve/{cveId}
  • Auth: None (public API)
  • Rate limit: Unknown; browser UA used

S3 Persistence

  • Archive: ⚠ Not yet implemented — requires record reconstruction (DB row → canonical JSON).
  • Quarantine: ⚠ Not yet implemented — same reason.
  • Likely reasons when implemented: enrich-error

This is an enrichment processor; it reads from CVEMetadata rather than ingesting raw feeds, so there is no original payload to archive verbatim. See S3 Persistence Contract § Processors whose unit-of-work is not a file.