enrich-first-epss Design

Enriches CVE records with FIRST.org EPSS (Exploit Prediction Scoring System) scores.

Overview

  • Batch size: 50 CVEs per run, issued as 2 chunked API calls (25 CVE IDs per URL)
  • Schedule: Hourly (cron(0 * * * ? *)), 256 CPU / 512 MB, EXPECTED_DURATION_MINUTES=3
  • Concurrency: Master lock via BulkDataDumpTracker (source=enrich_first_epss); exits if < 8 min since last run. The lock is reset to 0 on completion so the next hourly invocation is never blocked by it.
  • State: EpssScore table itself — a CVE with at least one score row is permanently skipped
  • Retry: CVEs where FIRST returns no data are retried on subsequent runs (no score saved = not done)

This is the new-CVE trickle, not the daily refresh. The anti-join skips any CVE that already has any EpssScore row, so this task only ever mints first-ever scores for CVEs published since the last bulk import. Re-scoring the existing history is epss-csv-backfill’s job.

Processing Flow

  1. Check lock (enrich_first_epss) — return early if a run started < 8 min ago
  2. Acquire lock
  3. Find batch (db.FindCVEsWithoutEpss, bounded to a 60 s context): CVEs from CVEMetadata matching ^CVE-[0-9]{4}-[0-9]+$ with a non-null datePublished and no EpssScore row, ordered by datePublished DESC (newest CVEs first). The query over-fetches 4 × limit and de-duplicates in Go, because the same cveId appears once per source.
  4. Batch API call, chunked at 25 IDs per request: GET https://api.first.org/data/v1/epss?cve=CVE-1,...,CVE-25. Chunking keeps the URL under typical query-length limits and stops one malformed ID from poisoning the whole 50-CVE call — a failing chunk is skipped, its CVEs simply do not appear in the merged response.
  5. Parse response: build scoreMap from the merged data[] array
  6. For each CVE in batch:
    • If found in scoreMap: parse score/percentile strings to float64, build EpssScoreRow (fetchedAt/createdAt = time.Now().UnixMilli(); dateString falls back to today’s UTC date when FIRST omits it)
    • If not found, or either float fails to parse: count as skipped (retried next run since no EpssScore row exists)
  7. Bulk upsert all rows via db.UpsertEpssScores() (ON CONFLICT DO NOTHING)
  8. Mark CVEMetadata.lastEnriched for CVEs that got scores — one UPDATE … WHERE "cveId" = $1 per CVE, which touches every source row for that CVE
  9. Release lock (set lastProcessedAt = 0)

Deadline behaviour

There is no application-level soft deadline in this binary. internal/notify starts its overtime watcher at exactly EXPECTED_DURATION_MINUTES (3), and on firing it publishes task.overtime — which pages — and cancels the run context. That budget has to cover the 60 s find query, up to 2 × 30 s of FIRST API calls, the bulk upsert, and 50 sequential lastEnriched updates. If overtime alerts appear for this task, raise the budget in both scripts/task-manager.toml and terraform/go-schedules.tf — the ECR hook injects the toml value, so the toml is what actually reaches the container.

Tables Written

  • EpssScore — EPSS scores (cve, dateString, score, percentile, modelVersion, fetchedAt)
  • CVEMetadata — lastEnriched timestamp for enriched CVEs
  • BulkDataDumpTracker — the enrich_first_epss master lock row

Tables Read

  • CVEMetadata — find CVEs needing EPSS scores
  • EpssScore — check which CVEs already have scores (NOT EXISTS anti-join)
  • BulkDataDumpTracker — master lock check

API

  • Endpoint: GET https://api.first.org/data/v1/epss?cve={cve1},{cve2},...
  • Auth: None (public API)
  • Batch: Comma-separated CVE IDs in query parameter, chunked at 25 per request (internal/firstepss/client.go maxBatchSize)
  • Response: JSON with data[] array of {cve, epss, percentile, date, model_version}
  • Errors: 404 → treated as “no data” (nil response, not an error); 429 and any other non-200 → chunk error. FetchBatch returns an error only when every chunk failed.
  • Rate limit: Unknown; browser-style headers set, 30 s per-request timeout

Key Differences from Coalition ESS

  • No VVD records, artifacts, links, or aliases — pure score enrichment
  • One chunked batch call per run instead of per-CVE requests
  • State tracked via EpssScore table, not per-CVE BulkDataDumpTracker entries
  • Simpler resource requirements: 256 CPU / 512 MB memory

S3 Persistence

  • Archive: ⚠ Not yet implemented — requires record reconstruction (DB row → canonical JSON).
  • Quarantine: ⚠ Not yet implemented — same reason.
  • Likely reasons when implemented: enrich-error

This is an enrichment processor; it reads from CVEMetadata rather than ingesting raw feeds, so there is no original payload to archive verbatim. See S3 Persistence Contract § Processors whose unit-of-work is not a file.