CISA KEV Processor — Design Document

1. Overview

Purpose: Fetch the CISA Known Exploited Vulnerabilities (KEV) catalog and upsert all entries into the Kev table.

Data source: CISA KEV JSON feed — https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json

Schedule: Daily at 01:30 UTC (cron(30 1 * * ? *) — Terraform EventBridge)

Timeout: 60 minutes

Resources: 256 CPU units, 512 MB memory

What it reads:

  • CISA KEV catalog (single JSON, ~1200+ entries, full re-fetch each run)
  • Read replica: BulkDataDumpTracker row for cisa_kev (elapsed-time freshness check)

What it writes:

  • Kev table (batch upsert, 500 entries at a time)
  • BulkDataDumpTracker

Environment variables:

  • DATABASE_URL — required
  • DATABASE_URL_READ — optional

2. Business Logic

Freshness Check (Time-Based)

Unlike SHA-based freshness, CISA KEV uses elapsed-time comparison:

  1. Read BulkDataDumpTracker for cisa_kev
  2. Compute elapsed = now - tracker.lastProcessedAt (milliseconds)
  3. Compute frequencyMs = tracker.frequency * 1000 — read from the row, not from the processor. The frequencySecs = 21600 constant in main.go is dead code: db.UpsertTracker hardcodes frequency = 86400 on insert and never updates it on conflict. The live cisa_kev row predates that helper and still carries 21600 (6 h), which is why the daily schedule is never gated.
  4. If elapsed < frequencyMs → skip with “data is fresh”

First run (no tracker row) always proceeds.

⚠ Do not “fix” the row to 86400. With a 24-hour window and a daily cron, the next day’s run starts before 24 h have elapsed (because lastProcessedAt is written at the end of the previous run, i.e. cron-time + runtime), so the run skips and the catalog effectively refreshes every second day. The sibling eol-json-processor and eukev-json-processor rows carry 86400 and demonstrably exhibit this. The freshness window must stay strictly shorter than the schedule interval.

Full Catalog Fetch

The CISA KEV endpoint always returns the complete catalog. There is no incremental/delta mechanism — every run re-fetches all entries.

Batch Upsert

Entries are processed 500 at a time. Each batch of 500 runs inside one transaction (db.WithTx), and each entry inside it is wrapped in its own SAVEPOINT sp_kev so a single failing upsert rolls back only that entry and the rest of the batch still commits. Failed entries increment an error counter but do not stop processing; any error at all makes the run exit 1 after the tracker is written.

Date Parsing

dateAdded and dueDate are YYYY-MM-DD strings. Parsed to Unix milliseconds. catalogReleaseDate tries RFC3339 first, then date-only format.

⚠ On parse failure parseDateMs substitutes fetchedAt (the run time) and the ON CONFLICT clause assigns it unconditionally (main.go:293,296), so an entry whose upstream date ever became unparseable would have its dateAdded/dueDate rewritten to the run day on every subsequent run. The sibling eukev-json-processor was fixed for exactly this (it COALESCEs to the stored value instead). No source='CISA' row is currently affected — measured: 0 of 1,661 rows have dateAdded = fetchedAt, and dateAdded spans 2021-11-03 … 2026-08-05 — because CISA has never emitted a malformed date. It remains a latent correctness bug in a column that drives remediation deadlines.

Deadline

EXPECTED_DURATION_MINUTES − 10 (50 min on the 60-min ECS budget), falling back to a hardcoded 15 minutes when the env var is unset — which is the state just go-cisa-kev-json-backfill creates, since the recipe unsets it. At ~1,700 entries the bound is never reached, but it contradicts the backfill-has-no-deadline rule; the deadline should be left at its zero value when the env var is absent, and the margin should come from rundeadline.Soft rather than a hardcoded mins − 10.

CWEs as JSONB

cwes field marshaled to JSON string and stored as cwesJSON (JSONB column).

Idempotency

ON CONFLICT ("cveID", "source") DO UPDATE — fully idempotent, safe to re-run.

Tracker Update

UpsertTracker(cisa_kev, sha256="", totalCVEs=count) — no SHA256 for CISA (time-based), sha256 stored as empty string.

3. Architecture Diagram

graph TD subgraph "cmd/cisa-kev-processor/" MAIN[main.go] UPSERT[upsertBatch_ — SQL upsert logic] end subgraph "internal/kev/" KEVPKG[cisa.go — Parse, Entry struct] end subgraph "internal/db/" POOL[pool.go — Pool] TRACK[tracker.go — GetTracker, UpsertTracker] end MAIN --> TRACK MAIN --> KEVPKG MAIN --> UPSERT MAIN --> POOL UPSERT -->|SQL| WRITE[pool.Write.Exec]

4. Deployment Diagram

flowchart TD GHA[GitHub Actions
go-ecr-deploy.yml] -->|push ARM64 image| ECR[ECR: vdb-manager
tag: cisa-kev-processor-sha-xxx] ECR --> TASKDEF[ECS Task Definition
go-processor-cisa-kev-processor] TASKDEF --> EB[EventBridge Schedule
go-cisa-kev-json-processor
cron 30 1 * * ? *] EB -->|trigger| FARGATE[ECS Fargate Task
vdb-scheduler cluster
ARM64 ap-southeast-2] FARGATE --> CW[CloudWatch Logs
/ecs/vdb-scheduler/go-cisa-kev-json-processor] FARGATE -->|HTTPS GET| CISA[cisa.gov/sites/.../
known_exploited_vulnerabilities.json] FARGATE --> WRITE[RDS Write Proxy] FARGATE --> READ[RDS Read Replica]

5. Processing Flow

flowchart TD START([Start]) --> ENV{DATABASE_URL set?} ENV -->|no| FAIL([Exit 1]) ENV -->|yes| CONNECT[Connect to DB pool] CONNECT --> FRESH{Tracker elapsed
< 6 hours?} FRESH -->|yes| EXIT0([Exit 0 — fresh]) FRESH -->|no or no tracker| FETCH[HTTP GET CISA KEV JSON] FETCH -->|error| FAIL FETCH -->|ok| PARSE[kev.Parse catalog JSON] PARSE -->|error| FAIL PARSE -->|ok, ~1200 entries| DATEPARSE[Parse catalogReleaseDate
RFC3339 then date-only] DATEPARSE --> BATCH[For each batch of 500 entries] BATCH --> ENTRY[For each entry] ENTRY --> CWEJSON[Marshal CWEs to JSON] CWEJSON --> SQL[INSERT ... ON CONFLICT UPDATE
Kev table] SQL -->|error| ERRCOUNT[increment errors] SQL -->|ok| COUNT[increment total] ERRCOUNT --> ENTRY COUNT --> ENTRY ENTRY -->|done| BATCH BATCH -->|done| TRACKER[UpsertTracker cisa_kev] TRACKER --> ERRCHECK{errors > 0?} ERRCHECK -->|yes| FAIL2([Exit 1]) ERRCHECK -->|no| DONE([Exit 0])

6. Data Mapping

erDiagram BulkDataDumpTracker { string source PK "cisa_kev" bigint lastProcessedAt "Unix ms — freshness basis" int frequency "21600 seconds = 6 hours" string sha256 "empty — not used" int totalCVEs "entries upserted" } Kev { string cveID PK string source PK "CISA" string vendorProject string product string vulnerabilityName bigint dateAdded "Unix ms from YYYY-MM-DD" string shortDescription string requiredAction bigint dueDate "Unix ms" string knownRansomwareCampaignUse string notes jsonb cwesJSON "array of CWE strings" bigint fetchedAt string catalogVersion bigint catalogReleaseDate }

7. S3 notes

The generated section below is derived from the compliance matrix; two details it cannot express:

  • The live prefix is cisa_kev/, with an underscore. The processor passes its sourceSlug = "cisa_kev" constant straight to uploader.Archive, while the matrix canonicalises _ → - for display. Search S3 for cisa_kev/files/….
  • Unit of work is one KEV entry, not the catalog file. Each entry is re-serialised to canonical JSON and keyed {cveID}.json (per S3 Persistence Contract § Processors whose unit-of-work is not a file). Payloads are computed before the batch transaction opens and uploaded after it settles, so no DB connection is ever held during an S3 PUT. The one exception is parse-error, which quarantines the whole known_exploited_vulnerabilities.json body — at that point no entry has been parsed out of it.
  • On a batch-level transaction failure, every entry previously marked committed in that batch is reclassified as failed and quarantined, since the rollback means none of them persisted.

S3 Persistence

  • Archive path: cisa-kev/files/{sha256}/{filename} ✓
  • Quarantine path: failed-feeds/cisa-kev-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
  • Failure reasons emitted: parse-error, store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[cisa-kev-processor] PROC -->|success| ARCHIVE[("S3: cisa-kev/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/cisa-kev-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.