MITRE CVE Processor — Design Document

1. Overview

Purpose: Download the MITRE cvelistV5 GitHub archive (tar.gz), stream-extract all CVE-*.json files, and enqueue them for downstream processing.

Data source: GitHub release archive — https://github.com/CVEProject/cvelistV5/archive/refs/heads/main.tar.gz

Schedule: Daily (cron 0 2 * * *)

Timeout: 180 minutes (the archive is ~1GB+ compressed)

Resources: 512 CPU units, 2048 MB memory

What it reads:

  • MITRE cvelistV5 tar.gz archive (streamed directly from HTTP response — no temp file)

What it writes:

  • QueueMessage (source=mitre-cve, batches of 100)
  • MitreCveFile (single summary record per archive)
  • BulkDataDumpTracker (tracker key: mitre_cve_org)

Environment variables:

  • DATABASE_URL — required
  • DATABASE_URL_READ — optional

2. Business Logic

Streaming Architecture (No Temp File)

The HTTP response body is streamed through a io.TeeReader that simultaneously:

  1. Feeds bytes to gzip.NewReader → tar.NewReader for extraction
  2. Hashes the entire stream with sha256.New to compute the archive SHA256

This means the archive SHA256 is only available after the stream is fully consumed. Files are enqueued during streaming; the tracker is updated at the end with the completed hash.

Archive SHA256 Computation

io.TeeReader(resp.Body, hasher) — every byte read from resp.Body is also written to the hasher. After all tar entries are processed, io.Copy(io.Discard, teeReader) drains any remaining bytes (e.g., gzip footer) to ensure the hash covers the complete stream.

archiveSHA256 = fmt.Sprintf("%x", hasher.Sum(nil))

File Filter

Within the tar archive: path.Base(hdr.Name) — filter for files where basename starts with CVE- and ends with .json. The archive includes many non-CVE files (README, delta files, etc.) which are skipped.

JSON Validation

json.Valid(data) — each file’s bytes are validated as JSON before enqueueing. Invalid files increment the error counter and are skipped.

Batch Enqueueing

Records accumulate in a slice; when it reaches 100 entries, InsertQueueMessages is called. fileName for all messages is cvelistV5-main.tar.gz.

MitreCveFile Summary Record

After processing the full archive, a single summary record is written to MitreCveFile with:

  • archiveSha256 — computed from stream
  • filename = 'cvelistV5-main.tar.gz'
  • cveCount — total files processed
  • status = 'completed'

This is an ON CONFLICT (archiveSha256, filename) DO UPDATE — each archive run is idempotent.

No SHA256 Dedup Check at Start

Unlike other processors, mitre-cve-processor does NOT check the tracker SHA256 before downloading. The archive must be streamed to compute its SHA256. This means every run downloads the full archive.

3. Architecture Diagram

graph TD subgraph "cmd/mitre-cve-processor/" MAIN[main.go] PROCESS[processArchive
io.TeeReader + tar + sha256] MITRE_FILES[upsertMitreCveFiles
MitreCveFile summary] end subgraph "internal/queue/" QUEUE[producer.go — InsertQueueMessages] end subgraph "internal/db/" POOL[pool.go — Pool] TRACK[tracker.go — UpsertTracker] end MAIN --> PROCESS MAIN --> TRACK MAIN --> POOL PROCESS -->|batches of 100| QUEUE PROCESS --> MITRE_FILES MITRE_FILES -->|SQL| WRITE[pool.Write.Exec]

4. Deployment Diagram

flowchart TD GHA[GitHub Actions
go-ecr-deploy.yml] -->|push ARM64 image| ECR[ECR: vdb-manager
tag: mitre-cve-processor-sha-xxx] ECR --> TASKDEF[ECS Task Definition
go-processor-mitre-cve-processor] TASKDEF --> EB[EventBridge Schedule
vdb-mitre-cve-daily
cron 0 2 * * ? *] EB -->|trigger| FARGATE[ECS Fargate Task
vdb-scheduler cluster
ARM64 ap-southeast-2] FARGATE --> CW[CloudWatch Logs
/ecs/vdb-scheduler/mitre-cve-daily] FARGATE -->|HTTPS GET streaming| GITHUB[github.com/CVEProject/cvelistV5
archive/refs/heads/main.tar.gz] FARGATE --> WRITE[RDS Write Proxy
QueueMessage + MitreCveFile + BulkDataDumpTracker]

5. Processing Flow

flowchart TD START([Start]) --> ENV{DATABASE_URL set?} ENV -->|no| FAIL([Exit 1]) ENV -->|yes| CONNECT[Connect to DB pool] CONNECT --> DOWNLOAD[GET github.com cvelistV5 main.tar.gz
streaming HTTP response] DOWNLOAD -->|error| FAIL DOWNLOAD -->|ok| STREAM[processArchive
io.TeeReader → hasher + gzip → tar] STREAM --> TARLOOP[For each tar entry] TARLOOP --> FILTER{basename starts with CVE-
and ends with .json?} FILTER -->|no| TARLOOP FILTER -->|yes| READ[io.ReadAll tar entry] READ -->|error| ERR[count error
continue] READ -->|ok| VALIDATE{json.Valid?} VALIDATE -->|no| ERR VALIDATE -->|yes| ACCUMULATE[Append to records slice] ACCUMULATE --> BATCH{records >= 100?} BATCH -->|yes| FLUSH[InsertQueueMessages
source=mitre-cve
fileName=cvelistV5-main.tar.gz] BATCH -->|no| TARLOOP FLUSH --> TARLOOP TARLOOP -->|EOF| DRAIN[io.Copy Discard — drain remaining bytes for hash] DRAIN --> FLUSH_FINAL[Flush remaining records] FLUSH_FINAL --> COMPUTE[archiveSHA256 = hasher.Sum nil] COMPUTE --> MITRE_FILE[upsertMitreCveFiles
archiveSHA256 + fileCount] MITRE_FILE --> TRACKER[UpsertTracker mitre_cve_org
sha256=archiveSHA256] TRACKER --> ERRCHECK{errors > 0?} ERRCHECK -->|yes| FAIL2([Exit 1]) ERRCHECK -->|no| DONE([Exit 0]) ERR --> TARLOOP

6. Data Mapping

erDiagram BulkDataDumpTracker { string source PK "mitre_cve_org" bigint lastProcessedAt string sha256 "full stream body SHA256" int totalCVEs "records enqueued" } MitreCveFile { string uuid PK string archiveSha256 UK "stream SHA256" string filename UK "cvelistV5-main.tar.gz" int fileIndex "0 — summary record" string status "completed" int cveCount "total CVE files extracted" bigint processedAt bigint createdAt bigint updatedAt } QueueMessage { string uuid PK string source "mitre-cve" string fileName "cvelistV5-main.tar.gz" jsonb payload "raw CVE5 JSON object" string status "pending" bigint createdAt } MitreCveFile ||--o{ QueueMessage : produces

S3 Persistence

  • Archive path: cve.org/files/{sha256}/{filename} ✓
  • Quarantine path: failed-feeds/mitre-cve-processor/{YYYY-MM-DD}/{reason}/{filename} ⚠ not yet wired
  • Failure reasons emitted: schema-violation, parse-error, store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[mitre-cve-processor] PROC -->|success| ARCHIVE[("S3: cve.org/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/mitre-cve-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.