CVE Prefix JSON Processor — Design Document

1. Overview

Purpose: Daily export of all CVEMetadata records grouped by ID prefix (e.g. CVE, GHSA, CNVD, MAL, BDU) as cvelistv5-format tar.gz archives to S3, with individual raw JSON per record for new/modified entries. The Vulnetix-minted prefixes (ACSC, ANCHORE, BUGCROWD, CERTCC, CERTFI, CESS, CISA, CLOUD, CONFSA, ESA, GENTOO, HCSEC, HSEC, ISC, MAGEIA, NCSC, OSM, TWILIO, ZSB) are excluded — they are exported via the GCVE feed (gcve-json-processor) instead.

Schedule: Daily at 02:00 UTC (cron 0 2 * * *)

Timeout: 120 minutes

Resources: 1024 CPU units, 6144 MB memory (raised from 4 GB after a late-run OOM; steady state is ~1.5 GB, but a single large prefix spikes; ephemeral local disk scales with total archive size — allow ~3× the largest merged .tar.gz for chunks + merged output). Sets READ_STATEMENT_TIMEOUT=10min so LoadDistinctPrefixes (a SPLIT_PART-grouped scan of 4.8M CVEMetadata rows) is not cancelled by the read pool’s 60s cap.

What it reads:

• Read replica: CVEMetadata table (one row per distinct cveId, best source via priority ordering, ordered by cveId)
• Read replica: Kev, VulnCheckKEVCVE, CVEAlias tables (per-prefix read-through cache; aliases are direct/one-hop only)
• S3: {prefix-lower}-archive/latest/manifest.json (previous run’s ID set for delta detection)

What it writes (per prefix, e.g. ghsa-archive/):

• S3: {prefix-lower}-archive/YYYY/MM/DD/{prefix-lower}-full.tar.gz — dated immutable snapshot
• S3: {prefix-lower}-archive/latest/{prefix-lower}-full.tar.gz — overwritten each run
• S3: {prefix-lower}-archive/recent/{prefix-lower}-new.tar.gz — records not in previous manifest
• S3: {prefix-lower}-archive/modified/{prefix-lower}-modified.tar.gz — records updated in last 24h
• S3: {prefix-lower}-archive/cves/{PREFIX}/{year}/{cveId}.json — individual raw JSON (new/modified only)
• S3: {prefix-lower}-archive/latest/manifest.json — updated ID+metadata index

Environment variables:

• DATABASE_URL — required (write-side, used as read fallback if DATABASE_URL_READ absent)
• DATABASE_URL_READ — optional read replica
• S3_BUCKET_NAME — required
• SNS_TOPIC_ARN — optional (enables overtime/error SNS notifications)
• EXPECTED_DURATION_MINUTES — optional (default none; triggers overtime notification if exceeded)

2. Business Logic

Prefix Discovery

On startup, LoadDistinctPrefixes queries CVEMetadata for all distinct SPLIT_PART("cveId", '-', 1) values excluding the Vulnetix-minted prefixes (ACSC, ANCHORE, BUGCROWD, CERTCC, CERTFI, CESS, CISA, CLOUD, CONFSA, ESA, GENTOO, HCSEC, HSEC, ISC, MAGEIA, NCSC, OSM, TWILIO, ZSB), ordered by distinct cveId count descending. Those minted prefixes are exported via the GCVE feed (gcve-json-processor) instead. The --prefix flag filters to a single prefix for targeted runs.

sanitizePrefix maps each prefix to a filesystem/S3-safe segment (replacing any character outside [a-z0-9._-]). Some CVEMetadata rows carry a URL as their id, so SPLIT_PART yields malformed “prefixes” like https://github.com/…; without sanitising, those broke os.MkdirTemp.

Soft-deadline guard

The prefix loop stops starting new prefixes once the run passes its soft deadline (rundeadline.Soft(EXPECTED_DURATION_MINUTES)), reporting task.completed with a prefixesDeferred count rather than running into the overtime watcher. The export is resumable per prefix (the on-disk record cache) and per CVE (the sourceFileHash skip fast-path), so deferred prefixes are picked up on the next run.

Deduplication at the DB Layer

LoadCVEMetadataByPrefix returns exactly one row per distinct cveId using DISTINCT ON ("cveId") with source-priority ordering:

1. cve.org → priority 1 (most authoritative)
2. nist-nvd → priority 2
3. github → priority 3
4. anchore_adp → priority 4
5. any other → priority 5

updatedAt is MAX(lastEnriched) across all sources for the same cveId, so isModified fires when any source was recently updated regardless of which source won the priority selection.

Delta Detection (Manifest-Based)

On each run the processor compares the current CVEMetadata row set against the previous manifest:

1. Load {prefix-lower}-archive/latest/manifest.json from S3 (previous run’s set of cveIds + metadata).
2. A record is new if its cveId does not appear in the previous manifest.
3. A record is modified if its updatedAt ≥ nowMs − 86_400_000 (updated within the last 24 hours).
4. If --force is not set and len(allRows) == len(previousIDs) with no new or modified records, skip the prefix without rebuilding archives.

Per-prefix read-through cache

Each prefix gets a fresh ExportCache (export.NewExportCache) — a read-through LRU for Kev, VulnCheckKEVCVE, and the record’s direct aliases. It is recreated per prefix, not shared across the run: prefixes are disjoint CVE-ID spaces so a shared cache yields no cross-prefix hits while retaining every prefix’s entries, which previously grew memory until the 4 GB OOM. Workers receive the ExportCache pointer; a cache miss falls through to a DB query.

Aliases are direct only — the one-hop neighbourhood from CVEAlias (db.LoadDirectAliases). The earlier design walked the full transitive closure (an unbounded BFS over the connected component), but the alias graph has hubs (e.g. CVE-2023-44487, degree ~18k) that merged unrelated CVEs and drove the OOM; the exported x_vulnetix.aliases now carries only direct.

Batched Record Building

The CVE prefix alone can produce ~280K records. Holding every JSON payload in memory plus the resulting bytes.Buffer archive would OOM the container (and the parent TTY when run locally), so records are built in bounded batches rather than all at once:

1. Records are processed in batches of batchSize = 1000 rows.
2. Inside each batch a parallel errgroup (default 20 workers) calls export.BuildRecord and marshals JSON.
3. After g.Wait() the main goroutine serially iterates the batch results, writing each entry to the appropriate on-disk chunked archives (full / recent / modified) and enqueueing the raw-JSON upload.
4. The batch’s results slice is released and runtime.GC() is called at the end of every batch so heap usage drops back to baseline before the next batch starts.

Peak batch memory is approximately batchSize × avg(JSON size) ≈ 1000 × 20 KB ≈ 20 MB regardless of how many records exist.

Streaming On-Disk Archives

Each of the three output archives (full, recent, modified) is backed by a db.StreamingArchive that writes tar entries straight through tar.Writer → gzip.Writer → os.File into a single .tar.gz at <tmp>/{name}.tar.gz. Peak memory is O(one entry); peak disk is the compressed archive size (roughly 1/10 of the uncompressed payload for JSON). No intermediate uncompressed chunks accumulate, so even the CVE prefix (~280 K records) fits comfortably on a Fargate ephemeral volume.

On Close the tar + gzip + file descriptors are flushed and closed. The file is uploaded directly from disk; Remove() deletes each archive right after its uploads finish so disk usage shrinks progressively. The temp dir is also removed via defer os.RemoveAll as a final safety net.

Raw JSON Uploads

Raw JSON uploads overlap with batch processing via a bounded channel (rawUploadQueueSize = 200) feeding a background errgroup of 10 workers. Entries for new/modified records (or all records under --force) are enqueued as each batch is collected. After the final batch the upload channel is closed and the pool drained before the archives are closed.

⚠ Two silent-loss paths, both non-fatal and both leaving the run green.

1. The channel send is non-blocking (select … default:): when all 200 slots are full the job is discarded and only uploadDropped is incremented. That record’s per-record {cveId}.json never lands in S3 and nothing retries it, while the manifest is saved as if it had.
2. A failed PutObject increments the error counter and logs a warning.

Separately, a prefix whose processPrefix returns an error is logged and skipped (continue), and its ExportCache init failure is handled the same way — the run still ends in task.completed. So a wholly failed prefix produces no alarm. Watch uploadsDropped in the per-prefix progress lines and the skipped total in the completion stats.

Archive Upload

The .tar.gz files written by StreamingArchive are uploaded from disk via PutObject with the *os.File as the Body. The SDK streams content and sets Content-Length from Stat().Size(), avoiding any bytes.Buffer copy. The full archive file is re-opened to upload once for the dated key and once for the latest key.

Every S3 put (s3PutFile / s3PutJSON / saveManifest) routes through retryPut, which builds a fresh context.WithTimeout rooted at context.Background() on every attempt — the caller-supplied ctx is polled only between attempts, never passed into the SDK. This makes each PutObject immune to poisoning from a cancelled or expired parent ctx. On context.DeadlineExceeded / context.Canceled the call is retried up to s3PutMaxAttempts (3) times with 500 ms × attempt-number backoff; non-ctx errors fall through unchanged. Per-attempt budgets: 10 min for archive files, 60 s for raw JSON blobs, 2 min for the manifest.

Idempotency

Re-running with the same data produces the same archives. The manifest diff ensures no-op exits when nothing has changed. --force bypasses the diff check.

Implicit Resume (Record Cache)

The expensive part of each prefix is export.BuildRecord — it issues multiple DB queries per record, and the CVE prefix alone has ~280K records. To survive crashes, container kills, and ECS timeouts without throwing that work away, every built record is persisted as a JSON file under a per-prefix resume cache directory:

<tmpRoot>/cveprefix-json-processor-cache/{prefix-lower}/records/{year}/{cveId}.json

Flow:

1. On startup main opens (and creates if missing) <tmpRoot>/cveprefix-json-processor-cache/. If --build-clean is set the entire cache base is wiped first, forcing every prefix to rebuild from the DB.
2. Each processPrefix opens its own per-prefix subdirectory. Entry count at start is logged for visibility.
3. Inside the worker, for each row the cache is consulted by path. A hit is valid only if the cached file’s mtime in milliseconds is ≥ the row’s updatedAt — any newer upstream change invalidates the cache entry and forces a rebuild.
4. On a miss the record is built, marshalled, and written atomically (*.tmp + rename) to the cache. The worker then proceeds exactly as before (archive writes, raw-JSON upload enqueue, manifest entry).
5. Archives themselves are not resumable — tar.gz has no safe append point — so they live in a separate ephemeral archive tmp dir (cveprefix-{prefix-lower}-archives-*) that is recreated every run and defer os.RemoveAll’d on exit. Rebuilding the tar from the cache is cheap; it’s the record building that was expensive.
6. On the successful completion of an individual prefix its per-prefix cache subdirectory is wiped. If a prefix errors out (return before wipe), its cache persists so the next run’s implicit resume picks up where it left off. Other prefixes that already succeeded are unaffected.

--build-clean wipes every prefix’s resume cache before any work starts, forcing all records to be rebuilt from the DB. Use it when upstream enrichment logic has changed in a way updatedAt cannot detect (e.g. a new field in BuildRecord output for unchanged rows).

Disk sizing note: the CVE prefix with ~280K records × ~20 KB compressed JSON ≈ 5.6 GB of cache after a run that processed everything. Fargate’s default 20 GB ephemeral volume fits this. The cache is only ever this large mid-interruption — successful runs reclaim it immediately.

3. Architecture Diagram

4. Deployment Diagram

5. Processing Flow

6. S3 Object Layout

{prefix-lower}-archive/          (e.g. ghsa-archive/, cve-archive/, cnvd-archive/)
├── YYYY/MM/DD/
│   └── {prefix-lower}-full.tar.gz        # immutable dated snapshot
├── latest/
│   ├── {prefix-lower}-full.tar.gz        # overwritten each run
│   └── manifest.json                     # {generatedAt, count, records: {cveId: {titles,vendors,...}}}
├── recent/
│   └── {prefix-lower}-new.tar.gz         # cveIds not in previous manifest
├── modified/
│   └── {prefix-lower}-modified.tar.gz    # cveIds with updatedAt in last 24h
└── cves/{PREFIX}/{year}/
    └── {cveId}.json                      # individual cvelistv5 JSON (new/modified runs only)

Each tar.gz contains entries at paths cves/{PREFIX}/{year}/{cveId}.json.

7. Data Mapping

Source Priority (for DISTINCT ON row selection):

8. Flags

S3 Persistence

• Archive: ⚠ Not yet implemented — requires record reconstruction (DB row → canonical JSON).
• Quarantine: ⚠ Not yet implemented — same reason.
• Likely reasons when implemented: enrich-error

This is an enrichment processor; it reads from CVEMetadata rather than ingesting raw feeds, so there is no original payload to archive verbatim. See S3 Persistence Contract § Processors whose unit-of-work is not a file.