GCVE JSON Processor — Design Document

1. Overview

Purpose: Hourly export of all GcveIssuance records as cvelistv5-format tar.gz archives to S3, with individual raw JSON per record for new/modified entries.

Identifier scheme: GCVE identifiers use Vulnetix’s GCVE Numbering Authority (GNA) id 110 — e.g. GCVE-110-OSM-2026-1. The GcveIssuance sequence pattern is GCVE-110-{PREFIX}-{year}-%. (Earlier GCVE-VVD-… ids and the cves/GCVE-VVD/ S3 path are retired in favour of the numeric GNA form.)

Schedule: Runs every 6 hours (cron(0 */6 * * ? *))

Timeout: 120 minutes (raised from 30 after measured 46/52/91-min runs; the export is resumable via its on-disk record cache)

Resources: 1024 CPU units, 4096 MB memory (terraform/go-schedules.tf is authoritative for CPU/memory; scripts/task-manager.toml still declares the pre-raise 512/1024 and is stale — the dashboard reads the toml, so its sizing column under-reports this task). Peak heap is ~100 MB; ephemeral local disk holds one .tar.gz per output, typically tens of MB because entries are gzipped as they are written.

ECS command: ["/app/gcve-json-processor", "--resume=true", "--max-age-hours=72"] — neither flag is declared in the task-manager.toml args block, which lists only force and limit.

What it reads:

• Read replica: GcveIssuance table (all rows, ordered by gcveId)
• Read replica: Kev, VulnCheckKEVCVE, CVEAlias tables (read-through cache; aliases are direct/one-hop only)
• S3: gcve-archive/latest/manifest.json (previous run’s ID set for delta detection)

What it writes:

• S3: gcve-archive/YYYY/MM/DD/HH/gcve-full.tar.gz — dated immutable snapshot
• S3: gcve-archive/latest/gcve-full.tar.gz — overwritten each run
• S3: gcve-archive/recent/gcve-new.tar.gz — records not in the previous manifest
• S3: gcve-archive/modified/gcve-modified.tar.gz — records updated in the last hour
• S3: gcve-archive/cves/GCVE-110/{year}/{gcveId}.json — individual raw JSON (new/modified only)
• S3: gcve-archive/latest/manifest.json — updated ID+metadata index

Environment variables:

• DATABASE_URL — required (write-side, but used as read fallback if DATABASE_URL_READ absent)
• DATABASE_URL_READ — optional read replica
• S3_BUCKET_NAME — required
• SNS_TOPIC_ARN — optional (enables overtime/error SNS notifications)
• EXPECTED_DURATION_MINUTES — optional (default none; triggers overtime notification if exceeded)

2. Business Logic

Delta Detection (Manifest-Based)

On each run the processor compares the current GcveIssuance row set against the previous manifest:

1. Load gcve-archive/latest/manifest.json from S3 (previous run’s set of gcveIds + metadata).
2. A record is new if its gcveId does not appear in the previous manifest.
3. A record is modified if its updatedAt ≥ nowMs − 3_600_000 (updated within the last hour).
4. If --force is not set and len(allRows) == len(previousIDs) with no new or modified records, exit early without rebuilding archives.

Source Priority

Each GcveIssuance row carries a source column. BuildRecord is called with that source, so the archive entry reflects the authoritative source for each GCVE ID.

Read-through cache

Workers share an ExportCache (export.NewExportCache) — a read-through LRU for Kev, VulnCheckKEVCVE, and each record’s direct aliases. Aliases are the one-hop CVEAlias neighbourhood via db.LoadDirectAliases; the full transitive closure/BFS was removed because the alias graph has hub CVEs that merged unrelated records. A cache miss falls through to a DB query.

Batched Record Building

Holding every JSON payload plus a full-archive bytes.Buffer in memory would OOM the container (and the parent TTY when run locally) as the GcveIssuance table grows. Records are built in bounded batches instead:

1. Rows are processed in batches of batchSize = 1000.
2. Inside each batch a parallel errgroup (default 20 workers) calls buildRecord → export.BuildRecord and marshals JSON.
3. After g.Wait() the main goroutine serially iterates batch results, writing each entry to the on-disk chunked archives (full / recent / modified) and enqueueing the raw-JSON upload.
4. The batch’s results slice is released and runtime.GC() runs at the end of every batch so heap usage drops back to baseline before the next batch starts.

Peak batch memory is approximately batchSize × avg(JSON size) ≈ 1000 × 20 KB ≈ 20 MB regardless of how many records exist.

Streaming On-Disk Archives

Each of the three output archives (full, recent, modified) is backed by a db.StreamingArchive that writes tar entries straight through tar.Writer → gzip.Writer → os.File into a single .tar.gz at <tmp>/{name}.tar.gz. Peak memory is O(one entry); peak disk is the compressed archive size (roughly 1/10 of the uncompressed payload for JSON). No intermediate uncompressed chunks accumulate, so even a backfill of every prefix fits comfortably on a Fargate ephemeral volume.

On Close the tar + gzip + file descriptors are flushed and closed. The file is uploaded directly from disk; Remove() deletes each archive right after its uploads finish so disk usage shrinks progressively. There is no defer os.RemoveAll safety net here (unlike cveprefix-json-processor) — cleanup happens only on the success path, which is deliberate: os.Exit does not run defers, and leaving the dir behind lets the next run reuse it (see § Implicit Resume, step 4).

Raw JSON Uploads

Raw JSON uploads overlap with batch processing via a bounded channel (rawUploadQueueSize = 200) feeding a background errgroup of 10 workers. Entries for new/modified records (or all records under --force, or stale-in-S3 records under --resume) are enqueued as each batch is written. After the final batch the channel is closed and the pool drained before the archives are closed.

⚠ Two silent-loss paths, both non-fatal and both leaving the run green.

1. The channel send is non-blocking (select … default:): when all 200 slots are full the job is discarded and only uploadDropped is incremented. That record’s per-record {gcveId}.json is never written to S3, and nothing retries it — the manifest is still saved as if it had been.
2. A failed PutObject increments uploadErrors and logs a warning.

Both counters surface in the task.completed payload as rawUploaded / rawUploadDrops, but neither escalates to task.errored. Likewise a record whose buildRecord fails is counted in skipped and the run still completes unless every record failed. Watch rawUploadDrops and skipped in the completion stats; a subsequent --resume run is what actually repairs dropped uploads.

Archive Upload

The .tar.gz files written by StreamingArchive are uploaded from disk via PutObject with the *os.File as the Body. The SDK streams content and sets Content-Length from Stat().Size(), avoiding any bytes.Buffer copy. The full archive file is re-opened to upload once for the dated key and once for the latest key.

Every S3 put (s3PutFile / s3PutJSON / saveManifest) routes through retryPut, which builds a fresh context.WithTimeout rooted at context.Background() on every attempt — the caller-supplied ctx is polled only between attempts, never passed into the SDK. This makes each PutObject immune to poisoning from a cancelled or expired parent ctx. On context.DeadlineExceeded / context.Canceled the call is retried up to s3PutMaxAttempts (3) times with 500 ms × attempt-number backoff; non-ctx errors fall through unchanged. Per-attempt budgets: 10 min for archive files, 60 s for raw JSON blobs, 2 min for the manifest.

Idempotency

Re-running with the same data produces the same archives. The manifest diff ensures no-op exits when nothing has changed. --force bypasses the diff check.

S3-Aware Resume (--resume)

The local filesystem resume cache survives ordinary crashes but not events that wipe /tmp — power outages, fresh containers, or distinct hosts. To recover safely from those cases, --resume lists the existing per-record raw JSON objects in S3 (gcve-archive/cves/GCVE-110/) at startup and builds a gcveId → LastModified map. Per-row classification then becomes:

• --force → always upload.
• --resume set → upload iff S3 has no object for the row, or the S3 object’s LastModified is before the row’s updatedAt. This is independent of the new/modified delta logic, since after a power outage every row will look “old” relative to the previous manifest yet may still be missing in S3.
• Otherwise → upload iff the row is new vs the previous manifest, or updatedAt falls in the last hour.

If --resume is set, every row is current in S3, and --force is not set, the run exits before any DB-heavy BuildRecord work or archive churn. When at least one row needs uploading, archives + manifest are still rebuilt as on a normal run (every row’s JSON is needed to produce the full archive, and the resume cache amortises the build cost across attempts).

Recency Window (--max-age-hours)

The EventBridge schedule does not need to consider every record on every fire. With --max-age-hours=72, rows are filtered immediately after load to those with updatedAt within the last N hours. Combined with --resume, the daily/hourly ECS task touches only the small slice of recently-changed records and skips uploads for any that are already current in S3.

Implicit Resume (Record Cache)

The expensive part of each run is export.BuildRecord — it issues multiple DB queries per record. To survive crashes, container kills, and ECS timeouts without throwing that work away, every built record is persisted as a JSON file under a stable resume cache directory:

<tmpRoot>/gcve-json-processor-cache/records/{year}/{gcveId}.json

Flow on every run:

1. On startup the cache directory is opened (created if missing). Its entry count is logged for visibility.
2. Inside the worker, for each row the cache is consulted by path. A hit is valid only if the cached file’s mtime in milliseconds is ≥ the row’s updatedAt — any newer upstream change invalidates the cache entry and forces a rebuild.
3. On a miss the record is built, marshalled, and written atomically (*.tmp + rename) to the cache. The worker then proceeds exactly as before (archive writes, raw-JSON upload enqueue, manifest entry).
4. Archives themselves are not resumable — tar.gz has no safe append point — so they are rebuilt from the resume cache on every run. They live at a stable path <tmpRoot>/gcve-json-processor-archives/ (not MkdirTemp) so a crashed prior run never leaves an orphan dir behind: any stale archive files from a partial run are wiped at startup before the rebuild begins. Cleanup happens on successful completion only — os.Exit paths leave the dir in place to be overwritten by the next run, since defer does not fire on exit. Rebuilding the tar from a hot cache is cheap; it’s the record building that was expensive.
5. After a full successful run both the resume cache and the archive dir are wiped. On crash/kill/timeout the cache persists and the next run implicitly picks up where the previous one left off — cache hits show up in the progress log as cacheHits / cacheMisses. If a crash lands between archive close and S3 upload, the next run rebuilds archives from the all-hit cache (fast) and re-attempts the uploads.
6. Defensive check after archive close: an explicit os.Stat confirms the archive file exists and is non-empty before any S3 upload is attempted. A missing or zero-byte archive is fatal and logged with the offending path, instead of producing a confusing nested s3PutFile stat error.

--build-clean wipes the resume cache before any work starts, forcing every record to be rebuilt from the DB. Use it when upstream enrichment logic has changed in a way updatedAt cannot detect (e.g. a new field in BuildRecord output for unchanged rows).

3. Architecture Diagram

4. Deployment Diagram

5. Processing Flow

6. S3 Object Layout

gcve-archive/
├── YYYY/MM/DD/HH/
│   └── gcve-full.tar.gz          # immutable dated snapshot
├── latest/
│   ├── gcve-full.tar.gz          # overwritten each run
│   └── manifest.json             # {generatedAt, count, records: {gcveId: {titles,vendors,...}}}
├── recent/
│   └── gcve-new.tar.gz           # gcveIds not in previous manifest
├── modified/
│   └── gcve-modified.tar.gz      # gcveIds with updatedAt in last hour
└── cves/GCVE-110/{year}/
    └── {gcveId}.json             # individual cvelistv5 JSON (new/modified runs only)

Each tar.gz contains entries at paths cves/GCVE-110/{year}/{gcveId}.json.

7. Data Mapping

8. Flags

S3 Persistence

• Archive: ⚠ Not yet implemented — requires record reconstruction (DB row → canonical JSON).
• Quarantine: ⚠ Not yet implemented — same reason.
• Likely reasons when implemented: enrich-error

This is an enrichment processor; it reads from CVEMetadata rather than ingesting raw feeds, so there is no original payload to archive verbatim. See S3 Persistence Contract § Processors whose unit-of-work is not a file.