NIST NVD Recent Processor — Design Document

1. Overview

Purpose: Fetch CVEs published in the last ~8 days from the NIST NVD REST API 2.0 and store them directly to CVEMetadata, then AI-enrich as many as the run’s time budget allows.

Data source: NVD REST API 2.0 — https://services.nvd.nist.gov/rest/json/cves/2.0, queried with a pubStartDate/pubEndDate window of the last 8 days (nvd.FetchPublished). The window is fixed at now − 8 days (main.go:93-94) — it does not move with the tracker.

An NVD_API_KEY (that exact name, main.go:55) would raise the rate limit from 5 requests / 30s to 50, which internal/nvd/api.go:20-21 implements as a 6.5s vs 0.7s inter-page sleep. It is not set anywhere: no terraform secret or environment entry defines it, and every ECS run logs "hasAPIKey":false. At 2,000 results per page an 8-day window is 1–2 pages, so the anonymous limit is not currently the bottleneck — but the feed is being polled unauthenticated and is one page-count increase away from mattering.

Schedule: Every 6 hours at 00:30 UTC, cron(30 0/6 * * ? *). Staggered 3 hours ahead of nist-nvd-modified (03:30) so the two never run at once — they write overlapping source="nist-nvd" rows and would otherwise contend on row locks.

Timeout: 165 minutes (EXPECTED_DURATION_MINUTES, capped so the two NVD runs cannot overlap on the 6-hour cadence).

Resources: 256 CPU units, 1024 MB memory.

What it reads:

• NVD REST API 2.0 (pubStartDate/pubEndDate, paginated)
• Read replica: BulkDataDumpTracker for nist_nvd_recent

What it writes:

• CVEMetadata and its child tables (source="nist-nvd"), directly — there is no intermediate queue. Via processor.StoreCVESourceData that means CVEMetadata, CVEAlias, CVEDescription, CVEMetadataReferences, CVEMetric, CVEProblemType and CVEAffected.
• BulkDataDumpTracker (nist_nvd_recent), only when the ingest completed.
• S3 archive and quarantine objects (main.go:264, main.go:271) — see the S3 Persistence section, which the generator gets wrong.

Environment variables:

• DATABASE_URL — required
• DATABASE_URL_READ — optional
• NVD_API_KEY — optional (rate-limit relief); currently unset in ECS
• EXPECTED_DURATION_MINUTES — the run budget; drives the soft deadline and the AI-enrichment budget. When unset the code still applies a hardcoded 3-hour soft deadline (main.go:60), so even the go-nist-nvd-recent-json-backfill recipe — which deliberately unsets the variable — is bounded.

2. Business Logic

Two-phase: ingest, then enrich

The run has two distinct phases so slow AI enrichment cannot starve ingestion:

1. Fetch the publication window from the REST API (nvd.FetchPublished, paginated, per-page retry).
2. Phase 1 — ingest. Store every fetched record to CVEMetadata (+ children) in batches of 100, each record in its own savepoint. No AI calls happen here.
3. Advance the tracker only if the ingest completed. If the soft deadline was hit mid-ingest, the tracker is left unchanged so the next run refetches the same window rather than skipping the records that were not reached.
4. Phase 2 — enrich. Run enricher.RunBatchUntil(ctx, targets, softDeadline) once over the records ingested this run, stopping at the soft deadline. Records not reached keep their stored row with no enrichment and are picked up on a later run.

An earlier design interleaved enrichment into the store loop and advanced the tracker unconditionally; that spent the whole budget enriching the first few hundred CVEs and silently dropped the rest of the window. The two-phase split fixes both.

Recent vs Modified

• recent (this processor): CVEs by publication date in the last 8 days.
• modified (nist-nvd-modified): CVEs by last-modified date, anchored to its own tracker with a 1-hour overlap.

Both write source="nist-nvd", which is exactly why their schedules are staggered.

The third member of the family, nist-nvd-year-json-processor, is the only one that can reach a CVE outside these rolling windows — it reads the bulk nvdcve-2.0-YYYY.json.gz year feeds and is on-demand only.

Tracker independence

Uses the tracker key nist_nvd_recent (modified uses nist_nvd_modified), so the two operate independently.

No per-record change detection

There is none. internal/nvd/mapper.go never populates CVESourceData.SourceFileHash, so CVEMetadata.sourceFileHash is NULL for all 372,743 nist-nvd rows and db.LoadProcessedHashes has nothing to resume from. The per-record SHA256 computed at main.go:218 is used only as the S3 archive key. Because the publication window is also fixed at 8 days, every run re-stores the entire window: a freshly published CVE is re-upserted on all ~32 recent runs it remains in the window for, plus every modified run that touches it. Measured in production 2026-08-06: nist-nvd rows published in the last 8 days average fetchCount 19.8 (max 38), settling at 5.2 once they age out of the window. This is the write amplification the schedule stagger exists to keep away from the modified processor.

3. Architecture Diagram

4. Processing Flow

5. Data Mapping

sourceFileHash is deliberately absent from the diagram: this processor never sets it (see “No per-record change detection” above).

6. AI Enrichment (aienrich)

Enrichment runs in Phase 2, after the whole window is ingested — not inside the store loop. The processor collects an aienrich.Target per stored record and calls enricher.RunBatchUntil(ctx, targets, softDeadline) once. RunBatchUntil stops starting new targets at the soft deadline (and at the process-wide inference budget), so a slow gateway cannot push the run into overtime; records it does not reach are simply un-enriched until a later run. See the aienrich overview.

The shared passes run per CVE (affected → attack → cwe → treesitter); the GHSA-only PoC pass self-gates to a no-op because the target source is nist-nvd. The CWE pass is particularly valuable here — NVD often publishes a CVE without a CWE on initial release.

Phase 2 is where the run’s time actually goes. Observed run task/nist-nvd-recent-json-processor/7db6bdd8cbab4206a2d1f75dc7bf6079 (2026-08-06 00:30 UTC): the API fetch returned 2,414 records in 10 seconds, Phase 1 stored all 2,414 with 0 errors, and the remaining ~2h34m went to Phase 2, which enriched 181 and deferred 2,233 before the budget expired, with repeated context deadline exceeded warnings against the Cloudflare AI Gateway. Total 155 minutes — exactly EXPECTED_DURATION_MINUTES − 10 — and the run still reported Outcome=success. Every scheduled run hits that same ceiling. Deferred targets are not queued anywhere; they are only re-attempted if a later run happens to reach them (see .repo/PROCESSOR_STABILIZATION.md ISSUE-22).

Disable globally by omitting PIX_INFERENCE_ENABLED from this task’s environment block in terraform/go-schedules.tf.

S3 Persistence

Compliant. The generator previously printed “Not used” here because scripts/docs/s3-status.yaml recorded status: none for this slug; that entry was wrong (cf. ORCH-09) and has been corrected. The real behaviour:

• Archive: nvd/files/{sha256}/{cveId}.json — uploader.Archive(ctx, sourceSlug, o.sha, o.filename, o.raw) at main.go:264, once per committed record, with sourceSlug = "nvd" (main.go:30).
• Quarantine: failed-feeds/nist-nvd-recent-json-processor/{YYYY-MM-DD}/{reason}/{cveId}.json — main.go:271.
• Failure reasons emitted: map-error (no cveId resolved) and store-error (main.go:267-270). A record whose CVE id cannot be guessed is filed under the first 16 hex chars of its SHA256.

Skipped entirely when S3_BUCKET_NAME is unset — uploader stays nil and both methods are nil-receiver no-ops.