NIST NVD Modified Processor — Design Document

1. Overview

Purpose: Fetch CVEs last-modified since the previous run from the NIST NVD REST API 2.0 and store them directly to CVEMetadata, then AI-enrich as many as the run’s time budget allows.

Data source: NVD REST API 2.0 — https://services.nvd.nist.gov/rest/json/cves/2.0, queried with a lastModStartDate/lastModEndDate window (nvd.FetchModified).

An NVD_API_KEY (that exact name, main.go:57) would raise the rate limit from 5 requests / 30s to 50, which internal/nvd/api.go:20-21 implements as a 6.5s vs 0.7s inter-page sleep. It is not set anywhere — no terraform secret or environment entry defines it, and every ECS run logs "hasAPIKey":false.

Schedule: Every 6 hours at 03:30 UTC, cron(30 3/6 * * ? *). Staggered 3 hours behind nist-nvd-recent (00:30) so the two never run at once — they write overlapping source="nist-nvd" rows and would otherwise contend on row locks.

Timeout: 165 minutes (EXPECTED_DURATION_MINUTES).

Resources: 256 CPU units, 1024 MB memory.

What it reads:

  • NVD REST API 2.0 (lastModStartDate/lastModEndDate, paginated)
  • Read replica: BulkDataDumpTracker for nist_nvd_modified

What it writes:

  • CVEMetadata and its child tables (source="nist-nvd"), directly — there is no intermediate queue. Via processor.StoreCVESourceData that means CVEMetadata, CVEAlias, CVEDescription, CVEMetadataReferences, CVEMetric, CVEProblemType and CVEAffected.
  • BulkDataDumpTracker (nist_nvd_modified), only when the ingest completed.
  • S3 archive and quarantine objects — see the S3 Persistence section.

Environment variables:

  • DATABASE_URL — required
  • DATABASE_URL_READ — optional
  • NVD_API_KEY — optional (rate-limit relief); currently unset in ECS
  • EXPECTED_DURATION_MINUTES — the run budget. When unset the code still applies a hardcoded 3-hour soft deadline (main.go:62), so even the go-nist-nvd-modified-json-backfill recipe — which deliberately unsets the variable — is bounded.

2. Business Logic

Lookback window

The modified window is anchored to the tracker, not a fixed feed:

  • Start at tracker.lastProcessedAt − 1 hour (the overlap guards against gaps between runs).
  • Clamp the start to at most 7 days ago (maxLookback).
  • On a first run (no tracker) default to the last 8 hours.
  • End at now.

Two-phase: ingest, then enrich

Identical discipline to nist-nvd-recent:

  1. Fetch the modified window from the REST API (nvd.FetchModified, paginated, per-page retry).
  2. Phase 1 — ingest. Store every fetched record to CVEMetadata (+ children) in batches of 100, each record in its own savepoint. No AI calls.
  3. Advance the tracker only if the ingest completed. A soft-deadline truncation leaves the tracker unchanged, so the window is refetched next run instead of being skipped.
  4. Phase 2 — enrich. enricher.RunBatchUntil(ctx, targets, softDeadline) once over the records ingested this run, stopping at the soft deadline.

Recent vs Modified

  • modified (this processor): CVEs by last-modified date since the tracker.
  • recent (nist-nvd-recent): CVEs by publication date in the last 8 days.

Both write source="nist-nvd"; the staggered schedules keep them from contending. The third member, nist-nvd-year-json-processor, reads the bulk year feeds and is the only one that can reach a CVE outside these rolling windows.

This processor is the one that keeps old CVEs current: an NVD revision to a 2019 CVE (CVSS re-score, CWE assignment, new reference, CPE change) appears only in the last-modified feed, never in the publication feed. That is why the maxLookback clamp matters — a gap longer than 7 days is not recoverable from this feed and needs a year backfill.

No per-record change detection

internal/nvd/mapper.go never populates CVESourceData.SourceFileHash, so CVEMetadata.sourceFileHash is NULL for all 372,743 nist-nvd rows and there is nothing for db.LoadProcessedHashes to resume from. Every record the API returns is re-stored in full. For this processor the impact is bounded — the upstream only lists genuinely-changed CVEs — but combined with nist-nvd-recent’s fixed 8-day re-ingest it pushes freshly published CVEs to an average fetchCount of 19.8 (production, 2026-08-06).

3. Architecture Diagram

graph TD subgraph "cmd/nist-nvd-modified-json-processor/" MAIN[main.go — two-phase ingest/enrich] end subgraph "internal/nvd/" API[api.go — FetchModified, paginated REST 2.0] MAP[mapper.go — MapNVDRecord] end subgraph "internal/processor/" STORE[StoreCVESourceData] end subgraph "internal/aienrich/" ENRICH[RunBatchUntil — budget-bounded] end subgraph "internal/db/" POOL[pool.go] TRACK[tracker.go — GetTracker, UpsertTracker] end MAIN --> API --> MAP MAIN --> STORE MAIN --> ENRICH MAIN --> TRACK MAIN --> POOL

4. Processing Flow

flowchart TD START([Start]) --> ENV{DATABASE_URL set?} ENV -->|no| FAIL([Exit 1]) ENV -->|yes| CONNECT[Connect to DB pool] CONNECT --> TRK[GetTracker nist_nvd_modified
window = lastProcessedAt-1h, clamped 7d] TRK --> FETCH[FetchModified
REST 2.0, paginated] FETCH -->|error| FAIL FETCH -->|ok| INGEST[Phase 1: store every record
batches of 100, per-record savepoint] INGEST --> DL{soft deadline hit mid-ingest?} DL -->|yes| SKIPTRK[leave tracker unchanged
window refetched next run] DL -->|no| UPTRK[UpsertTracker nist_nvd_modified] SKIPTRK --> ENRICH UPTRK --> ENRICH[Phase 2: RunBatchUntil
enrich until soft deadline] ENRICH --> DONE([task.completed])

5. Data Mapping

erDiagram BulkDataDumpTracker { string source PK "nist_nvd_modified" bigint lastProcessedAt "advanced only on complete ingest" int frequency "21600 seconds (6h)" int totalCVEs } CVEMetadata { string cveId PK string source PK "nist-nvd" int datePublished int dateUpdated json rawDataJSON } CVEMetadata ||--o{ CVEMetadataReferences : has CVEMetadata ||--o{ CVEMetric : has CVEMetadata ||--o{ CVEProblemType : has CVEMetadata ||--o{ CVEAffected : has CVEMetadata ||--o{ CVEDescription : has CVEMetadata ||--o{ CVEAlias : aliases

6. AI Enrichment (aienrich)

Enrichment runs in Phase 2, after the whole window is ingested — not inside the store loop. The processor collects an aienrich.Target per stored record and calls enricher.RunBatchUntil(ctx, targets, softDeadline) once, which stops at the soft deadline and the process-wide inference budget. Records not reached are un-enriched until a later run. See the aienrich overview.

Disable globally by omitting PIX_INFERENCE_ENABLED from this task’s environment block in terraform/go-schedules.tf.

Enrichment, not ingest, is what this task spends its money on. Observed run task/nist-nvd-modified-json-processor/54f2030c381547a4a57b0835358d8956 (2026-08-06 03:30 UTC): the API returned 103 records, Phase 1 stored all 103 in under a second, and the run then took 100.7 minutes to enrich those 103 (duration_seconds: 6041.7, enriched: 103, enrichDeferred: 0) — about 58 seconds per CVE, much of it context deadline exceeded against the Cloudflare AI Gateway. The preceding run (21:30 UTC) ran 150 minutes and its log ends on an enrichment warning with no complete line and no Outcome metric at all, i.e. it did not exit cleanly. Sizing and alarm thresholds derived from wall-clock duration here are measuring the gateway, not NVD.

S3 Persistence

Compliant. The generator previously printed “Not used” here because scripts/docs/s3-status.yaml recorded status: none for this slug; that entry was wrong (cf. ORCH-09) and has been corrected.

  • Archive: nvd/files/{sha256}/{cveId}.json — uploader.Archive once per committed record, sourceSlug = "nvd" (main.go:30).
  • Quarantine: failed-feeds/nist-nvd-modified-json-processor/{YYYY-MM-DD}/{reason}/{cveId}.json.
  • Failure reasons emitted: map-error (no cveId resolved) and store-error.

Skipped entirely when S3_BUCKET_NAME is unset — uploader stays nil and both methods are nil-receiver no-ops.