nist-nvd-year-json-processor

Status: Live, but never invoked in ECS Source: NVD JSON 2.0 bulk feeds Type: json (per-year .meta + gzip archive download) Source slug: nist-nvd Schedule: none. An ECS task definition and log group exist (terraform/go-schedules.tf:706-762) but there is no aws_scheduler_schedule and no cron. scripts/task-manager.toml:316 sets aws_schedule = "go-nist-nvd-year-json-processor", which names the ECS family, not a schedule — the task has category = "go-on-demand" and a required year argument. Every run to date has been a local one.

Overview

The third member of the nist-nvd family, and the only one that can reach a CVE outside a rolling window. Its two siblings poll the REST 2.0 API: nist-nvd-recent-json-processor takes the last 8 days of publications and nist-nvd-modified-json-processor takes modifications since its tracker, clamped to at most 7 days. Anything older than those windows — the entire historical corpus, and any revision missed during an outage longer than a week — is only reachable through this processor’s bulk year archives.

That is not theoretical: the production nist-nvd corpus was built here. BulkDataDumpTracker holds a nist_nvd_year_YYYY row for every year 2002–2026, all written between 2026-04-26 and 2026-05-01, with per-year totalCVEs from 1,555 (2003) to 36,700 (2025). The ECS log group /ecs/go-nist-nvd-year-json-processor holds 0 bytes, so those runs were the just go-nist-nvd-year-backfill-all loop, not the task definition.

Each run:

Fetches nvdcve-2.0-{year}.meta and parses lastModifiedDate, sha256 and gzSize (nvd.ParseMeta).
Compares that SHA256 against BulkDataDumpTracker for nist_nvd_year_{year}. Equal → logs feed unchanged, skipping, reports no_work, exits without downloading the archive (main.go:125-130).
Downloads and streams-parses the gzip archive (nvd.ParseGzip).
Stores in batches of 100, one SAVEPOINT sp_nvd per record.
Archives each committed record to S3 and quarantines each failure.
Writes the tracker with the feed’s SHA256 and the processed count.

The download uses a forced HTTP/1.1 client (httpclient.NewHTTP1, main.go:45) because NIST’s CDN intermittently resets HTTP/2 streams mid-body on the multi-MB year feeds; the small .meta request keeps HTTP/2 (main.go:44).

Records produced

Condition	Records
Every mapped CVE in the archive	`CVEMetadata` (`source="nist-nvd"`) plus, via `processor.StoreCVESourceData`, `CVEAlias`, `CVEDescription`, `CVEMetadataReferences`, `CVEMetric`, `CVEProblemType`, `CVEAffected`
Per run	`BulkDataDumpTracker` row `nist_nvd_year_{year}` carrying the feed SHA256, `totalCVEs` and `lastProcessedAt`
When `PIX_INFERENCE_ENABLED` is set	`aienrich` passes per committed record — `CVEAffected` routine fields, `CVEAttackTechnique` + children, `CVEProblemType` (`derivedBy = "vulnetix"`), `CVETreeSitter`. Note this processor calls `RunBatch` inside* `processBatch` (`main.go:272`), unlike its two siblings which defer enrichment to a second phase

Flags

Flag	Required	Description
`--year`	yes	Calendar year of the NVD archive to fetch and process. Missing → usage message and exit 1 (`main.go:54-57`)

Local invocation

# One year
just go-nist-nvd-year-backfill 2024
just go-nist-nvd-year-backfill 2024 prod

# Every year, 2002 → current, sequentially
just go-nist-nvd-year-backfill-all prod

Failure modes

The soft deadline is not actually removable. main.go:76-80 sets a hardcoded 3-hour default and, when EXPECTED_DURATION_MINUTES is present, mins − 10. go-nist-nvd-year-backfill unsets the variable specifically to lift the deadline, but the 3-hour default reinstates it — so a large year can stop at soft deadline reached, stopping batch loop (main.go:157-159) mid-archive. The tracker is then written anyway (main.go:182, outside any truncation check) with the feed’s SHA256, so the next run sees an unchanged feed and skips it: the unreached tail of that year is silently abandoned until NIST next revises the year feed. Both siblings guard against exactly this (nist-nvd-recent-json-processor/main.go:157-162); this one does not.
go-nist-nvd-year-backfill does not disable local inference. Unlike the -all variant and every other go-* recipe, the single-year recipe omits the unset PIX_INFERENCE_ENABLED CF_AIG_TOKEN AI_GATEWAY_TOKEN AI_GATEWAY_URL line, so just go-nist-nvd-year-backfill 2024 prod drives the Cloudflare AI Gateway from a dev machine — contrary to the repo-wide “local runs are inference-free by default” rule.
No API key needed — the bulk feeds are unauthenticated, so this processor is unaffected by the missing NVD_API_KEY that throttles its two REST siblings.

S3 Persistence

Compliant. The generator previously printed “Not used” here because scripts/docs/s3-status.yaml recorded status: none for this slug; that entry was wrong (cf. ORCH-09) and has been corrected.

Archive path: nvd/files/{sha256}/{cveId}.json — uploader.Archive at main.go:261, once per committed record, with sourceSlug = "nvd" (main.go:34).
Quarantine path: failed-feeds/nist-nvd-year-json-processor/{YYYY-MM-DD}/{reason}/{cveId}.json — main.go:268.
Failure reasons emitted: map-error (no cveId resolved) and store-error (main.go:264-267). A record whose CVE id cannot be guessed is filed under the first 16 hex chars of its SHA256.

Skipped entirely when S3_BUCKET_NAME is unset — uploader stays nil and both methods are nil-receiver no-ops.