wiz-json-processor

Status: Live, but making almost no progress — see Efficiency Source: Wiz.io Vulnerability Database (public Next.js data route, no auth) Type: json (buildId scrape + per-CVE Next.js data API, with an HTML __NEXT_DATA__ fallback) Source slug: wiz Schedule: Runs every 6 hours (cron(0 */6 * * ? *)), 256 CPU / 1024 MB, expected_duration_minutes = 60.

1. Overview

Purpose: Systematically query the Wiz.io public vulnerability database for CVEs/GHSAs in our database, creating enrichment records with Wiz’s structured vulnerability data (descriptions, CVSS scores, affected software, source feeds).

Data source: Wiz.io Vulnerability Database — https://www.wiz.io/vulnerability-database

API notes:

• Two-step API: first extract buildId from HTML page, then fetch per-CVE JSON via Next.js data route
• BuildId URL: https://www.wiz.io/vulnerability-database (HTML, regex "buildId":"<value>")
• Data URL: https://www.wiz.io/_next/data/{buildId}/en-us/vulnerability-database/cve/{lowercased-id}.json
• Not-found response: 404 Not Found — gracefully skipped, no RESERVED placeholder
• Rate limit response: 429 Too Many Requests — retried with exponential backoff
• Server error response: 5xx — retried with same backoff
• No authentication required — fully public

Schedule: Runs every 6 hours (cron(0 */6 * * ? *) — Terraform EventBridge)

Timeout: 60 minutes

Resources: 256 CPU units, 1024 MB memory

What it reads:

• Read replica: CVEMetadata (to find CVE/GHSA IDs needing Wiz lookup)
• Wiz.io vulnerability database (public, no auth)

What it writes:

• CVEMetadata (source=wiz — new rows or updates)
• CVEAlias — always, via db.InsertAliases(…, nil, …), so the same-cveId cross-source edges are written on every store even when Wiz supplies no aliases
• CVEDescription (English description from Wiz)
• CVEMetric (CVSS scores from top-level baseScore/severity)
• CVEAffected (affected software × source feed — see the caveat under Data Mapping)
• CVEMetadataReferences (advisory URL)
• With --emit-crit: CRIT candidate envelopes staged to crit-candidates/pending/ plus an S3QueueObject row per staged key, and inference offers for records the ranker could not attribute

Environment variables:

• DATABASE_URL — required
• DATABASE_URL_READ — optional (falls back to write)

2. Business Logic

Single-Phase Strategy Per Run

Each run performs exactly one phase processing up to 1000 new CVE/GHSA IDs. With a 600-900ms random delay between calls and up to 1000 total calls, a run takes approximately 12-15 minutes.

Phase A — New CVEs/GHSAs:

• Find up to 1000 CVE/GHSA IDs from CVEMetadata where cveId matches any allowedPrefixes (default: CVE-, GHSA-) AND no row exists with source='wiz' for that cveId
• Ordered by datePublished DESC (newest first — most likely to have Wiz entries)
• For each: extract buildId (once at startup), then fetch GET /_next/data/{buildId}/.../cve/{id}.json
  • 200 response: upsert CVEMetadata(source='wiz') + aliases + descriptions + metrics + affected + references, then archive the raw body to S3
  • 404 response: counted as notFound; no record of any kind is written
  • persistent 5xx: fall back to scraping the CVE’s HTML page and reading the embedded __NEXT_DATA__ JSON (htmlFallbacks in the run stats)
• Circuit breaker: abort after 10 consecutive network errors (circuitBreakerN)
• Random delay: 600-900ms between requests to avoid overloading the API

No RESERVED State — and why that stalls the processor

This processor does NOT create RESERVED placeholders for ids not found in Wiz, so a 404 leaves no trace and the same id is a candidate again on the next run. Combined with ORDER BY datePublished DESC, that means every run re-tries the same newest ids, and Wiz’s curated database does not carry most brand-new CVEs.

The measured result of the 2026-08-06 06:00 run (/ecs/vdb-scheduler/wiz-json-processor, stream task/wiz-json-processor/820a33d21b844b528fca498e111e787a): candidates found count=1000, then after 23 minutes processed=1000 found=1 notFound=999 errors=0. One new record for 1,000 requests, and the next run will request the same 999 misses again.

Two things drive it, both in runPhaseA:

1. No negative cache. Nothing records “Wiz does not have this id”, so misses are never retired.
2. Candidate duplication. The query selects DISTINCT c."cveId", c."source", c."datePublished", so a CVE present under ten sources yields ten candidate rows and is fetched ten times within one run — the 1,000-row limit therefore covers far fewer than 1,000 distinct ids.

BuildId Lifecycle

The buildId is extracted once at startup from the Wiz vulnerability database HTML page. It changes whenever Wiz deploys their Next.js application, and a stale buildId makes every data-route request 404. After 3 consecutive 404s (buildIDRefreshN) the processor re-fetches the buildId once per run; if it changed, the current id is retried with the new value. The circuit breaker (10 consecutive network errors) is a separate mechanism and does not fire on 404s.

Rate Limiting & Retry

600-900ms random sleep between each API call (randomDelay()) as a courtesy delay to avoid overloading the API.

If the API returns HTTP 429 (Too Many Requests) or HTTP 5xx (server error), the processor retries with exponential backoff (up to maxRetries=5 attempts):

• Backoff starts at 1s and doubles each attempt: 1s, 2s, 4s, 8s, 16s, capped at 60s
• Random jitter added (0 to half of current backoff) to prevent thundering herd

3. Data Mapping

CVEMetadata

CVEDescription

CVEMetric

CVEAffected

Caveat — the affected fan-out is a cartesian product and drops platforms. The loop is for each affectedSoftware × for each sourceFeed × for each affectedPlatform, but affectedHash is computed from (feedName, software, "", software) with no platform component, so all platforms of a feed collapse onto one CVEAffected row and platforms ends up holding whichever platform was iterated last. The product is also the raw affectedSoftware string for every feed, whether or not that feed actually covers it. In production this has produced 1,102,425 CVEAffected rows for 16,863 wiz CVEs — about 65 rows per CVE.

CVEMetadataReferences

4. Flags

5. ECS Schedule

Runs every 6 hours (cron(0 */6 * * ? *)), 256 CPU units, 1024 MB memory, expected_duration_minutes = 60.

The soft deadline defaults to 50 minutes even when EXPECTED_DURATION_MINUTES is unset, so a local backfill is bounded — which contradicts the AGENTS.md “backfill must not have a deadline” rule.

6. Key Files

S3 Persistence

• Archive path: wiz/files/{sha256}/{cveId}.json ✓ — the raw Wiz response body, uploaded after a successful store
• Quarantine path: failed-feeds/wiz-json-processor/{YYYY-MM-DD}/{reason}/{cveId}.json ✓
• Failure reasons emitted: parse-error (envelope / __NEXT_DATA__ decode failure), fetch-error (any other fetch failure where bytes were received), store-error (DB upsert failure)

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev). Note the unit of work here is a fetched payload — Wiz’s own JSON — so unlike a pure DB-to-DB enricher this processor has something verbatim to archive, and does.

See the S3 Persistence Contract for the full reason taxonomy.