EUVD Processor — Design Document

1. Overview

Purpose: Mirror the ENISA EU Vulnerability Database (EUVD) catalog into CVEMetadata/CVEAlias independently of the CVE identifiers we already track. The catalog is walked directly via /api/search, so EUVD records whose primary identifier is not a CVE (PWNO, GHSA-only entries, vendor IDs) are still ingested.

There are two binaries in this design:

Both share internal/euvd/ for HTTP, retry, and per-record write logic.

Data source: GET https://euvdservices.enisa.europa.eu/api/search (no auth).

API notes:

• Returns { items: Record[], total: int }. Catalog is ~373k records (production CVEMetadata count for source='euvd', 2026-08).
• Max size=100 per page; page is zero-indexed.
• Date filter: fromUpdatedDate=YYYY-MM-DD&toUpdatedDate=YYYY-MM-DD (and the equivalent fromDate/toDate on datePublished).
• Other supported filters (unused here): fromScore/toScore, fromEpss/toEpss, product, vendor, assigner, exploited, text.
• Item date format: "Jan 2, 2006, 3:04:05 PM" (US locale); RFC3339 accepted as fallback.
• CVSS vector field: baseScoreVector.
• No title field in response.
• Rate-limit / server-error responses (429, 5xx, 403) → exponential backoff (5 retries, base 1s, cap 60s, ±50% jitter).

Schedule: Daily at 03:00 UTC (cron cron(0 3 * * ? *) — Terraform EventBridge, unchanged from prior design).

Timeout: 60 minutes (daily processor); soft deadline derived from EXPECTED_DURATION_MINUTES - 10. The backfill has no deadline at all — per AGENTS.md “Backfill Must Not Have a Deadline”, just go-euvd-json-backfill unsets EXPECTED_DURATION_MINUTES and the binary applies no soft cap, so a full walk runs to completion.

Resources: 512 CPU units, 1024 MB memory.

What it reads:

• Read replica: BulkDataDumpTracker (cursor for delta sweep / resume marker for backfill).
• EUVD /api/search endpoint.

What it writes:

• CVEMetadata (source=euvd — EUVD records; minimal stubs for classified alias targets only when the target row is missing).
• CVEAlias (linking EUVD IDs to recognised aliases).
• BulkDataDumpTracker (source='euvd' for delta cursor; source='euvd_backfill' for backfill resume page).
• S3 archive / quarantine (see S3 Persistence below).

What it deliberately does NOT write — worth stating up front, because the low field coverage on source='euvd' rows looks like a defect and is not:

The product of this processor is therefore the alias bridge — EUVD ⇄ CVE / GHSA / PWNO identity edges no other feed publishes — plus the CVSS vector string. Judge its efficacy on CVEAlias coverage and vectorString fill rate, not on title coverage.

Environment variables:

• DATABASE_URL — required.
• DATABASE_URL_READ — optional.
• EXPECTED_DURATION_MINUTES — optional soft-deadline budget.

2. Business Logic

Daily delta sweep — euvd-json-processor

1. Window — read BulkDataDumpTracker(source='euvd').lastProcessedAt:
   • fromUpdatedDate = max(tracker.lastProcessedAt - 24h, now - 7d) (overlap absorbs clock skew; cap bounds catch-up after long outages).
   • toUpdatedDate = now (UTC date).
   • First-ever run (no tracker row) → now - 7d.
2. Walk — GET /api/search?size=100&page=N&fromUpdatedDate=…&toUpdatedDate=… starting at page=0. Capture total from page 0 to bound the loop. Stop when len(items) < 100 OR (page+1)*100 >= total. 200 ms courtesy delay between pages.
3. Per record — see “Per-record write” below.
4. Tracker — on clean completion, UpsertTracker("euvd", "", recordsProcessed) stamps lastProcessedAt = now.UnixMilli().
5. Circuit breaker — abort after 5 consecutive non-retriable /api/search errors. Tracker is not stamped on abort, so the next run re-covers the same window.

No CVE-driven query of CVEMetadata; no enisaid?id= per-record fetches; no RESERVED placeholders are written. Legacy RESERVED rows from the prior CVE-driven design are left in place — they are simply not produced any more.

Backfill — euvd-json-backfill (local-only)

1. Resume — read BulkDataDumpTracker(source='euvd_backfill').totalCVEs (interpreted as the next page to walk). --start-page overrides; --max-pages caps the walk length for testing.
2. Walk — GET /api/search?size=100&page=N (no date filter). On each successful page, UpsertTracker("euvd_backfill", "", page+1) to persist progress.
3. No soft deadline — the backfill applies no runtime cap (AGENTS.md “Backfill Must Not Have a Deadline”), and the justfile recipe unsets EXPECTED_DURATION_MINUTES. Progress is durable per page, so interrupting the run and re-invoking resumes from the last completed page.
4. End of catalog — when (page+1)*100 >= total or page returns empty, reset totalCVEs = 0 and stamp lastProcessedAt = now. The user re-runs manually when a fresh full re-walk is desired.

Per-record write (shared)

For each EUVD Record:

1. db.UpsertCVEMetadata(CVEMetadataRow{ cveID: rec.id, source: "euvd", state: "PUBLISHED", dataVersion: "1.0", datePublished, dateUpdated, vectorString }). Title is omitted: EUVD has no title field, and only this binary writes to (rec.id, "euvd") so there is nothing to preserve.
2. Parse rec.aliases (newline/whitespace/comma/semicolon separated). For each entry, classify by prefix:
   • CVE-… → cve.org
   • GHSA-… → ghsa
   • PWNO-… → pwno
   • EUVD-… → euvd
   • anything else → skip (debug-logged). We don’t stub-create rows for identifier schemes we don’t ingest.
3. For each classified alias (aliasID, aliasSource):
   • db.EnsureMinimalCVEMetadata(aliasID, aliasSource) to satisfy the FK target.
   • Compute canonical direction via db.CanonicalAliasDirection(rec.id, "euvd", aliasID, aliasSource).
   • INSERT INTO "CVEAlias" (...) VALUES (...) ON CONFLICT DO NOTHING with discoveredFrom='euvd'.

Bundle suppression is applied locally before step 3: an EUVD record carrying more than one CVE--prefixed alias is co-listing CVEs, not asserting alternative identifiers for one vulnerability, so all its CVE aliases are dropped. Non-CVE aliases (GHSA, PWNO, EUVD) are kept.

Known deviation — this is the one place in the codebase that writes CVEAlias by hand. AGENTS.md (“Alias Writes — CVEAlias is the source of truth”) requires every alias write to go through db.InsertAliases. internal/euvd/writer.go issues the INSERT itself. It does canonicalise direction and it does apply bundle suppression, but it skips the helper’s same-cveId cross-source peer backfill — the edges that link (cveId, 'euvd') to every other source already carrying that same id. EUVD records therefore get their CVE/GHSA/PWNO bridges but not their peer-source edges. Fixing this means calling db.InsertAliases(ctx, tx, rec.ID, "euvd", aliases, logger) instead.

All writes happen inside a db.WithTx transaction with a per-record savepoint, so a single bad item rolls back only its own writes.

Idempotency

UpsertCVEMetadata increments fetchCount on conflict; alias inserts are ON CONFLICT DO NOTHING. Re-running either binary is safe.

3. Architecture Diagram

4. Deployment Diagram

5. Processing Flow — Delta sweep

The backfill flow is identical except that there is no date filter, the loop start page comes from tracker.totalCVEs, the tracker is updated per page (resume cursor) rather than once at the end, and the soft-deadline exit short-circuits the loop without resetting state.

6. Data Mapping

Legacy RESERVED rows from the prior CVE-driven design (source='euvd' AND state='RESERVED') are not produced by this design. They remain in the table; no DELETE migration is part of this change.

S3 Persistence

• Archive path: euvd/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/euvd-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.