CIRCL Vulnerability Lookup Processor — Design Document

1. Overview

Purpose: Query the CIRCL Vulnerability Lookup API for CVEs in our database, storing the main CVE record plus cross-referenced advisory records (linked) and community sightings. Linked records from distribution maintainers (SUSE, CERT-FR, NVD, GHSA, FSTEC) carry authoritative affected product and fix version information.

Data source: CIRCL Vulnerability Lookup API — https://vulnerability.circl.lu/api/vulnerability/{CVE-ID}?with_sightings=true&with_linked=true

API notes:

• Response format: Standard CVE Record v5.1 + extension fields (vulnerability-lookup:linked, vulnerability-lookup:sightings)
• Not-found response: 404 Not Found
• Rate limit response: 429 Too Many Requests — exponential backoff
• Server error response: 5xx — retried with same backoff
• No authentication required (public API)

Schedule: Runs daily at 01:00 UTC (cron(0 1 * * ? *) — Terraform EventBridge)

Timeout: 300 minutes

Resources: 256 CPU units, 1024 MB memory

What it reads:

• Read replica: CVEMetadata (find CVEs needing CIRCL lookup, existing CIRCL records)
• Read replica: CVEAlias (find linked CIRCL counterparts)
• CIRCL API (public, no auth)

What it writes:

• CVEMetadata (source=circl — main CVE record + linked advisory records)
• CVEDescription, CVEMetadataReferences, CVEMetric, CVEProblemType, CVEAffected, CVEAffectedVersion (full relation tables for all records)
• CVEAlias (linking advisory IDs to CVE IDs)
• Exploit (source=circl-sighting — community sightings)
• ExploitCVE (linking sightings to CVE records)

Environment variables:

• DATABASE_URL — required
• DATABASE_URL_READ — optional

2. Business Logic

Two-Phase Strategy Per Run

Phase A — New CVEs (phaseALimit = 5000):

• Find CVE-prefix records from CVEMetadata where no CIRCL counterpart exists (circl_linked CTE AS MATERIALIZED + LEFT JOIN anti-join)
• Ordered by datePublished DESC (newest first)
• For each: GET /api/vulnerability/{CVE-ID}?with_sightings=true&with_linked=true
  • 200 response: store main CVE record + linked advisories + sightings (all in one transaction)
  • 404 / 204 / body without cveMetadata: create a state='RESERVED' placeholder row under source='circl' and call db.InsertAliases(cveID, "circl", nil) so the same-cveId cross-source edges are backfilled. The placeholder is what stops the same CVE being re-queried on every subsequent run.
• Circuit breaker: abort the phase after circuitBreakerThreshold = 10 consecutive network errors (main.go:318; the log line reports the constant, so no drift)
• Sustained HTTP 429 past the retry budget is counted as rateLimited, not an error: CIRCL is reachable, the record is simply deferred to the next run. It never trips the circuit breaker and never pages.

Phase B — Reprocess Oldest (phaseBLimit = 250):

• Find the oldest source='circl' records by lastFetchedAt ASC NULLS FIRST (rides the (source, lastFetchedAt) composite index)
• At most one alias per row is resolved via a LEFT JOIN LATERAL on CVEAlias. This replaced an earlier ROW_NUMBER() window pass over the whole ~750k-row circl source, which held LWLock:BufferMapping for 5+ minutes per run and starved the read replica.
• Re-query the API using the linked CVE ID when the row’s own id is not CVE-*
• On any fetch failure the row’s lastFetchedAt is bumped and fetchCount incremented so it rotates to the back of the queue regardless of failure class
• Circuit breaker: 10 consecutive network errors
• Skipped entirely if Phase A’s circuit breaker fired with zero API successes

Linked Record Processing

The vulnerability-lookup:linked field maps source keys to arrays of [id, record] tuples. Each record is stored as a full CVEMetadata record with all relation tables:

All linked records are stored with source='circl'.

Duplicate IDs (same advisory in different case, e.g., CERTFR-2024-AVI-0518 and certfr-2024-avi-0518) are normalised to uppercase and deduplicated.

What this actually produces at scale — and where it falls short

Because every linked document becomes its own CVEMetadata row, circl is the largest single source in production. Composition as of this audit (748,435 rows):

Two consequences worth knowing before reading any coverage metric off this source:

1. mapGenericLinked is a husk producer. It only reads top-level id / title / summary / description / published (internal/circl/parser.go:467-520). GSD, VARIoT and PYSEC linked records nest their content (gsd.osvSchema, namespaces["cve.org"], …), so 211,858 GSD rows land with no title, no description, no reference, no affected product and no CVSS — only rawDataJSON. Sampled 200 GSD rows: 0 CVEDescription, 0 CVEMetadataReferences, 0 CVEAffected. That single cluster is what drags the source’s title coverage down to ~67%; the data is present in the archived raw payload but is never extracted.
2. source='circl' is not provenance for ~403k rows. Linked docs are stored under circl, not under the publisher that wrote them, so CNVD/RHSA/SUSE/BDU/PYSEC/BIT advisories exist twice: once under their own processor’s source and again here. Anything that counts advisories per source, or treats source as “who published this”, must exclude the non-CVE-* slice of circl.

Bundle suppression on linked records

A linked document is often a rollup — a CERT-FR bulletin, a SUSE/openSUSE/CERT-Bund CSAF advisory or an FSTEC BDU entry that lists many unrelated CVEs. Co-listing on one document is bundling, not aliasing: it does not make those CVEs aliases of each other, and writing the edges anyway fans out transitive false chains through the advisory hub.

storeLinkedEntry therefore detects bundling — either the entry mapped to more than one record, or a single record carrying more than one distinct CVE-* alias — and when it fires:

• each record’s Aliases is trimmed to at most [parentCveID] (co-listed CVEs dropped);
• the explicit parent↔record db.InsertAliases call is skipped.

Non-bundled records keep the parent↔record edge. Every alias write in this processor goes through db.InsertAliases; there are no raw CVEAlias inserts.

Sighting Processing (MISP Taxonomy)

The vulnerability-lookup:sightings array contains community observations. Types follow the MISP sighting taxonomy:

Each sighting is stored as an Exploit record (exploitId=UUID, source=circl-sighting) with an ExploitCVE junction.

Rate Limiting & Retry

200 ms sleep between API calls as a courtesy delay. On 429 / 5xx / stall the fetch is retried up to maxRetries = 5 times with jittered exponential backoff (1s → 2s → 4s → 8s → 16s) capped at maxBackoffSecs = 300. When the 429 carries a Retry-After header (delta-seconds or HTTP-date) the upstream-suggested wait wins over the schedule, still capped at 300 s so a rogue header cannot pin the run for hours.

Transport is pinned to HTTP/1.1 (httpclient.NewHTTP1) because HTTP/2 stream flow-control stalls on the high-latency ap-southeast-2 → Europe path. Client.Timeout is 0; each fetch gets its own 90 s context rooted at context.Background() so one slow CVE cannot cascade-cancel unrelated requests, plus a 30 s idle watchdog (activityReader) that cancels a body transfer receiving no bytes.

Deadline

The soft deadline is EXPECTED_DURATION_MINUTES − 10 (290 min on the 300-min ECS budget). It falls back to a hardcoded 50 minutes when the env var is unset — which is the state just go-circl-json-backfill creates, since the recipe unsets it. A local backfill is therefore silently truncated at 50 minutes and will not get through a 5000- record Phase A. This contradicts the backfill-has-no-deadline rule; the deadline should be left at its zero value when the env var is absent.

DB work (processInTx, upsertReserved) deliberately uses context.Background() so a soft-deadline cancellation cannot abort a transaction mid-write.

Idempotency

All writes use ON CONFLICT DO UPDATE or ON CONFLICT DO NOTHING. Safe to re-run.

3. Architecture Diagram

4. Deployment Diagram

5. Processing Flow

6. Data Mapping

S3 Persistence

• Archive path: circl/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/circl-json-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.