Vulnetix KEV Processor — Design Document

1. Overview

Purpose: Derive the Vulnetix Known-Exploited-Vulnerabilities list — an independent, evidence-driven KEV catalogue for CVEs that have demonstrated exploitation evidence but are not yet listed by an authority catalogue: CISA KEV or ENISA EU KEV. VulnCheck KEV is not an authority for this purpose (it is a vendor catalogue that carries all of CISA plus its own additions), so it neither excludes a CVE from this list nor withholds one from view.

Scheduled since 2026-08-10. Between 2026-04-23 and 2026-08-10 this processor had no EventBridge schedule (the terraform/go-schedules.tf module was missing), so the live list was a single manual backfill of 270 rows frozen at lastEvaluatedAt = 2026-04-23. The schedule now exists and is ENABLED at cron(30 10 * * ? *); the KEV Watch trend shows daily additions from 2026-08-17 on.

A second consequence outlived the outage: the 270 April rows were admitted under the old rule (268 of them on CIRCL sightings that asserted nothing) and, because the processor never deleted, stayed listed with April due dates after the 2026-08-10 rule change. See Idempotency & de-listing for the evidence-based prune that now removes them on a complete run.

Data sources (all read-only, in-DB):

• CrowdSecSighting — CrowdSec honeypots

• Exploit — CIRCL vulnerability-lookup sightings (source='circl-sighting'), which carry MISP-taxonomy types. internal/circl.MapMISPSightingType gives each type its own category, so polarity is readable: exploited → weaponised (Tier A), confirmed → sighting_confirmed (Tier B), and the denials — not exploited → sighting_refuted, not confirmed → sighting_unconfirmed — plus the patch-status types are never read as evidence. An unrecognised future type falls through to the inert sighting, so it cannot silently start qualifying CVEs.

  This mapping previously collapsed everything except seen and exploited into one sighting value, and this processor selected exactly that value as evidence of exploitation — matching the denials while missing exploited entirely. scripts/backfill-circl-sighting-categories.sql repairs rows written under the old mapping, recovering the type from Exploit.title, which embeds it verbatim.

• Exploit also supplies weaponisation, keyed on source: metasploit, nuclei, nuclei-template, nmap-nse, plus gpz for the Project Zero 0-day feed.

• MsrcAdvisory — Microsoft’s isExploited flag

• CVEAiInWildExploitation, CVEAiAssistedExploit — curated in-the-wild and demonstrated-exploit records

• YaraRuleCVE — CVE-tagged YARA rules

• ShadowserverExploitedVulnerability + ShadowserverAnomaly — Shadowserver Foundation honeypots + anomaly surges

• SnortRuleCVE ⋈ SnortRule (enabled rules only) + CVEMetadataReferences (nucleiPath, exploitDbVerified, metasploit URLs) — weaponisation

• CVEMetric — CVSS scores

• CVEAffected + CVEAffectedVersion — fix-availability detection (mirrors vdb-api/internal/handler/v2_registry_fixes.go logic)

• CVEMetadata + CVEDescription — vendor / product / description fallbacks

• Kev (source CISA / enisa) — authority KEV exclusion list. VulnCheckKEVCVE was part of this until 2026-09-07 and is not any more

Schedule: daily at 10:30 UTC (cron_match = "30 10 * * *" in task-manager.toml, terraform/go-schedules.tf module.vulnetix_kev_processor).

Timeout: 45 minutes (expected_duration_minutes = 45; a measured production run spent 883 of 888 seconds in the candidate query, so 30 left no margin on a busy read replica)

Resources: 512 CPU units, 1024 MB memory (reads a handful of mid-size tables + one large one via indexed predicates)

Source of truth: this page. scripts/task-manager.toml points design_doc here. The // Design: scripts/go-processors/vulnetix-kev-processor.design.md comment at the top of main.go names a file that does not exist, as do the analytics/notebooks/11_exploits_sightings.ipynb and analytics/sql/24_vulnetix_kev_candidates.sql references that previously appeared in this document — there is no analytics/ directory in the repository. main.go is the only executable specification of the qualifying logic.

What it writes:

• Kev rows with source = 'vulnetix' — the official KEV row shape so consumers (API, CLI, website) hit one table.
• VulnetixKev rows — analysis metadata (qualifying reasons[] enum + lastEvaluatedAt). FK’d to Kev(cveID, source='vulnetix') with ON DELETE CASCADE.
• BulkDataDumpTracker row for source='vulnetix_kev' with frequency=86400 (24h). latestSha256 is repurposed as a resume checkpoint (last-processed CVE ID).
• SummaryLog row per run — label='vulnetix_kev_run', value=qualified, metadata as JSONB. Upserts on (label, timestamp) so same-timestamp re-runs overwrite.

Startup guard: zombieguard.CheckFromEnv(…, 60*time.Minute, …) runs before anything else, so a duplicate task that outlives its window is terminated rather than competing for the write pool.

Environment variables:

• DATABASE_URL — required (write pool)
• DATABASE_URL_READ — optional (read replica for candidate scan)
• VULNETIX_KEV_MODE — incremental (default) / resume / backfill
• VULNETIX_KEV_FORCE — 1 to bypass the freshness tracker
• VULNETIX_KEV_DRY_RUN — 1 to evaluate and report without writing
• VULNETIX_KEV_PRUNE_OVERRIDE — 1 to lift the prune size guard for one deliberate backfill after a rule change (ignored in every other mode; never lifts the completeness checks). See Idempotency & de-listing.
• EXPECTED_DURATION_MINUTES — optional (task definition sets 45; soft deadline fires at value − 5, default 25 minutes when unset in a scheduled mode)
• SNS_TOPIC_ARN — forwarded by notify.New; errors and overtime flush to Slack via SNS

No API keys.

Run modes

Backfill and resume modes are manual-only — they are never scheduled to EventBridge. Invoke via the just go-vulnetix-kev-{resume,backfill} recipes (each also takes TARGET=prod|local).

On EventBridge, mode=incremental is the only value used; the environment-variable default is incremental even if the task definition omits it.

2. Business Logic

The qualifying rule

A path must assert exploitation, not exploitability. Capability to exploit is not evidence that anyone has: a Metasploit module means an attacker could, not that one did. So a weaponisation artefact never puts a CVE on this list by itself.

A CVE enters Vulnetix KEV iff it is not already in CISA KEV or ENISA EU KEV and:

qualifies = tierA
         OR (tierB_gated AND tierC)
         OR (distinct tierB sensor networks >= 2)

Tier A — direct assertion

Mass exploitation sits in Tier A despite being a sensor reading: at that scale the breadth of the observation is itself the assertion.

Tier B — sensor observation (gated)

Sustained and surge are the same sensor network. They count as one vote, or a single fleet would corroborate itself. The two CIRCL polarities are likewise one vote. multi_source_sighting is emitted only when ≥ 2 distinct networks agree.

CIRCL exploited moved from Tier A to Tier B on 2026-09-07. The sightings are community submissions to vulnerability-lookup; the submitter is parsed but not stored, and the source URL (Exploit.originalUrl) is the only provenance kept. On 2026-09-06, 1,795 of the 2,484 live entries rested on one of these sightings and nothing else. Reading it as an observation to corroborate rather than an assertion to trust is the conservative call; the circl_exploited_source_hosts section of scripts/vulnetix-kev-calibration.sql is the input for any future decision to let a trusted-source subset qualify alone again.

Tier C — weaponisation artefact (never alone)

Keyed on Exploit.source, not Exploit.category: NULL is that column’s largest bucket, and nuclei, metasploit, github-poc and packetstorm all write no category at all.

Tier D — decorative

critical_cvss (MAX(CVEMetric.baseScore, Shadowserver.vulnerabilityScore) >= 9.0) is emitted by reasons() and appears in no qualifying path. It exists so the GUI can show severity beside the evidence. Any consumer presenting reasons[] as “why this CVE is listed” overstates it — the array labels the evidence, it is not a record of which rule admitted the entry.

Thresholds

Every value is a methodology decision. scripts/vulnetix-kev-calibration.sql reports each one against the live distribution; run it before changing any of them.

The ransomware test is affirmative, not merely non-empty: LOWER(TRIM(x)) NOT IN ('no','none','unknown','no known','not known','n/a','false','0'). A substring test cannot do this — “Known” and “Unknown” differ by a prefix and both contain “no”.

Rehearsing a rule change

Set VULNETIX_KEV_DRY_RUN=1 (or pass -dry-run) to evaluate the whole candidate set and report the qualifying count and reason distribution without writing. The membership of this list is a published claim; a rule change can move it by hundreds of entries, so rehearse first and compare.

critical_cvss appears in reasons() but in no qualifying path — it never puts a CVE on the list, it only annotates one that qualified some other way (about 60% of live rows carry it). Any UI that presents reasons[] as “why this CVE is listed” overstates it; the website’s charts and chips leave it out of “evidence” counts for that reason.

What the live list rests on. On 2026-09-06, of 2,484 rows, 2,306 carried misp_sighting and 1,795 carried no other evidence signal: three quarters of the list stood on a single CIRCL vulnerability-lookup sighting of type exploited, which Tier A then admitted alone. On 2026-09-07 that sighting moved to Tier B and a backfill re-evaluated the whole candidate set: 4,718 candidates, 442 qualified, 2,037 rows pruned (the CIRCL-only entries plus the ~250 April rows the old rule had admitted), 478 seconds. Reason distribution after the change: misp_sighting 351, critical_cvss 302 (context), snort_rule 221, nuclei_template 151, nse_script 61, crowdsec_sighting 60, metasploit_module 32, demonstrated_exploit 20, shadowserver_sighting 14, yara_rule 13, shadowserver_surge 5, multi_source_sighting 3. demonstrated_exploit (CVEAiAssistedExploit.success, a frozen 26-row academic seed) stays Tier A by decision; it is a capability result and the next review should weigh that.

Later on 2026-09-07 VulnCheck KEV left the authority set (it carries all of CISA plus its own additions, so it had been disqualifying every candidate it listed). The backfill after that change re-evaluated 4,744 candidates: 1,506 qualified, 0 pruned, 376 seconds. Listing dates span 2010-04-09 to 2026-09-01. Reason distribution after: nuclei_template 1,072, critical_cvss 1,048 (context), misp_sighting 1,004, snort_rule 767, shadowserver_sighting 730, multi_source_sighting 625, shadowserver_surge 540, crowdsec_sighting 479, metasploit_module 71, nse_script 69, demonstrated_exploit 23, yara_rule 22, zero_day_itw 4, vendor_exploited 3, reported_itw 1.

Kev row derivation (source=‘vulnetix’)

vulnerabilityName — title selection

chooseTitle() (main.go) walks a ranked list and takes the first candidate that survives cleaning, because most sources hand back a bare advisory identifier rather than prose:

1. CVEMetadata.title from cve.org / mitre (CNA-assigned)
2. CVEMetadata.title from cisa_adp (CISA Vulnrichment)
3. CVEMetadata.title from nist-nvd
4. BugBountySubmission.title — longest title across reports naming the CVE
5. CVEMetadataReferences.vlTitle (VulnerabilityLab)
6. CVEMetadataReferences.title on vendor-advisory / patch / mitigation
7. CVEMetadataReferences.title on third-party-advisory
8. CVEMetadata.title from any other source (excluding github / ghsa)
9. SnortRule.msg — longest rule message
10. First sentence of the longest English CVEDescription
11. CVEMetadata.title from github / ghsa (last resort — usually just GHSA-…)
12. Synthesised "<vendor> <product> vulnerability"
13. The bare CVE ID

A candidate is rejected when it is empty or matches identifierLikeRE — a bare CVE-…, GHSA-…, ZDI-…, RHSA-…, USN-…, DSA-…, openSUSE-SU-…, BDU:…, KB… and ~20 other advisory-code formats. Accepted candidates pass through cleanTitle(), which strips a leading identifier prefix ([GHSA-xxxx-xxxx-xxxx] Real title → Real title), collapses internal whitespace, trims trailing punctuation, and caps the result at 220 characters on a word boundary with an ellipsis.

Steps 1–9 are ordered by trustworthiness of the author, not by availability. Step 11 sits below the description lede deliberately: a GHSA title is more often an identifier than prose.

requiredAction branches

Fix-availability is determined from the CVEAffected / CVEAffectedVersion join used by the vdb-api registry-fixes handler: status='unaffected' OR lessThan IS NOT NULL OR lessThanOrEqual IS NOT NULL.

Idempotency & de-listing

• Kev upsert uses ON CONFLICT ("cveID", source) DO UPDATE on every field except dateAdded / createdAt; dueDate is re-anchored onto the stored dateAdded (see the derivation table).

• VulnetixKev upsert uses ON CONFLICT ("cveID") DO UPDATE on reasons, lastEvaluatedAt, updatedAt.

• Writes are batched: qualifying candidates accumulate and flush every 500 rows through one pgx.Batch, with both statements for a CVE in the same batch so the VulnetixKev → Kev foreign key is satisfied by ordering within it.

• Evidence-based prune, complete runs only. After a run that evaluated the whole candidate set, pruneStale deletes every source='vulnetix' row (and its VulnetixKev metadata) whose CVE no longer qualifies on evidence — including CVEs that dropped out of the candidate set. The run stats and SummaryLog carry kept and pruned.

  pruneAllowed refuses when any of these hold: mode is resume, dry run, soft deadline hit, context cancelled, any batch flush failed, nothing kept, or the run would keep less than half of the stored rows. The last one is the outage guard: a candidate query that silently lost an upstream table looks exactly like mass de-listing, so it raises a RecordError (“prune refused”) and leaves the rows alone. A deliberate rule change that de-lists more than half is rolled out with VULNETIX_KEV_PRUNE_OVERRIDE=1 on a manual backfill run (just go-vulnetix-kev-backfill TARGET=prod), after a dry run has shown the expected qualified count; the override is ignored in incremental and resume so a scheduled run can never inherit it.

Authority hand-off is a soft delete at read time, not a prune. An entry that CISA or ENISA has since ratified still qualifies on evidence, so the prune keeps the row: it is the record that Vulnetix listed the CVE first, and the KEV Watch statistics, trend and source comparison keep counting it. What changes is what a customer is served: every catalogue surface withholds the Vulnetix record once an authority lists the CVE.

• vdb-api /v2/vulnetix-kev and /v2/kev exclude authority-listed CVEs in their own queries.
• vdb-site KEV Watch withholds the Vulnetix record from the catalogue rows, the detail page, the recent feed and the org-overlap rows (kev_served), while /kev/stats, /kev/trend and /kev/source-comparison keep the row for analysis.

The unconditional end-of-run DELETE that preceded all of this removed every source='vulnetix' row not re-evaluated during that run, so a soft-deadline truncation at 10% left a list 90% shorter, and resume mode deleted exactly what the previous run had just written. Its removal then produced the opposite failure — 270 rows admitted under the old rule stayed listed for a month after the rule changed — which is what the guarded prune above resolves.

Consequence: any new consumer of Kev(source='vulnetix') must apply the authority exclusion itself, or it will serve entries that have already graduated.

Soft-deadline handling

Two independent mechanisms stop a long run, and the loop condition checks both:

• notify.Notifier.SetOvertimeCancel(cancel) cancels the context on overtime — observed as ctx.Err() != nil.
• An in-process softDeadline computed at startup: EXPECTED_DURATION_MINUTES − 5, defaulting to 25 minutes when the variable is unset. Not applied in backfill mode, which runs to completion.

On hit, the loop breaks and the run:

1. Records RecordError("soft deadline reached after N of M CVEs …") so the completion event carries a partial-run warning through to Slack.
2. Persists the last processed CVE ID to BulkDataDumpTracker.latestSha256 via db.UpsertTracker, so a subsequent resume run can pick up from there.
3. Flushes any pending upsert batch, so work already done is not lost.
4. Writes the SummaryLog row with softDeadlineHit: true.
5. Notifies completion (not error) with the truncated count — the soft-deadline warning shows up as amber in Slack, not red.

A truncated run is now merely incomplete rather than destructive: with pruning gone, the entries it did not reach are simply left as they were.

Candidate ordering

Ordering comes from ORDER BY c.cve in candidateSQL, which keeps the resume checkpoint deterministic in one Postgres sort pass.

This replaced a hand-written insertion sort in Go, justified by a comment claiming the data was “already near-sorted from the query” — but the query had no ORDER BY and ended in a multi-way UNION, so row order was whatever the planner emitted. That made it O(n²) over a ~30-field struct on adversarial ordering.

Error handling

• Per-CVE errors (requiredAction computation, upsert failure) are collected via notify.RecordError and logged at Warn; the run continues.
• Database-level errors (connection loss, fatal constraint violation on the tracker) call notify.Errored and exit non-zero so EventBridge retries.
• All messages flow to Slack via SNS_TOPIC_ARN when set (EventBridge/ECS runtime); no-op locally.

3. Operations

Local invocation

just go-vulnetix-kev                    # incremental, against local DB via ../saas/.env
just go-vulnetix-kev-resume             # resume from last checkpoint
just go-vulnetix-kev-backfill           # full re-evaluation, extended deadline
just go-vulnetix-kev TARGET=prod        # incremental against production (use sparingly)
just go-vulnetix-kev-backfill TARGET=prod

EventBridge

Schedule name: go-vulnetix-kev-processor. Cron: cron(30 10 * * ? *). No args.

Observability

• CloudWatch log group /ecs/vdb-scheduler/vulnetix-kev-processor — structured JSON.
• SummaryLog row appended each run with label='vulnetix_kev_run', value=qualified (not processed), and metadata containing {mode, processed, qualified, skipped, reasonsDistribution, softDeadlineHit, elapsedSeconds}.
• Slack: task.started / task.completed / task.errored / task.overtime via SNS Slack notifier. A completion with RecordError’d messages surfaces amber; a hard exit surfaces red.

⚠ Staleness is invisible to the health check. scripts/processor-health-check.sql maps vulnetix-kev-processor to source 'vulnetix' — a string it shares with crowdsec-json-processor, scanner-processor, scan-watchdog and summary-processor. The check reports the latest timestamp across every table a source writes, so those actively-running processors keep 'vulnetix' looking fresh no matter how long this one has been dead. That is why a 105-day outage went unreported.

The processor’s own tracker uses source='vulnetix_kev', which the health check does not map at all. Keying the check on BulkDataDumpTracker('vulnetix_kev'), or on MAX("updatedAt") FROM "Kev" WHERE source='vulnetix', would surface it.

S3 Persistence

• Archive: ⚠ Not yet implemented — requires record reconstruction (DB row → canonical JSON).
• Quarantine: ⚠ Not yet implemented — same reason.
• Likely reasons when implemented: enrich-error

This is an enrichment processor; it reads from CVEMetadata rather than ingesting raw feeds, so there is no original payload to archive verbatim. See S3 Persistence Contract § Processors whose unit-of-work is not a file.