summary-processor
Status: Live Source: internal — reads the VDB’s own tables, fetches nothing Type: aggregate (no external upstream) Source slug: none — writes no
CVEMetadatarows Schedule: Daily at 10:00 (cron(0 10 * * ? *)).
Overview
Every statistic the public API and website show about the database as a whole is
precomputed here. Nothing else in the system is allowed to aggregate the corpus
on demand: the counts span tens of millions of rows across CVEMetadata,
CVEMetadataReferences, CVEAffected, Exploit, EpssScore, PackageVersion
and the malware tables, and running them per request would make the API
unusable.
The contract is deliberately narrow. This processor performs reads and one
kind of write: one SummaryLog row per statistic per run, keyed
(label, timestamp). It never touches advisory data, so it cannot corrupt
anything it measures. Consumers read the newest row for a label and render it.
Because each run appends rather than overwrites, SummaryLog doubles as the
time series behind every trend chart — the 12-month package histogram and the
24-month malware IOC graph are just successive rows of the same label.
Records produced
| Condition | Records |
|---|---|
| Every run, per statistic | one SummaryLog row: label, value (BigInt), expr (the expression used, for auditability), timestamp (ms), and metadata (JSONB) for structured payloads |
| Vendor archive build | vendor_archive.<vendor> index rows plus vendor_archive.<vendor>.<YYYY-MM> per-period rows — around 2,300 labels in total |
Scalar statistics carry their number in value; composite payloads (trends,
histograms, campaign rollups, top-N lists) carry JSON in metadata and use
value as a count.
Statistics produced
Grouped by what they answer. All of these are read by vdb-api.
| Group | Labels |
|---|---|
| Corpus size and identity | cve_meta.total_rows, cve_meta.distinct_cves, id_stats.ghsa_total, id_stats.ghsa_no_cve, id_stats.non_cve_total |
| Scoring coverage | cvss.total, cvss.v2, cvss.v3, cvss.v4, severity, epss_count, epss_stats.avg_epss, epss_stats.high_epss |
| Weakness and classification | cwe_count, top_cwes, capec_count, ssvc_count, cess_count |
| Exploitation | kev_total, cves_with_exploits, exploit.total, exploit.malicious_packages, exploit_trends, snort_total, snort_enriched_ids |
| Enrichment coverage | enriched_cpe, enriched_purl, vulnetix_enriched, references.total, refs_cve_count, patchable.all_sources, patchable.cve_only, patchable.enriched |
| Packages | package_totals, package_recent, package_ecosystem_counts, package_publish_histogram_12m |
| Malware | malware_campaigns, malware_trends, malware_ioc_ecosystem_counts, malware_ioc_monthly_24m |
| Vendors | top_vendors, vendor_trends, vendor_archives |
| AI research | ai_discovered_vulns_outcomes, ai_assisted_exploit_creation_outcomes |
Read timeouts — READ_STATEMENT_TIMEOUT is not optional here
internal/db/pool.go puts a 60-second statement_timeout on every
read-replica connection. Several statistics here aggregate tens of millions of
rows and cannot finish inside that, so the processor must run with
READ_STATEMENT_TIMEOUT raised (30 minutes is the value used in practice). A
full -force run needs a per-statistic budget closer to 10 minutes.
The failure mode is quiet: an aggregate that exceeds the cap raises
SQLSTATE 57014 and that one statistic is skipped, leaving the previous day’s
row as the newest. The API then serves a stale number with no visible error. If
a label’s newest timestamp falls behind the others, suspect this first.
Failure modes
| Symptom | Cause |
|---|---|
| One label’s newest row is older than the rest | that aggregate hit the read statement_timeout (see above) |
| A label stops advancing entirely | its producer stopped, not this processor — e.g. enriched_crit and vulnetix_kev_run are stale because CRIT staging and the Vulnetix KEV run are not emitting, not because the summary run failed |
| Trend chart flat-lines | the label’s history is intact but the underlying source stopped writing; check that source’s freshness before this processor |
Because the run is sequential and the labels are independent, a single failed statistic never aborts the rest — which is why partial staleness is the characteristic symptom rather than an outright failed run.