Shadowserver Dashboard Processor — Design Document

1. Overview

Purpose: Scrape the public Shadowserver Foundation honeypot exploitation dashboard for per-CVE attack intelligence. Shadowserver operates a global honeypot sensor network that tags inbound attacks with the CVE / EDB / CNVD / MSB / ZDI identifier being exploited; the dashboard exposes daily “Top Exploited” rankings, statistical anomaly detections, per-tag time-series, vendor/country aggregates, and a per-CVE popup with CVSS and CISA KEV overlay.

Data source: https://dashboard.shadowserver.org/statistics/honeypot/vulnerability/ — the partner API (transform.shadowserver.org/api2/) is credential-gated and NOT used; every public panel is obtained via its own URL suffixed with ?embed=1, which returns {"content": "<html-fragment>"}.

Schedule: Every 6 hours (cron cron(5 */6 * * ? *))

Timeout: 30 minutes (expected_duration_minutes = 30)

Resources: 256 CPU units, 512 MB memory

What it reads:

• Shadowserver dashboard panels (unauthenticated HTTP GET)

What it writes:

• ShadowserverFetchLog — one row per panel hit (always written, success or fail)
• ShadowserverExploitedVulnerability — per-tag “Top” ranking snapshots, upserted by (observationDate, tagId, statistic, hostType) and enriched with popup detail (CVSS, severity, type, class, flattened CISA KEV section)
• ShadowserverAnomaly — percentage-spike detections per (windowDays, statistic, hostType)
• ShadowserverTimeSeries — per-tag daily counts, idempotent on (tagId, observationDate, statistic, hostType)
• ShadowserverGeoDistribution — per-country aggregates
• ShadowserverVendorRanking — per-vendor aggregates
• BulkDataDumpTracker row for source = "shadowserver_dashboard" with frequency = 21600 (6h)

Environment variables:

• DATABASE_URL — required
• DATABASE_URL_READ — optional
• SHADOWSERVER_MODE — incremental (default, 6h cadence) / resume / backfill
• SHADOWSERVER_FORCE — 1 to bypass the freshness tracker check
• EXPECTED_DURATION_MINUTES — optional (default 30; soft deadline is value-5)
• SNS_TOPIC_ARN — forwarded by notify.New; errors flush to Slack via SNS

No API keys.

Run modes

The incremental mode is what runs on EventBridge — it only fetches “today’s” Top panel (date_range=30 is the smallest value the dashboard accepts for that view) and a short 7-day time-series tail, because the underlying honeypot data is append-only day-by-day. The tracker check (BulkDataDumpTracker.lastProcessedAt) short-circuits if < 5h50m since the last run, guaranteeing we don’t re-scrape within a period.

resume mode is intended for operational catch-up after an outage: reads MAX(observationDate) from the DB and skips any rows at-or-before that anchor client-side, while still hitting wider windows so any late-arriving backdated data is captured. Idempotent against the unique indexes. Resume and backfill are manual-only — they bypass the freshness tracker, so must never be wired to EventBridge; invoke via the just go-shadowserver-dashboard-{resume,backfill} recipes which also take TARGET=prod|local.

backfill mode walks the full 365d sweep — used once when seeding a fresh DB. Soft deadline is auto-extended to 4 hours.

2. Business Logic

Endpoint catalogue

The processor hits six distinct endpoint shapes. All accept embed=1 except the popup, which returns raw HTML.

Parameter sweep

The processor walks multiple windows × statistics × host types so we capture short-term + long-term behaviour in one run:

• Monitoring (Top): date_range ∈ {30, 90, 365} × statistic ∈ {unique_ips, connections}
• Anomaly: date_range ∈ {7, 30} × statistic ∈ {unique_ips, connections}
• Time-series: date_range ∈ {30, 90} × statistic × host_type ∈ {src, dst}
• Visualisation (vendor/geo): date_range ∈ {30, 90, 365} × statistic (host_type=src, count_as=avg)

group_by=vulnerability is forced on the time-series endpoint — without it the dashboard falls back to continent-level grouping and returns “Asia”/“Europe” series instead of CVE IDs.

Tag normalisation

Shadowserver observations carry a tag that may be a CVE, an ExploitDB id (EDB-12345), a CNVD id (CNVD-2024-XXXXX), an MSBulletin reference (MSB-...), a ZDI advisory, or free-text. shadowserver.NormaliseTag classifies these into cve / edb / cnvd / msb / zdi / other and returns an upper-cased canonical id. Only the cve kind populates the cveId column; all rows retain the raw tagId so we don’t lose non-CVE signal.

Idempotency

Every insert uses ON CONFLICT (<natural key>) DO UPDATE keyed on the unique index declared in the migration. Re-runs within the same day refresh counts without duplicating rows. Time-series uses (tagId, observationDate, statistic, hostType) so overlapping windows (30d/90d series contain the same recent days) collapse to a single row.

Popup enrichment (capped)

After the Top panels are ingested we enqueue each distinct tag’s popup URL (collected from the details action in the row) and fetch up to popupMaxPerRun = 75 popups per run — enough to cover the top ~75 CVEs without hammering the dashboard. Popup detail updates the matching ShadowserverExploitedVulnerability rows (CVSS score/severity, device class, type, total connections, total unique IPs, and the flattened CISA KEV section when present).

Soft deadline

The main loop checks time.Now().After(softDeadline) between each panel fetch and bails gracefully when the 25-minute soft deadline is hit (task timeout is 30 minutes — a 5-minute buffer lets the notifier flush).

No FK to CVEMetadata

CVE linkage is advisory. A CVE observed in the wild may not yet exist in CVEMetadata (new n-day, or a non-standard identifier). cveId is indexed but not an FK, matching the pattern in EpssScore / CessScore. Downstream consumers should LEFT JOIN when blending with the CVE graph.

3. Architecture Diagram

4. Deployment Diagram

5. Processing Flow

6. Data Mapping

Monitoring “Top” row → ShadowserverExploitedVulnerability

Popup → same row (UPDATE)

Time-series → ShadowserverTimeSeries

• data-bb JSON blob on the chart wrapper has data.columns = [["x", dates...], ["TAG", values...], ...]
• One row per (tag, date), upserted on (tagId, observationDate, statistic, hostType)

7. Joining with Vulnetix CVE graph

-- Top exploited CVEs this week that we have metadata for
SELECT s."tagId", s."count30dAvg", cm."datePublished", cm."vectorString"
FROM "ShadowserverExploitedVulnerability" s
JOIN "CVEMetadata" cm ON cm."cveId" = s."cveId"
WHERE s."tagKind" = 'cve'
  AND s."observationDate" > (EXTRACT(EPOCH FROM NOW() - INTERVAL '7 days') * 1000)::bigint
ORDER BY s."count30dAvg" DESC NULLS LAST
LIMIT 50;

Shadowserver honeypot sightings sit in the sighting tier of Vulnetix’s exploit-intelligence taxonomy, alongside CrowdSec (CrowdSecSighting) and CIRCL. The summary-processor can now include Shadowserver as a third sighting source.

8. Failure modes

Slack / SNS wiring

Uses the shared internal/notify package (SNS_TOPIC_ARN is injected by the ECS task definition via go_task_secrets → sns-ops). Events:

• task.started at boot
• task.no_work when the freshness tracker short-circuits (6h skip)
• task.errored when any RecordError was called (error body contains every accumulated per-item failure — operators see exactly which window/statistic/tag broke)
• task.completed on clean run (with runId, mode, totals map)

S3 Persistence

Not used. This processor does not currently archive payloads or quarantine failures to S3. Per the S3 Persistence Contract this is non-compliant — see the compliance matrix for the implementation roadmap.

Expected paths when implemented:

• Archive: shadowserver-dashboard/files/{sha256}/{filename}
• Quarantine: failed-feeds/shadowserver-dashboard-processor/{YYYY-MM-DD}/{reason}/{filename}
• Likely reasons: (none documented)