CrowdSec Processor — Design Document

1. Overview

Purpose: Fetch honeypot IP sighting data from CrowdSec for CVEs in our database, in two phases per run. Phase 1 ingests the two public (unauthenticated) CrowdSec free feeds. Phase 2 loops through CVEs in priority order against the authenticated CTI API until 50 successful responses (daily quota used productively), 3 consecutive HTTP 429s (quota exhausted), or all CVEs have been checked. Processes records matching allowedPrefixes (default: CVE-; configured via allowedPrefixes var in main.go).

Data sources:

• Public free feeds (no auth, MISP Event JSON): https://feeds.crowdsec.net/free/1e4f328d-3db6-550f-93d3-3539adc8493f.json and https://feeds.crowdsec.net/free/2be9a716-39b8-5c18-bc9e-4ba7aefd8831.json
• CrowdSec CTI API — https://cti.api.crowdsec.net/v2/smoke/search?query=cves:"<CVE-ID>"&since=30d

Schedule: Daily at 20:30 UTC (cron(30 20 * * ? *) — Terraform EventBridge)

Timeout: 60 minutes

Resources: 256 CPU units, 512 MB memory

What it reads:

• Write pool (primary): CVEMetadata, Kev, ExploitCVE, CrowdSecLog for priority selection (write pool used to avoid read-replica replication lag)
• Read pool: CVEMetadata batch lookup of (cveId → source) for feed sightings (the CrowdSecSighting composite FK requires the real source)
• CrowdSec public free feeds (no auth) and the CrowdSec CTI API (authenticated with x-api-key header)

What it writes:

• CrowdSecLog — Phase 1: one row per public-feed attribute (keyed on the attribute uuid). Phase 2: one row per CVE attempted, even on 403/429.
• CrowdSecSighting — Phase 1: one row per (feed attribute × CVE), UUID v5 so re-runs are idempotent. Phase 2: one row per IP in the response items[], on 200 only.
• S3 archive / quarantine (see § S3 Persistence)

No CVEMetadata, CVEAlias, Kev or Exploit rows are written — this processor is sighting-only and never mints or mutates advisory records.

Environment variables:

• DATABASE_URL — required
• DATABASE_URL_READ — optional
• CROWDSEC_APIKEY — CrowdSec CTI API key (required)

2. Business Logic

Phase 1 — Public free feeds (no API key)

Before any authenticated call, both public feed URLs are fetched and parsed as MISP Event JSON. Each Event.Attribute entry is an IP whose comment field carries a CVEs: [...] list and whose Tag[] entries carry reputation / confidence / first-seen / last-seen. For every attribute with at least one CVE:

• one CrowdSecLog row keyed on the attribute’s own uuid (ON CONFLICT DO NOTHING, httpStatus = 200, cveId = comma-joined CVE list, url = the feed URL);
• one CrowdSecSighting per CVE, with a deterministic UUID v5 derived from logUuid + ":" + cveId so re-runs are idempotent.

CrowdSecSighting has a composite FK (cveId, source) → CVEMetadata, so all CVE ids in the feed are batch-resolved to their real source in one query up front and any CVE not yet in CVEMetadata is skipped rather than raising an FK violation. Feed failures are non-fatal: the phase logs and continues to the next feed, then to Phase 2.

Rate Limit & Loop (Phase 2)

CrowdSec CTI API: ~50 calls/day on the tier used. The processor loops through CVEs in each run until one of these stop conditions:

• 50 successful (HTTP 200) responses in one run → daily quota used productively → exit 0
• 3 consecutive HTTP 429 responses → quota exhausted → exit 0
• No CVEs found → all CVEs checked → exit 0

On HTTP 429: insert a CrowdSecLog entry (so that CVE is skipped by priorities 1–5 on the next iteration), sleep 60 seconds, then continue the loop. The next findNextCVE call may select a different CVE. If quota is truly exhausted, subsequent calls will also 429, hitting the 3-consecutive limit.

On HTTP 403 or other non-200/429: log it and continue to the next CVE.

⚠ A rate-limited attempt permanently retires the CVE from the new-CVE queue. Priorities 1–5 all filter on NOT EXISTS (SELECT 1 FROM "CrowdSecLog" WHERE cveId = …), and the CrowdSecLog insert happens before the status is examined (main.go:207-215), so a 429 or 403 counts as “checked” for ever. The CVE can then only be revisited through priority 6, a round-robin over every logged CVE at ≤50 successful calls per day.

Production state as of this audit: 220,254 CrowdSecLog rows across 111,640 distinct CVEs, of which 132,284 are HTTP 429 and 4,827 are HTTP 403 — i.e. ~62% of all recorded attempts carry no data, and the priority-6 cycle over 111,640 CVEs at 50/day is roughly a six-year round trip. A KEV CVE whose first attempt was throttled loses its priority-1 standing permanently.

Fix: either exclude non-200 log rows from the priority filters (AND cl."httpStatus" = 200), or do not insert a CrowdSecLog row for 429 at all and rely on the in-run consecutiveTooMany counter to stop the loop. The intent of the current behaviour — “don’t retry the same CVE twice in one run” — needs an in-run set, not a permanent table row.

CVE Selection Priority

Only IDs matching allowedPrefixes (default: CVE-) are processed. All 6 priority queries apply a configurable prefix filter via LIKE $N (single prefix) or LIKE ANY($N::text[]) (multiple prefixes), driven by prefixCondition(). Each priority is checked in sequence; the first result found is processed. After inserting a CrowdSecLog for a CVE (regardless of HTTP status), that CVE is excluded from priorities 1–5 on subsequent iterations within the same run (because NOT EXISTS on CrowdSecLog excludes it). Priority 6 uses the write pool to see freshly inserted log rows.

Priority 1 — KEV CVEs not yet in CrowdSecLog CVEs from the Known Exploited Vulnerabilities (KEV/CISA) catalogue that have never been queried.

Priority 2 — Exploit-linked CVEs not yet in CrowdSecLog CVEs referenced by any ExploitCVE record that have never been queried.

Priority 3 — CVEMetadata published in last 30 days, not in CrowdSecLog, oldest first Most recently published CVEs — oldest within window first so the run catches up progressively.

Priority 4 — CVEMetadata published 30–90 days ago, not in CrowdSecLog, oldest first Mid-age CVEs not yet queried.

Priority 5 — All remaining CVEMetadata not in CrowdSecLog, newest first All other CVEs ordered by most recent — ensures new CVEs are seen before very old ones.

Priority 6 — Recheck oldest DISTINCT CrowdSecLog record When all CVEs have at least one log entry, rechecks the CVE whose last query is most stale (MIN of MAX(createdAt) across all log entries per CVE). Inserting a new log entry pushes this CVE to the back of the refresh queue naturally.

Write Pool for CVE Selection

All priority queries use pool.Write (the RDS primary) to avoid read-replica replication lag. After inserting a CrowdSecLog row, the same run’s next findNextCVE call immediately sees the new row — preventing the same CVE from being selected twice in one run.

CrowdSecLog — Always Written

Even on API failure, a CrowdSecLog record is inserted with:

• httpStatus — actual HTTP status (or 0 on network error)
• errorMessage — error string (null on success)
• totalItems — count from items[] array (0 on error)
• r2Path — a path string (/crowdsec/<date>/<uuid>.jsonc). ⚠ Nothing is ever written there. The raw body is archived by s3client.Uploader at crowdsec/files/{sha256}/{cveId}.json (see § S3 Persistence), so the r2Path column records a location that does not exist and cannot be used to retrieve the payload.

Sighting Insertion

For 200 responses: iterates items[] array. Each item with a non-empty ip field is inserted as CrowdSecSighting. Uses ON CONFLICT DO NOTHING.

Attack Details Filtering

The filterAttackDetails function removes entries from attack_details that reference other CVE IDs. Looks at name, label, description fields for CVE- prefix patterns; if a different CVE ID is found, excludes that entry. This keeps sightings relevant to the target CVE only.

CSV Fields

Several fields are stored as CSV strings (not arrays):

• behaviorsCsv — behaviors[].name joined with ,
• attackDetailsCsv — filtered attack_details[].name
• classificationsCsv — classifications.classifications[].name
• mitreTechniquesCsv — mitre_techniques[].name

3. Architecture Diagram

4. Deployment Diagram

5. Processing Flow

6. Data Mapping

S3 Persistence

• Archive path: crowdsec/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/crowdsec-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: parse-error, fetch-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.