community-snort-processor

Status: Live Source: community Snort rulesets on GitHub (16 raw files — travisbgreen/hunting-rules, thereisnotime/Snort-Rules) Type: snort (Snort 2.x rule text, parsed by the shared internal/emergingthreats v2 parser) Source slug: community-snort (Exploit, ExploitCVE, placeholder CVEMetadata); SnortRule rows are written under the shared slug snort Schedule: Daily at 09:00 UTC (cron(0 9 * * ? *)), 256 CPU / 512 MB, expected_duration_minutes = 15.

Overview

Detection coverage is a decision input, not trivia: when a CVE has no patch available, “is there an IDS signature we can deploy today?” is the only remaining mitigation question. This processor supplies the community half of that answer — the emergingthreats-snort-processor covers the commercial-grade ET open ruleset, and community-suricata-snort-processor covers Suricata-format community feeds.

Each run downloads a fixed list of 16 raw rule files, concatenates them, parses every rule with emergingthreats.ParseV2Rules, and keeps only rules that carry at least one reference:cve, reference:cnvd or GHSA reference (cmd/community-snort-processor/main.go:144-150). Disabled rules (#alert …) are parsed and stored with disabled=true — a commented-out signature is still evidence that a signature exists.

Rules are stored in batches of 100, each rule inside its own SAVEPOINT sp_sid_{SID} so a single bad rule cannot poison the batch (cmd/community-snort-processor/main.go:213-231).

Records produced

ConditionRecords
Every stored ruleExploit (source="community-snort", exploitId=SID, title=msg, category=classtype, platform from rule metadata, cveIds JSON)
Each reference:cve on the ruleExploitCVE junction, plus db.EnsureMinimalCVEMetadata(cveId, "community-snort") so the FK resolves — a placeholder CVEMetadata row (state="PUBLISHED", datePublished=0, no title, no rawDataJSON)
Every stored ruleSnortRule (source="snort", full parsed rule: action/protocol/addresses/ports, msg, flow, classtype, disabled, rawText, confidence, signature severity, affected products, attack targets, tags, MITRE technique/tactic ids, body options)
Each CVE / CNVD / GHSA id on the ruleSnortRuleCVE junction (the cveId column holds non-CVE identifiers too — 34 CNVD ids in production)
Successful batchRaw rule text archived to community-snort/files/{sha256}/{SID}.rules
Failed rule or failed batchRule text quarantined to failed-feeds/community-snort-processor/{date}/store-error/{SID}.rules

Placeholder rows under source='community-snort'

The CVEMetadata rows this processor creates exist only to satisfy the ExploitCVE foreign key. Treat source='community-snort' as an alias-shell source: exclude it from any title/CVSS coverage denominator. Production holds 4,381 such rows, of which 1,305 are pure placeholders; the remaining ~3,076 carry titles and dates written by an earlier revision of this processor that cloned full CVE records, and are no longer refreshed by the current code.

Known issues

  1. Exploit.originalUrl points at the wrong ruleset. MapToExploitRow is shared with the Emerging Threats processor and hardcodes https://rules.emergingthreats.net/open/snort-2.9.0/emerging-all.rules#sid-{SID} (internal/emergingthreats/mapper.go:12,34). All 2,146 production community-snort exploit rows carry that URL instead of the GitHub raw URL the rule came from, so the provenance link is misleading. The fix is to pass the per-rule source URL through the mapper.
  2. Resume set is shared with the Suricata processor. Both processors call db.LoadProcessedSnortRuleIDs(ctx, pool.Read, "snort") (cmd/community-snort-processor/main.go:111) and both write SnortRule rows under source='snort', which is unique on (snortId, source) (internal/db/snortrule.go:72). Snort SIDs are only unique within a ruleset, so whichever processor runs first claims a SID and the other silently skips its own, different rule with that number. No overlap is observable in production today (0 shared SIDs between the two Exploit sources), but a skipped rule leaves no trace, so absence of overlap is not proof of absence of loss. Give each ruleset its own SnortRule.source, or key the resume set on (source, SID).
  3. The soft deadline applies to backfills. The batch loop stops at a hardcoded 20-minute deadline when EXPECTED_DURATION_MINUTES is unset (cmd/community-snort-processor/main.go:84-88,176-179), and just go-community-snort-backfill unsets it — contrary to the “backfills must run to completion” rule in scripts/go-processors/AGENTS.md.

Failure modes

FailureBehaviour
One rule file 404s / times outLogged, RecordError, run continues with the remaining files (3 attempts each, 3 s × attempt backoff)
All 16 fetches failtask.errored to SNS and exit 1
Zero CVE-bearing rules parsedtask.completed with stored=0, no error
Single rule fails to storeRolled back to its savepoint, quarantined to S3, counted in errors; run continues
Batch transaction failsWhole batch quarantined, RecordError, next batch attempted
Any accumulated error at end of runtask.errored (Slack alert) instead of task.completed

Flags

FlagDefaultDescription
--allfalseReprocess every parsed rule, bypassing the SID resume set
--limit0Maximum rules to store (0 = unlimited)
--forcefalseSame effect as --all for the resume set

Local run: just go-community-snort-backfill prod (positional: TARGET ALL LIMIT FORCE).

Key files

FilePurpose
cmd/community-snort-processor/main.goRule-file list, fetch/parse loop, batch + savepoint writes
internal/emergingthreats/parser_v2.goSnort 2.x rule parser (shared)
internal/emergingthreats/mapper.goMapToExploitRow, MapToSnortRuleRow (shared)
internal/db/snortrule.goUpsertSnortRule, InsertSnortRuleCVEs, LoadProcessedSnortRuleIDs
internal/db/exploit.goUpsertExploit, InsertExploitCVE

Verification

-- rules stored by this processor (Exploit side)
SELECT count(*) FROM "Exploit" WHERE source = 'community-snort';

-- CVE junctions
SELECT count(*) FROM "SnortRuleCVE" src
JOIN "SnortRule" sr ON sr.uuid = src."snortRuleUuid"
WHERE sr.source = 'snort';

-- placeholder CVEMetadata rows (expected: no title, datePublished = 0)
SELECT count(*) FILTER (WHERE title IS NULL) AS placeholders, count(*) AS total
FROM "CVEMetadata" WHERE source = 'community-snort';

S3 Persistence

  • Archive path: community-snort/files/{sha256}/{filename} ✓
  • Quarantine path: failed-feeds/community-snort-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
  • Failure reasons emitted: (none documented)

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[community-snort-processor] PROC -->|success| ARCHIVE[("S3: community-snort/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/community-snort-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.