community-suricata-snort-processor

Status: Live Source: quadrantsec/suricata-rules (single file) + daffainfo/suricata-rules per-year CVE-{year}.rules (2000→current) and CNVD-{year}.rules (2018→current) Type: snort (Suricata rule text — syntax-compatible with the shared Snort 2.x parser) Source slug: community-suricata (Exploit, ExploitCVE, placeholder CVEMetadata); SnortRule rows are written under the shared slug snort Schedule: Daily at 10:00 UTC (cron(0 10 * * ? *)), 256 CPU / 512 MB, expected_duration_minutes = 15.

Overview

Suricata is the more common open IDS deployment today, and the daffainfo feeds are organised per vulnerability year — including a CNVD series that carries signatures for Chinese national-vulnerability-database advisories that never receive a CVE. That CNVD coverage is what this processor adds over community-snort-processor (GitHub Snort rulesets) and emergingthreats-snort-processor (ET open ruleset): it is the only feed in the VDB that links a CNVD-… identifier to a deployable network signature.

The URL list is built at runtime from the current year (cmd/community-suricata-snort-processor/main.go:51-75), so a new year’s file is picked up automatically. Missing years return HTTP 404 and are skipped quietly — a 404 is expected, not an error. Rules are parsed with the shared emergingthreats.ParseV2Rules and only those carrying a CVE, CNVD or GHSA reference are stored, in batches of 100 with a per-rule savepoint.

Records produced

CNVD ids get no placeholder CVEMetadata row (only reference:cve ids do), so a CNVD-only rule is reachable through SnortRuleCVE but has no Exploit↔CVE junction. That is intentional: CVEMetadata is keyed by (cveId, source) and a CNVD advisory belongs to the cnvd source, not to this one.

Placeholder rows under source='community-suricata'

All 501 production CVEMetadata rows for this source are FK placeholders — no title, no rawDataJSON, datePublished = 0. This is correct behaviour, not a defect: exclude the source from title/CVSS coverage measurements entirely (see the alias-shell rule in .repo/efficacy/README.md).

Known issues

1. Shared resume set and shared SnortRule.source. Like the Snort processor, this one loads its resume set with db.LoadProcessedSnortRuleIDs(ctx, pool.Read, "snort") (cmd/community-suricata-snort-processor/main.go:123) and writes rules under source='snort', unique on (snortId, source) (internal/db/snortrule.go:72). Because it runs an hour after community-snort-processor, any SID already claimed by a GitHub Snort rule causes this processor to skip its own Suricata rule with the same number, silently. Production shows 0 overlapping SIDs between the two Exploit sources — but a skipped rule leaves no row, so that is not proof of no loss.
2. Exploit.originalUrl is wrong. The shared MapToExploitRow hardcodes the Emerging Threats ruleset URL (internal/emergingthreats/mapper.go:12,34); all 573 community-suricata exploit rows point there rather than at the daffainfo/quadrantsec file the rule came from.
3. The soft deadline applies to backfills. The batch loop stops at a hardcoded 20-minute deadline when EXPECTED_DURATION_MINUTES is unset (cmd/community-suricata-snort-processor/main.go:96-100,191-194), which is exactly what just go-community-suricata-snort-backfill does. With ~35 feed URLs to fetch, a cold backfill can be truncated.

Failure modes

Flags

Local run: just go-community-suricata-snort-backfill prod (positional: TARGET ALL LIMIT FORCE).

Key files

Verification

-- rules stored by this processor
SELECT count(*) FROM "Exploit" WHERE source = 'community-suricata';

-- CNVD identifiers linked to signatures
SELECT src."cveId" FROM "SnortRuleCVE" src
JOIN "SnortRule" sr ON sr.uuid = src."snortRuleUuid"
WHERE sr.source = 'snort' AND src."cveId" LIKE 'CNVD-%'
LIMIT 10;

S3 Persistence

• Archive path: community-suricata/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/community-suricata-snort-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: (none documented)

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.