GitHub Yara Rules Fetcher — Design

Status: Live Source: GitHub Code Search API (*.yar / *.yara files) Type: fetch (Code Search + contents API) Source slug: github-yara (Exploit.source; YaraRule.source is github-yara:{owner}/{repo}) — no CVEMetadata rows Schedule: Runs daily at 06:00 UTC (cron(0 6 * * ? *)).

Overview

Discovers YARA detection rules across GitHub that reference vulnerability identifiers and stores them as YaraRule + YaraRuleCVE rows (plus a backward-compatible Exploit + ExploitCVE row per file). Mirrors the github-poc-processor architecture: GitHub Code Search is sharded by year × prefix × extension to bypass the 1,000-result cap. Vuln-ID prefixes (CVE-, GHSA-, CNVD-, BDU-) are extracted from rule content.

This is the only source of host/file-level detection content in the VDB: the rules it collects are what /api/vdb/v2/yara-rules serves and what the detection-rules / incident-respond CLI skills hand a defender when a CVE has no patch yet. Production holds ~5.7k rules from ~219 repositories. Without it, “here is how to detect this in the wild” collapses to Nuclei templates only.

How the walk works

enrichment.SearchAndProcessYaraRules builds one query per (prefix, year, extension) — CVE-/GHSA- × 1999…next-year × yar/yara, ≈120 shards — and pages each shard up to 10 pages of 100 results, sorted indexed desc. Rate-limit and abuse-limit errors sleep until the reset and retry the same page. For every hit it:

  1. Checks the per-file resume row (below) and stops the whole shard at the first file whose blob SHA is unchanged — the newest-indexed ordering makes that a cheap “caught up” signal, at the cost of skipping anything older in that shard that was never processed.
  2. Fetches the file contents, extracts vuln IDs with the --prefixes set, and drops the file when none match.
  3. Parses it with internal/yara.Parse; a file that yields zero rules is marked processed and skipped.
  4. Writes one YaraRule + its YaraRuleCVE links per parsed rule, each in its own transaction, then one Exploit + ExploitCVE set for the file.

--limit caps files per run; a 50-minute soft deadline (EXPECTED_DURATION_MINUTES − 10) stops the walk cleanly.

Concurrency guard and resume

ConcernMechanism
Overlapping runsBulkDataDumpTracker row github_yara used as a lock (enrichment.CheckLock/AcquireLock), re-acquired every 5 minutes as a heartbeat and released on clean exit
Per-file resumeOne BulkDataDumpTracker row per discovered file, source = "github_yara_file:{owner}/{repo}/{path}", sha256 = the git blob SHA (1,342 such rows in production)
Rule identitybodyContentHash = SHA-256 of the rule body with comments stripped, so a reformatted comment does not fork the rule

Data Source

PropertyValue
SourceGitHub Code Search API
AuthGITHUB_PAT (required)
Query strategySharded by year × ID prefix × file extension
ContentYARA rule files referencing CVE/GHSA/CNVD/BDU IDs

Storage

TableRows inserted
YaraRuleOne per parsed rule (not per file); source='github-yara:{owner}/{repo}', sourceUrl = the GitHub HTML URL, sourceCommitSha = blob SHA, plus meta/strings/condition/tags/imports JSON, author/description/reference from rule meta, and datePublished/dateUpdated from the file’s first commit date
YaraRuleCVEOne per (rule, vuln-ID) link
ExploitOne per file — exploitId='{owner}/{repo}@{path}', source='github-yara', platform='yara', category='detection', cveIds JSON array
ExploitCVEOne per (file, resolved (cveId, source)) pair — only for CVEs already present in CVEMetadata
BulkDataDumpTrackerThe github_yara run lock plus one github_yara_file:{owner}/{repo}/{path} resume row per discovered file

No CVEMetadata, CVEAlias or CVEMetadataReferences rows are written — this processor only attaches detection content to CVEs that already exist.

Failure modes

ConditionBehaviour
GITHUB_PAT unsetfatal, exit 1
Lock held by a recent runnotifier.NoWork, exit 0
Search / contents / parse / upsert failurelogged at warn and skipped
Soft deadline or --limit reachedwalk stops cleanly, lock released

⚠ Known defect (audit 2026-08-06) — failures are invisible. SearchAndProcessYaraRules always returns nil and every per-shard, per-file and per-rule failure is a logger.Warn with no counter (internal/enrichment/yara.go:116,160,166,249,254). main.go:132 then calls notifier.Completed unconditionally, so a run in which every shard 4xx’d and nothing was stored is reported as a success and exits 0. Add error counters and finalise through notifier.Finalize (as alpine-apk/homebrew do). Related: nothing is ever quarantined, and the archive key is yara/{owner}/{repo}/{path} (internal/enrichment/yara.go:186) rather than the contract’s content-addressed {source}/files/{sha256}/{filename}, so the generated S3 block above overstates compliance.

Flags

FlagDefaultDescription
--limit0Maximum files to process (0 = unlimited)
--prefixesCVE-,GHSA-,CNVD-,BDU-Vuln-ID prefixes extracted from rule content. Search shards only use CVE-/GHSA- (the year-bearing prefixes); CNVD-/BDU- are picked up from the body of files those shards already matched

ECS Schedule

Runs daily at 06:00 UTC (cron(0 6 * * ? *)); 256 CPU / 512 MB, expected_duration_minutes = 50 (the walk’s own soft deadline is 50 − 10 = 40 minutes).

CloudWatch logs retain 3 days, then archive to S3 (Glacier Deep Archive).

S3 Persistence

  • Archive path: github-yara/files/{sha256}/{filename} ✓
  • Quarantine path: failed-feeds/github-yara-fetch-processor/{YYYY-MM-DD}/{reason}/{filename} ⚠ not yet wired
  • Failure reasons emitted: fetch-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[github-yara-fetch-processor] PROC -->|success| ARCHIVE[("S3: github-yara/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/github-yara-fetch-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.