homebrew-fetch-processor

Status: Live Source: formulae.brew.sh API + GitHub (taps & upstreams) Type: fetch (JSON API + GitHub API + raw Ruby source) Source slug: homebrew (casks recorded under ecosystem homebrew-cask) Schedule: Runs daily at 10:00 UTC (cron(0 10 * * ? *)).

Overview

The Homebrew counterpart to aur-fetch-processor, with full feature parity (PackageVersion records, malware detection, GCVE-110-BREW minting, threat-actor linkage, IOC retention). Homebrew is GitHub-native, so the processor’s defining invariant is:

GitHub repository data (repo + contributors + license) resolved from a Homebrew source is persisted to the DB FIRST — in its own committed transaction — regardless of whether the malware detection / actor / IOC / CVE follow-up runs or stores anything more.

Two acquisition modes (`--mode=api|taps|both`)

Mode	Source	Surface	Resume
`api`	`formulae.brew.sh/api/formula.json` + `cask.json`	homebrew-core + homebrew-cask (curated)	per-item `ruby_source_checksum` (exact change detection)
`taps`	GitHub repo search `homebrew- in:name sort:updated`	third-party taps (the open, AUR-equivalent malware surface)	`pushed_at` watermark (`BulkDataDumpTracker` `homebrew-taps`)

flowchart TD subgraph A["Mode A — official API"] FA[formula.json + cask.json] --> RA{checksum changed?} RA -->|no| SKIP[skip] RA -->|yes| UA[resolve upstream GitHub
homepage / url / head] end subgraph B["Mode B — tap discovery"] SB[GitHub search homebrew-* sort:updated] --> WB{pushed_at > watermark?} WB -->|no| STOPB[stop walk] WB -->|yes| TB[tap repo = GitHub source] TB --> FB[enumerate /Formula /Casks *.rb] end UA --> GH1[(Tx1: GitHubRepository
+contributors+license
COMMIT FIRST)] TB --> GH1 FB --> GH1 GH1 --> RUBY[fetch Ruby source] RUBY --> DET[aurdetect.Detect over Ruby DSL
+ Homebrew name typosquat] DET --> PV[(Tx2: PackageVersion
ecosystem homebrew / homebrew-cask)] PV --> MAL{evidence detection?} MAL -->|no| DONE[done — GitHub data already persisted] MAL -->|yes| MINT[(Tx3: GCVE-110-BREW mint
+ actors + IOCs)]

GitHub-first ordering

Tx1 (GitHub repo) commits before Tx2 (PackageVersion) and Tx3 (mint). For Mode A the GitHub repo is the formula’s resolved upstream (homepage/urls.stable.url/ urls.head.url); for Mode B it is the tap repo itself (and the referenced upstream too). Even if detection or minting fails, the GitHub graph is enriched — thousands of Homebrew upstream projects flow into GitHubRepository + GitHubRepoContributor + license regardless of malware outcome.

Detection

Reuses the shared malscan-engine detect engine (the package formerly vendored as internal/aurdetect; 200+ pattern rules + shell-obfuscation) directly over the entire Ruby formula/cask source — the DSL embeds install logic (system "curl … | bash", shell_output, cask installer script/postflight) anywhere in the class body, so the download-and-execute / reverse-shell / GTFOBins / source-url families apply verbatim. Homebrew tuning adds a popular-formula typosquat list and treats analytics install-counts as the popularity (context) signal — the Homebrew analog of AUR votes. Findings carry one of three classes: evidence (factual malicious behaviour — mints on its own), trigger (a weak corroborating signal such as a high-entropy embedded payload or an ownership/identity change — never mints alone) and context (reputation only, recorded as metadata). detect.CombinedVerdict mints on any evidence, a known-bad owner, a high-entropy payload combined with an ownership/identity change, or two independent identity-change families changing together. The package source is also matched against the public per-ecosystem known-bad STIX feed (domains, IPs, URLs) — a hit is evidence (CWE-506), recorded as a MalwareIoc with file/line + STIX provenance. The full model and the per-ecosystem capability config (22 capabilities) are documented in Malware Detection.

Threat-actor linkage (actor vs victim)

Mirrors the AUR model with GitHub as the identity platform:

Signal	Linked as	Platform / form	Confidence
Third-party tap owner	actor	`github` / `tap-owner`	high
Formula-file latest commit author (third-party tap)	actor	`email`\|`git` / `commit-author`	high/medium
Official-tap committer (homebrew/core\|cask)	not linked	—	— (trusted maintainer)
Declared upstream GitHub project	claimed/victim repo	`MalwareAttribution.claimedRepo`	—
Emails embedded in the Ruby source	actor	`email` / `payload-contact`	medium

flowchart TD M[Malicious formula/cask] --> OFF{official tap?} OFF -->|homebrew/core or cask| VIC[no actor — committers are trusted maintainers;
upstream recorded as claimed/victim → PostPass backstop] OFF -->|third-party tap| TO[tap owner → github/tap-owner · high] OFF -->|third-party tap| CA[commit author → email/commit-author · high] M --> UP[upstream GitHub → claimed/victim repo] M --> EM[embedded emails → payload-contact] TO & CA & EM --> CNT{linked ≥ 1 actor?} CNT -->|yes| ATTR[MalwareAttribution = attributed + claimed repo] CNT -->|no| POST[no attribution row → malwareactor.PostPass backstop] M --> IOC[ExtractIOCs → MalwareIoc]

IOC retention reuses aur.ExtractIOCs over the Ruby source (exfil endpoints, IPs, wallets, emails, install-commands, file-hashes).

Records produced

Condition	Records
Every item with a GitHub source	`GitHubRepository` + `GitHubRepoContributor` + license (committed first)
Every scanned item	`PackageVersion` (ecosystem `homebrew`/`homebrew-cask`; metadata = tap/homepage/urls/license/deps/installs/checksum/context)
Malicious (≥1 evidence)	`CVEMetadata` (`source="homebrew"`, `GCVE-110-BREW-YYYY-NNNNNN`), per-detection `CVEDescription`, `CVEProblemType` (CWE-506+), `CVEAffected` (vendor `homebrew`), `CVEMetadataReferences`, `PackageVersionCVE`, `GcveIssuance`
Malicious + actor resolved	`ThreatActor` + `MalwareThreatActor` + `MalwareAttribution`
Malicious	`MalwareIoc`

Idempotency & resume

Mode A: skip items whose ruby_source_checksum is unchanged (skip-set built from PackageVersion.metadata).
Mode B: walk newest-pushed taps until the pushed_at watermark; the watermark never advances past the oldest failed tap (transient failures are retried).
Minting reuses an item’s existing GCVE-110-BREW id (findExistingBrewCveID), so re-runs and new malicious revisions update the same identifier.

Local run

just go-homebrew-fetch-backfill prod false api 0 5     # API mode, 5 items
just go-homebrew-fetch-backfill prod false taps 1 5    # tap discovery, 1 page

GITHUB_PAT (or GITHUB_TOKEN) is required for tap discovery, upstream repo enrichment, and commit-author resolution.

Gates before a mint (audit 2026-08-06)

Three gates sit between a malicious verdict and an advisory, and all three can only ever remove a detection:

Gate	Where	Effect
Cross-registry ownership (`hijack.Verdict`)	`main.go:404`	Folds ownership/identity change and owner-known-bad (P4) into the verdict; content evidence (P0) still mints on its own. No prior-revision maintainer is passed, so the live ownership signal is owner-known-bad.
LLM false-positive gate (`aimalgate`)	`main.go:414-428`	With `PIX_INFERENCE_ENABLED` + AI-gateway credentials set, the model is asked whether the item is genuinely malicious; only a confident “benign” drops the verdict (counted as `aiCleared`). Gateway/parse errors fail open. Local runs are inference-free by default.
Human curation (`curationgate.PackageCleared`)	`main.go:455`	A formula/cask/version a reviewer already cleared is never re-minted.

Actor candidates pass legitmaintainer.SkipActor before being linked, the PackageVersion write runs under db.WithTxRetry (package-firewall holds row locks for tens of seconds), and the run finalises through notifier.Finalize, so a handful of transient per-record failures do not page.

Attribution

Detection engine and patterns derived from Sohimaster/traur (MIT) via the shared malscan-engine/detect package.

S3 Persistence

Archive path: homebrew/files/{sha256}/{filename} ✓
Quarantine path: failed-feeds/homebrew-fetch-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
Failure reasons emitted: store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[homebrew-fetch-processor] PROC -->|success| ARCHIVE[("S3: homebrew/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/homebrew-fetch-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.