aur-fetch-processor

Status: Live Source: Arch User Repository Type: fetch (HTML scraping + RPC API + git clone) Source slug: aur Schedule: Runs hourly (at minute 50) (cron(50 * * * ? *)).

Overview

Continuously watches the AUR for new and updated packages and flags malicious PKGBUILDs. Each run walks the public package listing sorted by last-modified descending (?O= pagination, 250/page) and stops at a watermark, so only packages changed since the previous run are processed.

For every candidate package it gathers a deep dataset:

AUR RPC v5 info — votes, popularity, maintainer, submitter, dates, license, depends/makedepends.
AUR git repo (https://aur.archlinux.org/<pkgbase>.git, cloned at runtime via pure-Go go-git) — PKGBUILD, .SRCINFO, *.install scripts, commit authors/timestamps, and the latest-commit diff.
Upstream GitHub repo (only when the package’s upstream URL is a github.com project) — repository, contributors, and license.

Records produced

Condition	Records
Every scanned package	`PackageVersion` (ecosystem `aur`); `updatedAt` = upstream last-modified; AUR facts + non-minting context signals in `metadata` JSON
Upstream is github.com	`GitHubRepository` + `GitHubRepoContributor` + license fields
Malicious (≥1 evidence detection)	`CVEMetadata` (`source="aur"`, `GCVE-110-AUR-YYYY-NNNNNN`, `isMaliciousPackage=true`), one `CVEDescription` per detection, `CVEProblemType` (CWE-506 + specifics), `CVEAffected` (vendor `aur`, all versions), `CVEMetadataReferences` (AUR page + source URLs), `PackageVersionCVE`, `GcveIssuance`
Malicious + actor resolved	`ThreatActor` (maintainer/submitter/commit-author/bin-source-org) + `MalwareThreatActor` edges + `MalwareAttribution` (`attributed`, claimed/victim upstream)
Malicious	`MalwareIoc` rows (exfil endpoints, IPs, wallets, emails, install-commands, file hashes)

Detection engine

The shared malscan-engine detect package (originally internal/aurdetect, extracted so every registry processor and the package-firewall share one engine) is a Go port of Sohimaster/traur (MIT). Its data/patterns.toml carries 200+ pattern rules across four families — pkgbuild_analysis, install_script_analysis, source_url_analysis, gtfobins_analysis — plus code-derived shell-obfuscation, typosquat, bin-source-mismatch, orphan-takeover, and PKGBUILD-diff detectors.

The port deliberately drops traur’s weighted trust score and tiers. Per Vulnetix policy it adopts only the factual evaluations: any single ClassEvidence detection (download-and-execute, reverse shell, credential theft, obfuscated payloads, GTFOBins abuse, typosquatting, malicious diff, etc.) marks the package malicious. Reputation-only signals (zero votes, low popularity, few GitHub stars, missing license, package age, single/new maintainer) are ClassContext — recorded in PackageVersion.metadata but never sufficient on their own to mint an advisory. A malicious verdict is produced by static analysis and is subject to human review, which can overturn it via the existing process.

Between those two extremes sits a third finding class, trigger — a weak corroborating signal (a high-entropy embedded payload, or a supply-chain ownership/identity change such as an orphan takeover or a maintainer swap) that never mints alone. detect.CombinedVerdict folds the finding set into a verdict: it mints on any evidence, a known-bad package owner, a high-entropy payload combined with an ownership/identity change, or two independent identity-change families changing together. The package source is additionally matched against the public per-ecosystem known-bad STIX feed (domains, IPs, URLs) — a hit is evidence (CWE-506) and is recorded as a MalwareIoc with file/line + STIX provenance. The full finding-class model and the per-ecosystem capability config (22 capabilities) are documented in Malware Detection.

Gates before a mint, and what happens after (audit 2026-08-06)

Stage	Where	Effect
Known-bad hash set	`main.go:136-144`	The embedded seed list plus every hash-type `MalwareIoc` is loaded at startup; a declared/source checksum in that set is standalone evidence.
Known-bad actor set	`main.go:150-156`	Confirmed AUR perpetrator handles (high/medium confidence only, never “likely victim” rows) feed the owner-known-bad signal.
LLM false-positive gate (`aimalgate`)	`main.go:452-470`	With `PIX_INFERENCE_ENABLED` + gateway credentials set, the model is asked whether the package is genuinely malicious; only a confident “benign” drops the verdict. Gateway/parse errors fail open. Local runs are inference-free by default.
Legit-maintainer guard	`main.go:706` (`legitmaintainer.SkipActor`)	A known-legitimate identity is never linked as a threat actor.
Lock-timeout deferral	`main.go:539-545`	A `55P03` from the `PackageVersion` write (package-firewall contention) marks the package `deferred` rather than failed, and the watermark is held below it so the next run retries.
Watermark floor	`main.go:259-289`	The tracker (`BulkDataDumpTracker` row `aur`, unix seconds in `sha256`) never advances past the oldest failed/deferred package — AUR has no per-package resume hash, so the watermark is the only resume.
Actor post-pass	`main.go:276`	`malwareactor.PostPass` runs the shared actor-resolution engine over `aur` records with no attribution yet.
OSM publication	`main.go:557-559`	Off by default. Only when `OSM_SUBMIT_ENABLED=true` and `OSM_API_TOKEN` is set is a novel-indicator verdict published to OpenSourceMalware, after the DB commit, best-effort.

“Novel indicator” means the verdict rests on something beyond entropy alone and beyond an already-known bad hash; it gates both the file-hash IOC persistence and the OSM publication.

Threat-actor linkage & IOC retention

Actor attribution is a critical function of this processor: every minted malicious package is linked, as exhaustively as the available data allows, to the threat actors behind it (ThreatActor + MalwareThreatActor edges) and to the indicators it carries (MalwareIoc). This feeds the live GCVE GNA-110 actor graph (thousands of threat actors and malware edges at scale, across AUR, npm and the other registries). Implementation: internal/aur/actors.go, internal/aur/iocs.go, wired in cmd/aur-fetch-processor/main.go:linkActorsAndIOCs.

Actor vs victim — the core conditional

The 2026-06 AUR compromise worked by replacing the legitimate Maintainer field and PKGBUILD contributor lines with attacker-controlled handles/emails on the malicious revision. Attribution therefore hinges on telling the actor apart from the victim — the most important business logic in the processor:

Signal	Linked as	Platform / form	Confidence	Rationale
Current AUR `Maintainer`	actor	`aur` / `package-maintainer`	high	The handle on the malicious revision is the impostor
AUR `Submitter` (≠ maintainer), takeover detected	actor (provenance)	`aur` / `package-submitter`	low	Original creator is the likely hijack victim
AUR `Submitter` (≠ maintainer), no takeover	actor	`aur` / `package-submitter`	medium	From-scratch malicious package — creator is suspect
Latest commit author email	actor	`email` / `commit-author`	high	Author of the malicious commit (+ `commitHash`)
Latest commit author name (no email)	actor	`git` / `commit-author`	medium	Fallback identity for the malicious commit
Prior commit authors	not linked	—	—	Legitimate history → victims
Declared `github.com` upstream	claimed repo (victim)	`MalwareAttribution.claimedRepo`	—	Impersonation target, never an actor
`-bin` source GitHub org ≠ upstream org	actor	`github` / `bin-source-owner`	medium	The mismatching download org is the suspect
Emails added in the malicious commit diff	actor	`email` / `payload-contact`	medium	Exfil/contact addresses introduced by the attacker

Takeover is detected from the engine’s B-ORPHAN-TAKEOVER / B-SUBMITTER-CHANGED evidence; bin-source mismatch from B-BIN-GITHUB-ORG-MISMATCH (the suspect org is parsed from the finding’s matched source URL).

Attribution outcome conditional

flowchart TD M[Malicious package minted] --> B[BuildActorCandidates] B --> MN{Maintainer present?} MN -->|yes| MA[aur / package-maintainer · high] B --> SB{Submitter ≠ maintainer?} SB -->|takeover| SV[aur / package-submitter · low · victim-provenance] SB -->|no takeover| SS[aur / package-submitter · medium] B --> CA{Latest commit author?} CA -->|email| CE[email / commit-author · high · +commitHash] CA -->|name only| CG[git / commit-author · medium] B --> BM{bin-source org mismatch?} BM -->|yes| BO[github / bin-source-owner · medium] B --> DE{Emails in malicious diff?} DE -->|yes| DC[email / payload-contact · medium] MA & SV & SS & CE & CG & BO & DC --> CNT{linked ≥ 1 actor?} CNT -->|yes| ATTR[SetMalwareAttribution = attributed
+ claimed/victim upstream] CNT -->|no| POST[no attribution row →
malwareactor.PostPass backstop tries engine] M --> IOC[ExtractIOCs → MalwareIoc
exfil-endpoint · ipv4 · wallet · email · install-command · file-hash]

Why no attribution row on zero actors: writing a MalwareAttribution row marks the CVE enriched and excludes it from LoadPendingMalwareSubjects. When the processor resolves no actor directly, it deliberately leaves the row unset so the malwareactor.PostPass backstop (the shared GitHub/registry-API actor engine) can still attempt resolution on a later run. Mirrors the malwareactor.persist convention, including BumpThreatActorCount per new edge.

Actor graph data model

erDiagram PackageVersion ||--o{ PackageVersionCVE : "affected by" PackageVersionCVE }o--|| CVEMetadata : "cveId" CVEMetadata ||--o{ MalwareThreatActor : "cveId+source" MalwareThreatActor }o--|| ThreatActor : "threatActorUuid" ThreatActor ||--o{ ThreatActorKey : "keys (ssh/gpg)" CVEMetadata ||--o| MalwareAttribution : "outcome + claimed repo" CVEMetadata ||--o{ MalwareIoc : "indicators" CVEMetadata ||--|| GcveIssuance : "GCVE-110-AUR-YYYY-N" CVEMetadata ||--o{ CVEMetadataReferences : "sources" GitHubRepository ||--o{ GitHubRepoContributor : "contributors" GitHubRepository { string licenseSpdxId int stargazersCount } ThreatActor { string platform "aur|email|github|git" string identifier string actorType int threatCount } MalwareThreatActor { string attributionForm string attributionBasis string confidence string commitHash } MalwareIoc { string iocType string value string ecosystem }

IOC extraction

ExtractIOCs scans the PKGBUILD, install scripts, and the malicious commit’s added lines (capped at 60 indicators/package), classifying into the production open set: exfil-endpoint (Discord/Telegram webhooks, pastebins, URL shorteners, dynamic-DNS, webhook.site, transfer.sh…), ipv4 (excluding loopback), wallet (Monero/ETH/BTC), email, install-command (npm/bun/pip install of fetched payloads), and file-hash (64-hex from added lines outside checksum arrays). Plain source=() download URLs are not emitted — only exfil-class URLs — to keep the indicator set high-signal.

CWE mapping

All malicious packages get CWE-506 (Embedded Malicious Code). Specific detections map to more precise weaknesses where applicable: credential/secret theft → CWE-522, exfiltration/recon → CWE-200, reverse shells → CWE-94, cryptojacking → CWE-400.

Local run

just go-aur-fetch-backfill            # local DB, first page only via --max-pages
just go-aur-fetch-backfill prod false 1 5   # prod read, 1 page, 5 packages

Backfill runs impose no deadline and are inference-free by default. Set GITHUB_PAT (or GITHUB_TOKEN) to enable GitHub upstream enrichment.

Attribution

S3 Persistence

Archive path: aur/files/{sha256}/{filename} ✓
Quarantine path: failed-feeds/aur-fetch-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
Failure reasons emitted: store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[aur-fetch-processor] PROC -->|success| ARCHIVE[("S3: aur/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/aur-fetch-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.