hex-json-processor

Status: Live Source: Hex.pm API + repo.hex.pm tarballs Type: json (recent-packages JSON feed + per-package JSON API + release tarball fetch) Source slug: hex Schedule: Runs hourly (at minute 0) (cron(0 * * * ? *)).

Overview

Continuously watches the Hex.pm registry (the Elixir / Erlang BEAM ecosystem) for new and updated packages and flags malicious publishes. Each run walks the recent-packages feed (hex.pm/api/packages?sort=updated_at, descending) from a stored updated_at watermark, so only packages changed since the previous run are processed. Pages are walked until a whole page falls at or below the watermark (or the page/limit caps are hit); duplicate package names within a run are skipped.

For every candidate package it resolves the latest stable version (latest_stable_version, else latest_version) and gathers:

• Package metadata — version, homepage and repository links (meta.links), license (meta.licenses[0]), description, and updated_at.
• mix.exs build manifest — mix.exs executes arbitrary Elixir at compile time (the classic Hex supply-chain execution vector). It feeds the install-script detectors.
• Release source tree — the published tarball (repo.hex.pm/tarballs/<name>-<version>.tar) is an uncompressed tar holding metadata.config (an Erlang term manifest) and contents.tar.gz. The processor reads the outer tar, gunzips contents.tar.gz, and concatenates mix.exs + .ex/.exs/.erl sources (bounded ~512 KB) for deep static detection (System.cmd, :os.cmd, Code.eval_string, :erlang.binary_to_term, base64-decode-then-eval).
• Upstream GitHub repo (only when the package links to a github.com project) — repository, contributors, and license, persisted GitHub-first in its own committed transaction.

Records produced

Gates before minting

Three independent gates sit between a detection and an advisory:

• LLM false-positive gate (internal/aimalgate, main.go:291-304) — when PIX_INFERENCE_ENABLED + an AI-Gateway token are configured, the evidence set and the scanned source are put to the model before minting. Only a confident legitimate verdict drops the detection; any gateway or parse error fails open and keeps the verdict, so the gate can only remove false positives, never suppress a real detection on infrastructure failure. Cleared packages are counted as aiCleared in the run summary.
• Human curation gate (curationgate.PackageCleared, main.go:327) — a package/version a reviewer has already cleared is never re-minted, even when the detectors fire again. The PackageVersion row is still written.
• Legitimate-maintainer guard (legitmaintainer.SkipActor, main.go:397) — a well-known upstream identity is never branded a threat actor, which is what stops an ownership-reputation signal cascading onto every package a legitimate maintainer publishes.

Detection

Reuses the shared malscan-engine detect engine. The mix.exs manifest is scanned by the install-script detectors; the full source bundle (metadata.config + mix.exs + .ex/.erl) by the general / shell / source-url detectors. BEAM-specific evidence rules target compile-time command execution (System.cmd, :os.cmd, :os.cmd in a Mix.Project/compiler callback), code injection (Code.eval_string, Code.eval_quoted, :erlang.binary_to_term of attacker data — CWE-94/CWE-502), and base64-decode-then-eval payloads (CWE-94); environment/secret exfiltration via System.get_env to a network sink (CWE-200). Findings carry one of three classes: evidence (a factual malicious behaviour — mints the advisory on its own), trigger (a weak corroborating signal such as a high-entropy embedded payload or a supply-chain ownership/identity change — never mints alone), and context (reputation/risk, recorded as PackageVersion.metadata only). detect.CombinedVerdict mints on any evidence, a known-bad package owner, a high-entropy payload combined with an ownership/identity change, or two independent identity-change families changing together. The full finding-class model and the per-ecosystem capability config (22 capabilities) live in Malware Detection.

Known-bad IOC matching — the package’s declared source is also matched against the public per-ecosystem known-bad STIX feed (domains, IPs, URLs), loaded once per run and matched in memory. A hit is evidence (CWE-506) folded into the finding set before the combination gate, and is recorded as a MalwareIoc row whose references carry the file/line and STIX provenance.

Resume

An updated_at watermark (RFC3339) is stored in BulkDataDumpTracker under source hex-recent (the sha256 column holds the maximum processed updated_at). The first run (no tracker) or a --force run processes the full feed; subsequent runs resume from the saved watermark.

S3 Persistence

• Archive path: hex/files/hex-{package}-{version}/{package}/sources.txt ✓
• Quarantine path: failed-feeds/hex-json-processor/{YYYY-MM-DD}/{reason}/{package}/sources.txt ✓
• Failure reasons emitted: store-error

The unit of work is a package release, not a file: the concatenated mix.exs + Elixir/Erlang source text that was actually scanned is archived after a successful store (main.go:385) and quarantined when the store transaction fails (main.go:379). The archive key slot holds the hex-{package}-{version} coordinate rather than a content hash, so re-scanning the same release overwrites in place. Skipped when S3_BUCKET_NAME is unset (local dev).