go-json-processor

Status: Live Source: Go module index + module proxy Type: json (module index feed + module proxy metadata + .zip source fetch) Source slug: go Schedule: Runs hourly (at minute 30) (cron(30 * * * ? *)).

Overview

Continuously watches the Go module ecosystem for new and updated module@version publishes and flags malicious ones. Each run walks the public module index (index.golang.org/index?since=<RFC3339>&limit=2000) from a stored timestamp watermark, so only modules published since the previous run are processed. The feed is newline-delimited JSON ordered chronologically; duplicate module paths within a run are processed only once.

For every candidate module it resolves the upstream GitHub repository from the module path (github.com/owner/repo) and queries the module proxy (proxy.golang.org) using the GOPROXY case-encoding (each uppercase letter X escaped as !x) to gather:

• Module metadata — version, published time (.info), homepage (pkg.go.dev/<path>), and derived repository URL.
• go.mod — the module’s require / replace / retract directives. Unlike npm/PyPI, Go runs no install scripts at go get, so the go.mod is treated as a cheap supply-chain signal and feeds the install-script detectors.
• Module sources — the published .zip is downloaded (bounded ~8 MB) and its .go files plus go.mod concatenated (bounded ~512 KB) for deep static detection. The Go malware vector is malicious code that runs when the module is imported/built — os/exec / syscall.Exec, base64-decode-then-exec, cgo download stubs, and //go:generate directives that shell out.
• Upstream GitHub repo (when the module path is a github.com project) — repository, contributors, and license, persisted GitHub-first in its own committed transaction.

Records produced

Gates before minting

Three independent gates sit between a detection and an advisory:

• LLM false-positive gate (internal/aimalgate, main.go:304-316) — when PIX_INFERENCE_ENABLED + an AI-Gateway token are configured, the evidence set and the scanned source are put to the model before minting. Only a confident legitimate verdict drops the detection; any gateway or parse error fails open and keeps the verdict, so the gate can only remove false positives, never suppress a real detection on infrastructure failure. Cleared packages are counted as aiCleared in the run summary.
• Human curation gate (curationgate.PackageCleared, main.go:340) — a package/version a reviewer has already cleared is never re-minted, even when the detectors fire again. The PackageVersion row is still written.
• Legitimate-maintainer guard (legitmaintainer.SkipActor, main.go:410) — a well-known upstream identity is never branded a threat actor, which is what stops an ownership-reputation signal cascading onto every package a legitimate maintainer publishes.

Detection

Reuses the shared malscan-engine detect engine. The go.mod is scanned by the install-script detectors; the concatenated .go sources by the general / shell / source-url detectors. Go-specific evidence rules include exec.Command/os/exec invoking a shell, syscall.Exec, base64-decode-then-exec of an embedded payload, cgo download stubs, and //go:generate directives that run curl/wget (CWE-94/CWE-506); environment/credential exfiltration over the network (CWE-200). Findings carry one of three classes: evidence (a factual malicious behaviour — mints the advisory on its own), trigger (a weak corroborating signal such as a high-entropy embedded payload or a supply-chain ownership/identity change — never mints alone), and context (reputation/risk, recorded as PackageVersion.metadata only). detect.CombinedVerdict mints on any evidence, a known-bad package owner, a high-entropy payload combined with an ownership/identity change, or two independent identity-change families changing together. The full finding-class model and the per-ecosystem capability config (22 capabilities) live in Malware Detection.

Known-bad IOC matching — the package’s declared source is also matched against the public per-ecosystem known-bad STIX feed (domains, IPs, URLs), loaded once per run and matched in memory. A hit is evidence (CWE-506) folded into the finding set before the combination gate, and is recorded as a MalwareIoc row whose references carry the file/line and STIX provenance.

There is no first-class registry publisher account in the Go ecosystem (the proxy mirrors whatever the VCS host serves), so unlike npm there is no maintainer-account actor — the declared upstream GitHub project is recorded as the claimed/victim (typosquatting target) and embedded emails as payload-contact actors.

Resume

A timestamp watermark is stored in BulkDataDumpTracker under source go-index (the sha256 column holds the last processed index RFC3339 timestamp string). The first run (no tracker) starts a configurable window before now (--backfill-hours, default 48) so recent activity is swept; subsequent runs resume from the saved timestamp.

S3 Persistence

• Archive path: go/files/go-{module}-{version}/{module}/sources.go ✓
• Quarantine path: failed-feeds/go-json-processor/{YYYY-MM-DD}/{reason}/{module}/sources.go ✓
• Failure reasons emitted: store-error

The unit of work is a module, not a file: the concatenated go.mod + .go source text that was actually scanned is archived after a successful store (main.go:398) and quarantined when the store transaction fails (main.go:392). The archive key slot holds the go-{module}-{version} coordinate rather than a content hash, so re-scanning the same module@version overwrites in place instead of accumulating a new object per run. Skipped when S3_BUCKET_NAME is unset (local dev).