npm-json-processor

Status: Live Source: npm registry + replicate.npmjs.com _changes Type: json (CouchDB _changes firehose + per-package JSON API + tarball fetch) Source slug: npm Schedule: every 30 minutes (cron(0/30 * * * ? *))

Overview

Continuously watches the npm registry for new and updated packages and flags malicious publishes. Each run walks the CouchDB _changes feed (replicate.npmjs.com/_changes?since=<seq>) from a stored sequence watermark, so only packages changed since the previous run are processed. Deleted/unpublished change rows and duplicate package ids within a run are skipped.

For every candidate package it fetches the registry document (registry.npmjs.org/<name>), resolves the latest dist-tags.latest version, and gathers:

• Package metadata — version, homepage, repository, license, maintainers (publisher accounts), author, dependencies, dist (tarball URL, shasum, integrity), and the scripts block.
• Lifecycle install hooks — preinstall / install / postinstall / prepare script bodies, the classic npm supply-chain execution vector. These feed the install-script detectors.
• Tarball JavaScript — for packages that declare lifecycle hooks (the high-risk subset), the published .tgz is downloaded (bounded) and its .js/.cjs/.mjs/.ts sources concatenated for deep static detection (eval(Buffer.from(...)), child_process exec of decoded payloads, process.env exfiltration, obfuscator signatures).
• Upstream GitHub repo (only when the package’s repository/homepage is a github.com project) — repository, contributors, and license, persisted GitHub-first in its own committed transaction.

Records produced

Gates before minting

npm is the noisiest ecosystem in the fleet (build/dev tooling invoked through npx, and owner-reputation cascades that carry no malicious code of their own), so three independent gates sit between a detection and an advisory:

• LLM false-positive gate (internal/aimalgate, main.go:308-321) — when PIX_INFERENCE_ENABLED + an AI-Gateway token are configured, the evidence set and the scanned source are put to the model before minting. Only a confident benign verdict drops the detection; any gateway or parse error fails open and keeps the verdict, so the gate can only remove false positives, never suppress a real detection on infrastructure failure. Cleared packages are counted as aiCleared in the run summary.
• Human curation gate (curationgate.PackageCleared, main.go:367) — a package/version a reviewer has already cleared is never re-minted, even when the detectors fire again. The PackageVersion row is still written.
• Legitimate-maintainer guard (legitmaintainer.SkipActor, main.go:445) — a well-known upstream maintainer is never branded a threat actor, which is what stops an ownership-reputation signal cascading onto every package they publish.

Detection

Reuses the shared malscan-engine detect engine. Lifecycle hook bodies are scanned by the install-script detectors; the full scripts block plus tarball JavaScript by the general / shell / source-url detectors. npm/JS-specific evidence rules include eval()/new Function() of base64-decoded payloads, child_process exec of decoded commands, node -e/--eval install hooks, and base64-decoding install hooks (CWE-94); process.env network exfiltration (CWE-200); and obfuscator signatures (context). Findings carry one of three classes: evidence (a factual malicious behaviour — mints the advisory on its own), trigger (a weak corroborating signal such as a high-entropy embedded payload or a supply-chain ownership/identity change — never mints alone), and context (reputation/risk, recorded as PackageVersion.metadata only). detect.CombinedVerdict mints on any evidence, a known-bad package owner, a high-entropy payload combined with an ownership/identity change, or two independent identity-change families changing together. The full finding-class model and the per-ecosystem capability config (22 capabilities) live in Malware Detection.

Known-bad IOC matching — the package’s declared source is also matched against the public per-ecosystem known-bad STIX feed (domains, IPs, URLs), loaded once per run and matched in memory. A hit is evidence (CWE-506) folded into the finding set before the combination gate, and is recorded as a MalwareIoc row whose references carry the file/line and STIX provenance.

Resume

A sequence watermark is stored in BulkDataDumpTracker under source npm-changes (the sha256 column holds the last processed _changes sequence). The first run (no tracker) starts a configurable window before the registry head so recent activity is swept; subsequent runs resume from the saved sequence.

S3 Persistence

• Archive path: npm/files/{dist.shasum}/{package}/scripts.js ✓ (falls back to npm-{package}-{version} when the registry document carries no shasum)
• Quarantine path: failed-feeds/npm-json-processor/{YYYY-MM-DD}/{reason}/{package}/scripts.js ✓
• Failure reasons emitted: store-error
• Malware-sample archive: encrypted .tgz under the shared malware-samples/ prefix (internal/malsample) — a separate, attribution-gated path from the raw-payload archive above

The unit of work is a package version, not a file: the concatenated lifecycle-hook scripts plus tarball JavaScript that was actually scanned is archived after a successful store (main.go:433) and quarantined when the store transaction fails (main.go:427). Skipped when S3_BUCKET_NAME is unset (local dev) — which also disables live malware-sample capture (main.go:328).