nuget-json-processor

Status: Live Source: NuGet v3 catalog + per-leaf detail JSON + flat container .nupkg Type: json (catalog event-log index + per-page leaves + per-leaf JSON + .nupkg fetch) Source slug: nuget Schedule: Runs hourly (at minute 35) (cron(35 * * * ? *)).

Overview

Continuously watches the NuGet (.NET) ecosystem for new and updated package versions and flags malicious publishes. The NuGet v3 catalog is a chronological, append-only event log: the catalog index (api.nuget.org/v3/catalog0/index.json) lists catalog pages (each carrying a commitTimeStamp watermark and a count); each page lists leaves (per-package-version publish / delete events).

Each run reads the catalog index, selects pages whose commitTimeStamp is after a stored RFC3339 watermark (oldest-first), and processes their nuget:PackageDetails leaves (PackageDelete events are skipped). On the first run (no watermark) it walks only the most-recent --backfill-pages pages (default 3) so recent activity is swept without replaying the entire catalog. Leaves are deduplicated by package id within a run, keeping the latest version.

For every candidate leaf it fetches the package detail document and gathers:

• Package metadata — id, version, projectUrl, licenseExpression / licenseUrl, authors, description, repository{url,type}, and packageContent (the .nupkg URL).
• Install / build execution vectors — the published .nupkg (a ZIP) is downloaded (bounded) and its execution vectors extracted: legacy PowerShell install hooks (install.ps1 / init.ps1), MSBuild .targets / .props (which can run inline tasks at restore/build time), the .nuspec manifest, and a bounded sample of .cs source text. The PowerShell and MSBuild content feeds the install-script detectors.
• Upstream GitHub repo (only when the package’s repository/projectUrl is a github.com project) — repository, contributors, and license, persisted GitHub-first in its own committed transaction.

Records produced

Gates before minting

Three independent gates sit between a detection and an advisory:

• LLM false-positive gate (internal/aimalgate, main.go:325-338) — when PIX_INFERENCE_ENABLED + an AI-Gateway token are configured, the evidence set and the scanned source are put to the model before minting. Only a confident legitimate verdict drops the detection; any gateway or parse error fails open and keeps the verdict, so the gate can only remove false positives, never suppress a real detection on infrastructure failure. Cleared packages are counted as aiCleared in the run summary.
• Human curation gate (curationgate.PackageCleared, main.go:361) — a package/version a reviewer has already cleared is never re-minted, even when the detectors fire again. The PackageVersion row is still written.
• Legitimate-maintainer guard (legitmaintainer.SkipActor, main.go:431) — a well-known upstream identity is never branded a threat actor, which is what stops an ownership-reputation signal cascading onto every package a legitimate author publishes.

Detection

Reuses the shared malscan-engine detect engine. The PowerShell install hooks and MSBuild .targets/.props content are scanned by the install-script detectors; the full extracted sources (.nuspec + scripts + sampled .cs) by the general / shell / source-url detectors. NuGet/.NET-specific evidence rules include PowerShell download-and-execute (IEX (New-Object Net.WebClient).DownloadString(...)), Invoke-Expression of decoded payloads, -EncodedCommand / -enc base64 PowerShell, and MSBuild inline-task code execution (CWE-94); credential / environment exfiltration (CWE-200); and obfuscation signatures (context). Findings carry one of three classes: evidence (a factual malicious behaviour — mints the advisory on its own), trigger (a weak corroborating signal such as a high-entropy embedded payload or a supply-chain ownership/identity change — never mints alone), and context (reputation/risk, recorded as PackageVersion.metadata only). detect.CombinedVerdict mints on any evidence, a known-bad package owner, a high-entropy payload combined with an ownership/identity change, or two independent identity-change families changing together. The full finding-class model and the per-ecosystem capability config (22 capabilities) live in Malware Detection.

Known-bad IOC matching — the package’s declared source is also matched against the public per-ecosystem known-bad STIX feed (domains, IPs, URLs), loaded once per run and matched in memory. A hit is evidence (CWE-506) folded into the finding set before the combination gate, and is recorded as a MalwareIoc row whose references carry the file/line and STIX provenance.

Resume

A watermark is stored in BulkDataDumpTracker under source nuget-catalog (the sha256 column holds the RFC3339 commitTimeStamp of the most-recent catalog page processed). The first run (no tracker) walks the most-recent --backfill-pages pages of the catalog; subsequent runs resume after the saved watermark.

Flags

S3 Persistence

• Archive path: nuget/files/nuget-{package}-{version}/{package}/sources.txt ✓
• Quarantine path: failed-feeds/nuget-json-processor/{YYYY-MM-DD}/{reason}/{package}/sources.txt ✓
• Failure reasons emitted: store-error

The unit of work is a package version, not a file: the concatenated PowerShell install hooks, MSBuild targets and package sources that were actually scanned are archived after a successful store (main.go:419) and quarantined when the store transaction fails (main.go:413). The archive key slot holds the nuget-{package}-{version} coordinate rather than a content hash, so re-scanning the same version overwrites in place. Skipped when S3_BUCKET_NAME is unset (local dev).