pypi-json-processor
Status: Live Source: PyPI XML-RPC changelog + JSON API + sdist Type:
json(XML-RPC changelog serial cursor + per-package JSON API + sdist fetch) Source slug:pypiSchedule: every 30 minutes (cron(0/30 * * * ? *))
Overview
Continuously watches PyPI for new and updated releases and flags malicious
publishes. Each run calls the XML-RPC changelog_since_serial(<serial>) method
from a stored serial watermark, so only packages changed since the previous run
are processed (deduplicated to the latest release per package).
For every candidate it fetches the JSON API document
(pypi.org/pypi/<name>/json) and gathers version, project URLs, author /
maintainer (name + email), license, requirements, and the sdist download URL +
sha256. The sdist is downloaded (bounded) and its setup.py (which executes at
pip install time — the prime PyPI supply-chain vector), setup.cfg,
pyproject.toml, PKG-INFO, and .py sources are concatenated for static
detection. Records are enriched with conda-forge presence (loaded once per
run from conda.anaconda.org/conda-forge/channeldata.json).
The declared upstream GitHub project (from project_urls/homepage) is resolved
and persisted GitHub-first when present.
Records produced
| Condition | Records |
|---|---|
| Every scanned package | PackageVersion (ecosystem pypi); updatedAt = sdist upload time; version/author/maintainer/sdist + condaForge flag + context signals in metadata JSON |
| Upstream is github.com | GitHubRepository + GitHubRepoContributor + license fields |
| Malicious (≥1 evidence detection) | CVEMetadata (source="pypi", GCVE-110-PYPI-YYYY-NNNNNN, isMaliciousPackage=true), CVEDescription per detection, CVEProblemType (CWE-506 + CWE-94/CWE-200), CVEAffected (vendor pypi), CVEMetadataReferences, PackageVersionCVE, GcveIssuance |
| Malicious + actor resolved | ThreatActor (author/maintainer + embedded emails) + MalwareThreatActor + MalwareAttribution (claimed/victim upstream). Candidates rejected by internal/legitmaintainer (known-good maintainer allowlist) are skipped so a legitimate PyPI maintainer is never branded an actor |
| Malicious | MalwareIoc rows, plus ownership-change IOCs for any hijack/ownership trigger on the verdict path and one IOC per known-bad STIX match |
| Post-batch, every run | malwareactor.PostPass attributes still-unattributed source='pypi' malware records via internal/actorintel, writing ThreatActor, ThreatActorKey, MalwareThreatActor and MalwareAttribution. Records that already carry a MalwareAttribution row are skipped, so steady-state runs do ~0 work (observed in ECS: processed=0 attributed=0) |
Shared source slug.
source='pypi'is also written bypypi-git-processor(the PyPA advisory-database ingest). Anything measuring this processor must filter onisMaliciousPackage = true/ theGCVE-110-PYPI-id prefix: of the 12,256source='pypi'rows in production, 7,989 are this processor’s malware mints and 4,267 belong to the advisory ingest.
Detection
Reuses the shared malscan-engine detect engine. setup.py is scanned by the
install-script detectors; concatenated sources by the general / shell /
source-url detectors. Python-specific evidence rules include exec/eval of
base64/hex/zlib-decoded payloads, exec(compile(...)), __import__ of decoded
names (CWE-94); subprocess/os.system and network downloads at install time;
and env/host recon-then-network exfiltration (CWE-200).
Findings carry one of three classes: evidence (a factual malicious behaviour
— mints the advisory on its own), trigger (a weak corroborating signal such
as a high-entropy embedded payload or a supply-chain ownership/identity change
— never mints alone), and context (reputation/risk, recorded as
PackageVersion.metadata only). detect.CombinedVerdict mints on any
evidence, a known-bad package owner, a high-entropy payload combined with an
ownership/identity change, or two independent identity-change families changing
together. The full finding-class model and the per-ecosystem capability config
(22 capabilities) live in Malware Detection.
Known-bad IOC matching — the package’s declared source is also matched
against the public per-ecosystem known-bad
STIX feed (domains, IPs, URLs), loaded
once per run and matched in memory. A hit is evidence (CWE-506) folded into
the finding set before the combination gate, and is recorded as a MalwareIoc
row whose references carry the file/line and STIX provenance.
Known false-positive class — the STIX matcher
(internal/malscan/iocscan.go:73-84) matches over the whole concatenated source
blob, including README/markdown text and comments, and its hits are
ClassEvidence that mint on their own (iocscan.go:88-96). Because the feed
that is matched against is generated from this database’s own MalwareHost /
MalwareIoc rows by
malscan-stix-processor, a badly-extracted IOC
becomes a published indicator and then mints every package that mentions it.
Production examples of indicators in MalwareHost at severity critical:
docs.astral.sh (the uv documentation site), click.group (the Python
@click.group decorator, mis-parsed as a .group domain) and huggingface.co.
Treat a pypi mint whose only evidence is IOC-STIX-MATCH as unconfirmed.
Suppression gates
Two independent gates can drop a detection before it mints. Neither can ever add one — both fail open:
| Gate | Mechanism |
|---|---|
Customer curation (internal/curationgate) | MalwareCurationConsensus rows marked fp_cleared for this (ecosystem, package, version) clear the verdict inside the store transaction |
LLM false-positive gate (internal/aimalgate) | Before minting, the finding set + source is put to the model configured by GO_MALWARE_GATE_MODEL (@cf/zai-org/glm-5.2 in ECS). Only a confident “benign” verdict drops the detection; a gateway or parse error keeps it. Cleared detections are counted as aiCleared in the run stats — this gate is demonstrably active in ECS, e.g. it cleared aether-robotics, metacountregressor and seliky on the 2026-08-06 08:00 run |
Resume
A serial watermark is stored in BulkDataDumpTracker under source
pypi-changelog (the sha256 column holds the last processed changelog serial).
The first run (no tracker) starts a configurable window before the current
serial; subsequent runs resume from the saved serial.
The watermark is the highest serial in the fetched changelog set, computed
before the processing loop (cmd/pypi-json-processor/main.go:220-226), so a run
truncated by --limit or the soft deadline still advances past packages it
never scanned. There is no per-package resume hash to recover them, so those
packages are skipped until they publish again. Observed ECS runs finish inside
the budget (19 min of a 25 min budget for 171 packages on 2026-08-06 08:00), so
the loss is latent rather than routine — but do not pass --limit to a
scheduled run.
S3 Persistence
- Archive path:
pypi/files/{sha256}/{filename}✓ — the concatenated sdist source set is archived as{package}/sources.pyafter a successful store (cmd/pypi-json-processor/main.go:378) - Quarantine path:
failed-feeds/pypi-json-processor/{YYYY-MM-DD}/{reason}/{filename}✓ - Failure reasons emitted:
store-error
Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).
See the S3 Persistence Contract for the full reason taxonomy.