packagist-json-processor

Status: Live Source: Packagist (changes feed + repo.packagist.org p2 metadata) Type: json (changes feed + per-package p2 JSON metadata + dist ZIP fetch) Source slug: packagist Schedule: Runs hourly (at minute 20) (cron(20 * * * ? *)).

Overview

Continuously watches Packagist for new and updated PHP/Composer packages and flags malicious publishes. Each run walks the changes feed (packagist.org/metadata/changes.json?since=<cursor>) from a stored cursor watermark, so only packages changed since the previous run are processed. The cursor is a monotonic microsecond timestamp Packagist returns as timestamp; each response’s timestamp is the next cursor. Action time values are Unix seconds. delete actions and duplicate package ids within a run are skipped.

For every candidate package it fetches the per-package metadata document (repo.packagist.org/p2/<vendor>/<name>.json), resolves the latest non-dev version, and gathers:

Package metadata — version, homepage, source repository, license, authors, and the dist (ZIP) URL.
composer.json lifecycle scripts — the scripts block (post-install-cmd / post-update-cmd / post-autoload-dump etc.), the classic Composer supply-chain execution vector. These feed the install-script detectors.
Dist ZIP sources — the published dist ZIP is downloaded (bounded) and its composer.json + .php sources concatenated for deep static detection (eval(base64_decode(...)), system/exec/shell_exec/passthru of decoded payloads, assert(/create_function dynamic exec, gzinflate/ str_rot13 obfuscation).
Upstream GitHub repo (only when the package’s source/homepage is a github.com project) — repository, contributors, and license, persisted GitHub-first in its own committed transaction.

Records produced

Condition	Records
Every scanned package	`PackageVersion` (ecosystem `packagist`); `updatedAt` = metadata/changes time; version/authors/dist/scripts + non-minting context signals in `metadata` JSON
Upstream is github.com	`GitHubRepository` + `GitHubRepoContributor` + license fields
Malicious (≥1 evidence detection)	`CVEMetadata` (`source="packagist"`, `GCVE-110-COMPOSER-YYYY-NNNNNN`, `isMaliciousPackage=true`), one `CVEDescription` per detection, `CVEProblemType` (CWE-506 + CWE-94/CWE-200 specifics), `CVEAffected` (vendor `packagist`, all versions), `CVEMetadataReferences` (packagist page + repo + homepage), `PackageVersionCVE`, `GcveIssuance`
Malicious + actor resolved	`ThreatActor` (composer.json authors + embedded contact emails) + `MalwareThreatActor` edges + `MalwareAttribution` (`attributed`, claimed/victim upstream GitHub). Candidates rejected by `internal/legitmaintainer` (known-good maintainer allowlist) are skipped so a legitimate Composer author is never branded an actor
Malicious	`MalwareIoc` rows (exfil endpoints, IPs, domains, URLs, emails, install-commands, file hashes), plus ownership-change IOCs for any hijack/ownership trigger on the verdict path and one IOC per known-bad STIX match
Post-batch, every run	`malwareactor.PostPass` attributes still-unattributed `source='packagist'` malware records via `internal/actorintel`, writing `ThreatActor`, `ThreatActorKey`, `MalwareThreatActor` and `MalwareAttribution`. Records that already carry a `MalwareAttribution` row are skipped, so steady-state runs do ~0 work. Bounded by the same soft deadline as the walk

Detection

Reuses the shared malscan-engine detect engine. The composer.json scripts block is scanned by the install-script detectors; the full concatenated composer.json + PHP sources by the general / shell / source-url detectors. PHP/Composer-specific evidence rules include eval(base64_decode(...)) / assert($…) / create_function dynamic code execution (CWE-94), a system/exec/shell_exec/passthru/ proc_open/popen call whose argument is a variable (PHP-SHELL-EXEC-VAR, CWE-94 — note this matches any variable argument, not only a decoded payload), gzinflate / str_rot13 obfuscation chains (PHP-OBFUSCATION-CHAIN, CWE-506), and post-install-cmd hooks piping curl/wget to a shell (CWE-94). Findings carry one of three classes: evidence (a factual malicious behaviour — mints the advisory on its own), trigger (a weak corroborating signal such as a high-entropy embedded payload or a supply-chain ownership/identity change — never mints alone), and context (reputation/risk, recorded as PackageVersion.metadata only). detect.CombinedVerdict mints on any evidence, a known-bad package owner, a high-entropy payload combined with an ownership/identity change, or two independent identity-change families changing together. The full finding-class model and the per-ecosystem capability config (22 capabilities) live in Malware Detection.

Known-bad IOC matching — the package’s declared source is also matched against the public per-ecosystem known-bad STIX feed (domains, IPs, URLs), loaded once per run and matched in memory. A hit is evidence (CWE-506) folded into the finding set before the combination gate, and is recorded as a MalwareIoc row whose references carry the file/line and STIX provenance.

Known false-positive class — PHP-SHELL-EXEC-VAR is implemented as the regex (system|exec|shell_exec|passthru|proc_open|popen)\s*\(\s*\$ with no left-hand word boundary (detect/data/patterns.toml:2104), so it also matches the substring exec inside ordinary PHP calls such as curl_exec($ch) and $pdo->exec($sql), and matches inside doc comments. At 45 points it clears the engine’s evidence threshold of 40 and therefore mints on its own. It is the dominant rule in production packagist mints, so any consumer of source='packagist' must treat a mint whose only evidence is PHP-SHELL-EXEC-VAR as unconfirmed until the pattern is anchored.

Suppression gates

Two independent gates can drop a detection before it mints. Neither can ever add one — both fail open:

Gate	Mechanism
Customer curation (`internal/curationgate`)	`MalwareCurationConsensus` rows marked `fp_cleared` for this `(ecosystem, package, version)` clear the verdict inside the store transaction
LLM false-positive gate (`internal/aimalgate`)	Before minting, the finding set + source is put to the model configured by `GO_MALWARE_GATE_MODEL` (`@cf/zai-org/glm-5.2` in ECS). Only a confident “benign” verdict drops the detection; a gateway or parse error keeps it. Cleared detections are counted as `aiCleared` in the run stats

Resume

A cursor watermark is stored in BulkDataDumpTracker under source packagist-changes (the sha256 column holds the last processed changes timestamp cursor). The first run (no tracker) starts a configurable microsecond window before the changes head so recent activity is swept; subsequent runs resume from the saved cursor.

S3 Persistence

Archive path: packagist/files/{sha256}/{filename} ✓ — the concatenated scanned source set is archived as {package}/sources.txt after a successful store (cmd/packagist-json-processor/main.go:393)
Quarantine path: failed-feeds/packagist-json-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
Failure reasons emitted: store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Packagist changes feed] --> PROC[packagist-json-processor] PROC -->|success| ARCHIVE[("S3: packagist/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/packagist-json-processor/{date}/store-error/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.