cargo-json-processor

Status: Live Source: crates.io REST API Type: json (recent-updates feed + per-crate JSON detail + .crate source fetch) Source slug: cargo Schedule: Runs hourly (at minute 45) (cron(45 * * * ? *)).

Overview

Continuously watches the crates.io registry for new and updated crates and flags malicious publishes. Each run pages the recent-updates feed (crates.io/api/v1/crates?sort=recent-updates&per_page=100&page=N) from a stored updated_at watermark. The feed is sorted by update time descending, so the walk stops at the first crate at or below the saved watermark (unless --force), processing only crates changed since the previous run. Yanked / 404 crates and duplicate crate ids within a run are skipped.

crates.io requires a descriptive User-Agent (it rejects anonymous/identifying crawlers) and rate-limits at roughly one request per second. Every request uses the registered Vulnetix crawler User-Agent and crate-detail / download requests are spaced with a polite ~1.1s delay.

For every candidate crate it fetches the detail document (crates.io/api/v1/crates/<name>), resolves the newest version, and gathers:

Crate metadata — version, homepage, repository, license, description, and the newest version’s dl_path.
Build script (build.rs) — the prime Rust supply-chain execution vector: build.rs EXECUTES at cargo build time. Its body feeds the install / build-time detectors.
Crate source — the published .crate archive (a gzip’d tar) is downloaded (bounded) and its Cargo.toml, build.rs, and src/**.rs sources concatenated for deep static detection (Command::new / std::process in build.rs, download-and-exec, include_bytes! + decode, environment exfiltration).
Upstream GitHub repo (only when the crate’s repository/homepage is a github.com project) — repository, contributors, and license, persisted GitHub-first in its own committed transaction.

Records produced

Condition	Records
Every scanned crate	`PackageVersion` (ecosystem `cargo`); `updatedAt` = crate `updated_at`; version/license/repository/description + non-minting context signals in `metadata` JSON
Upstream is github.com	`GitHubRepository` + `GitHubRepoContributor` + license fields
Malicious (≥1 evidence detection)	`CVEMetadata` (`source="cargo"`, `GCVE-110-CARGO-YYYY-NNNNNN`, `isMaliciousPackage=true`), one `CVEDescription` per detection, `CVEProblemType` (CWE-506 + CWE-94/CWE-200 specifics), `CVEAffected` (vendor `cargo`, all versions), `CVEMetadataReferences` (crates.io page + repo + homepage), `PackageVersionCVE`, `GcveIssuance`
Malicious + actor resolved	`ThreatActor` (declared author emails + embedded contact emails) + `MalwareThreatActor` edges + `MalwareAttribution` (`attributed`, claimed/victim upstream GitHub)
Post-batch, first time only	`malwareactor.PostPass` attributes up to `MALWARE_ACTOR_BATCH` (default 150) still-unattributed `source='cargo'` malware records via `internal/actorintel`, writing `ThreatActor`, `ThreatActorKey`, `MalwareThreatActor` and `MalwareAttribution`. Records that already carry a `MalwareAttribution` row are skipped, so steady-state runs do ~0 work. Disabled with `MALWARE_ACTORS=false`; GitHub lookups use `GITHUB_PAT`/`GITHUB_TOKEN`
Malicious	`MalwareIoc` rows (exfil endpoints, IPs, domains, URLs, emails, install-commands, file hashes)

Detection

Reuses the shared malscan-engine detect engine. The build.rs body is scanned by the install/build-time detectors; the full extracted source set (Cargo.toml + build.rs + src/**.rs) by the general / shell / source-url detectors. Rust/cargo-specific evidence rules include build.rs running Command::new / std::process (arbitrary command execution at build time), download-and-exec in build.rs, include_bytes! + runtime decode of an embedded blob (CWE-94), and environment exfiltration via std::env::var piped to a network sink (CWE-200). Findings carry one of three classes: evidence (a factual malicious behaviour — mints the advisory on its own), trigger (a weak corroborating signal such as a high-entropy embedded payload or a supply-chain ownership/identity change — never mints alone), and context (reputation/risk, recorded as PackageVersion.metadata only). detect.CombinedVerdict mints on any evidence, a known-bad package owner, a high-entropy payload combined with an ownership/identity change, or two independent identity-change families changing together. The full finding-class model and the per-ecosystem capability config (22 capabilities) live in Malware Detection.

Known-bad IOC matching — the package’s declared source is also matched against the public per-ecosystem known-bad STIX feed (domains, IPs, URLs), loaded once per run and matched in memory. A hit is evidence (CWE-506) folded into the finding set before the combination gate, and is recorded as a MalwareIoc row whose references carry the file/line and STIX provenance.

Resume

An updated_at watermark (unix seconds of the newest crate seen) is stored in BulkDataDumpTracker under source cargo-recent (the sha256 column holds the watermark). The first run (no tracker) walks from the head of the feed; subsequent runs stop at the first crate at or below the saved watermark. --force ignores the watermark and walks the full feed.

S3 Persistence

Archive path: cargo/files/{sha256}/{filename} ✓
Quarantine path: failed-feeds/cargo-json-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
Failure reasons emitted: store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[cargo-json-processor] PROC -->|success| ARCHIVE[("S3: cargo/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/cargo-json-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.