cran-json-processor

Status: Live Source: CRAN via the crandb mirror Type: json (crandb recent-releases feed + per-package DESCRIPTION JSON + source tarball fetch) Source slug: cran Schedule: Runs hourly (at minute 10) (cron(10 * * * ? *)).

Overview

Continuously watches CRAN (the Comprehensive R Archive Network) for new and updated R packages and flags malicious publishes. Each run fetches the crandb recent-releases feed (crandb.r-pkg.org/-/pkgreleases?limit=N, newest-first after sorting), so only packages released since the previous run are processed. Duplicate package names within a run are skipped. When the recent feed is unavailable, the processor falls back to crandb.r-pkg.org/-/latest (the latest version of every package), bounded by --limit.

🛑 Broken — the feed request returns the OLDEST releases, so this processor is permanently stuck in 1991. crandb.r-pkg.org/-/pkgreleases?limit=N is a CouchDB view and defaults to ascending order: ?limit=200 returns the 200 earliest release events in CRAN history (verified 2026-08-06: first 1991-11-26, last 2000-07-02). internal/cran/api.go:124 sorts descending, but only within that ancient window, so walkReleases never sees a modern release.

The consequences are visible in production:

• The saved watermark is 2002-07-03T12:19:53+00:00 — the newest date in the --force window (limit=1000). Every later run fetches the same pre-2002 events, finds them all <= watermark, skips every one, and returns maxDate == watermark, so the tracker is never bumped. BulkDataDumpTracker.cran-recent.lastProcessedAt has not moved since 2026-06-17 despite an hourly schedule.
• Only 191 PackageVersion rows and 10 CVEMetadata rows exist, all written in one ~2-minute burst on 2026-06-17.
• All 10 minted advisories are ancient, well-established, legitimate CRAN packages — sm, pastecs, e1071, sn, RMySQL, coda, relimp, quantreg, Rmpi, car — i.e. a 100% false-positive rate on a corpus of 1990s R code whose configure scripts legitimately call system().

Fix: append &descending=true to the pkgreleases request (internal/cran/api.go:100). Verified: ?limit=5&descending=true returns 2026-08-06T06:00:08+00:00 first. The watermark must then be reset (it is 24 years in the past) and the 10 false-positive advisories retracted.

For every candidate package it fetches the DESCRIPTION document (crandb.r-pkg.org/<name>) and gathers:

• Package metadata — version, homepage (first URL from the URL field), repository (github URL from BugReports / URL), author, maintainer (name + email parsed from the Maintainer "Name <email>" field), license, and Date/Publication.
• Install-time scripts — the source tarball’s configure / configure.win / cleanup shell scripts and src/Makevars, which execute at R CMD INSTALL time, the classic CRAN supply-chain execution vector. These feed the install-script detectors.
• R sources — DESCRIPTION + .R files + NAMESPACE (bounded ~512 KB) concatenated for deep static detection, including .onLoad / .onAttach hooks that run at library-load.
• Upstream GitHub repo (only when the package’s URL/BugReports is a github.com project) — repository, contributors, and license, persisted GitHub-first in its own committed transaction.

The source tarball is downloaded from cran.r-project.org/src/contrib/<name>_<version>.tar.gz. Archived/404 tarballs are skipped best-effort — the PackageVersion is still recorded from metadata.

Records produced

Detection

Reuses the shared malscan-engine detect engine. Install-time script bodies (configure / configure.win / cleanup / src/Makevars) are scanned by the install-script detectors; the full extracted sources (DESCRIPTION + .R files + NAMESPACE) by the general / shell / source-url detectors. R/CRAN-specific evidence rules include system() / system2() / shell() calls in install scripts or .R code, download.file() followed by source() / eval(), base64enc-decoded payloads passed to eval(parse(text=)) (CWE-94), and .onLoad/.onAttach hooks that invoke system() (CWE-506). Findings carry one of three classes: evidence (a factual malicious behaviour — mints the advisory on its own), trigger (a weak corroborating signal such as a high-entropy embedded payload or a supply-chain ownership/identity change — never mints alone), and context (reputation/risk, recorded as PackageVersion.metadata only). detect.CombinedVerdict mints on any evidence, a known-bad package owner, a high-entropy payload combined with an ownership/identity change, or two independent identity-change families changing together. The full finding-class model and the per-ecosystem capability config (22 capabilities) live in Malware Detection.

Known-bad IOC matching — the package’s declared source is also matched against the public per-ecosystem known-bad STIX feed (domains, IPs, URLs), loaded once per run and matched in memory. A hit is evidence (CWE-506) folded into the finding set before the combination gate, and is recorded as a MalwareIoc row whose references carry the file/line and STIX provenance.

Resume

A release-date watermark (RFC3339) is stored in BulkDataDumpTracker under source cran-recent (the sha256 column holds the last processed release date). The first run (no tracker) sweeps the recent feed in full; subsequent runs process only releases newer than the saved watermark. --force reprocesses the recent window regardless of the watermark; --limit bounds packages per run.

S3 Persistence

• Archive path: cran/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/cran-json-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.