rubygems-json-processor

Status: Live Source: RubyGems.org (activity/just_updated.json + activity/latest.json) Type: json (activity feeds + per-gem JSON API + .gem archive fetch) Source slug: rubygems Schedule: Runs hourly (at minute 40) (cron(40 * * * ? *)).

Overview

Continuously watches RubyGems.org for new and updated gems and flags malicious publishes. Each run fetches both activity feeds (activity/just_updated.json and activity/latest.json), deduplicates by gem name, and — because RubyGems exposes no monotonic serial — filters by a stored UNIX-seconds watermark (the max version_created_at processed on the previous run), so only gems published since the previous run are processed. The first run (no watermark) and --force runs process every gem from both feeds.

For every candidate gem it normalises the activity record, optionally enriches missing fields from the per-gem detail document (api/v1/gems/<name>.json), and gathers:

Gem metadata — version, homepage, source code URI, project URI, authors, first license, dependencies, and version_created_at.
Gemspec manifest — the gzipped YAML gemspec (metadata.gz inside the .gem archive). The gemspec declares native extensions (compiled at gem install time via extconf.rb — the classic RubyGems supply-chain execution vector), post_install_message, and executables. This feeds the install-script detectors.
Gem sources — the published .gem is an uncompressed tar containing metadata.gz, data.tar.gz, and checksums.yaml.gz. The data.tar.gz is gunzipped and its .rb/.c/.h files plus any ext/** build scripts concatenated (bounded ~512 KB) for deep static detection (eval(...), system(...)/backtick exec, Kernel.exec, base64-decode-and-eval, Net::HTTP exfiltration).
Upstream GitHub repo (only when the gem’s source_code_uri/homepage/ project_uri is a github.com project) — repository, contributors, and license, persisted GitHub-first in its own committed transaction.

Records produced

Condition	Records
Every scanned gem	`PackageVersion` (ecosystem `rubygems`); `updatedAt` = `version_created_at`; version/authors/license/dependencies + non-minting context signals in `metadata` JSON
Upstream is github.com	`GitHubRepository` + `GitHubRepoContributor` + license fields
Malicious (≥1 evidence detection)	`CVEMetadata` (`source="rubygems"`, `GCVE-110-GEM-YYYY-NNNNNN`, `isMaliciousPackage=true`), one `CVEDescription` per detection, `CVEProblemType` (CWE-506 + specifics), `CVEAffected` (vendor `rubygems`, all versions), `CVEMetadataReferences` (gem page + repo + homepage), `PackageVersionCVE`, `GcveIssuance`
Malicious + actor resolved	`ThreatActor` (gem authors + embedded contact emails) + `MalwareThreatActor` edges + `MalwareAttribution` (`attributed`, claimed/victim upstream GitHub). Candidates rejected by `internal/legitmaintainer` (known-good maintainer allowlist) are skipped so a legitimate gem author is never branded an actor
Malicious	`MalwareIoc` rows (exfil endpoints, IPs, domains, URLs, emails, install-commands, file hashes), plus ownership-change IOCs for any hijack/ownership trigger on the verdict path and one IOC per known-bad STIX match
Post-batch, every run	`malwareactor.PostPass` attributes still-unattributed `source='rubygems'` malware records via `internal/actorintel`, writing `ThreatActor`, `ThreatActorKey`, `MalwareThreatActor` and `MalwareAttribution`. Records that already carry a `MalwareAttribution` row are skipped, so steady-state runs do ~0 work

Detection

Reuses the shared malscan-engine detect engine. The gemspec YAML is scanned by the install-script detectors (native extensions build is the install vector); the concatenated Ruby/C sources by the general / shell / source-url detectors. Ruby-specific evidence rules include eval(...)/Kernel.exec/system(...) and backtick command execution of decoded payloads, base64-decode-and-eval, and Net::HTTP network exfiltration; obfuscator signatures are recorded as context. Findings carry one of three classes: evidence (a factual malicious behaviour — mints the advisory on its own), trigger (a weak corroborating signal such as a high-entropy embedded payload or a supply-chain ownership/identity change — never mints alone), and context (reputation/risk, recorded as PackageVersion.metadata only). detect.CombinedVerdict mints on any evidence, a known-bad package owner, a high-entropy payload combined with an ownership/identity change, or two independent identity-change families changing together. The full finding-class model and the per-ecosystem capability config (22 capabilities) live in Malware Detection.

Known-bad IOC matching — the package’s declared source is also matched against the public per-ecosystem known-bad STIX feed (domains, IPs, URLs), loaded once per run and matched in memory. A hit is evidence (CWE-506) folded into the finding set before the combination gate, and is recorded as a MalwareIoc row whose references carry the file/line and STIX provenance.

Known false-positive classes — the concatenated Ruby/C source is matched as plain text, so rules also fire on comments and on library functionality:

P-AT-JOB (“at job scheduling”) matches the English word at followed by a clock time, which minted aws-sdk-ssm (GCVE-110-GEM-2026-000895) off the comment # window to run on the third Tuesday of every month at 11:30 PM. and aws-sdk-redshift (GCVE-110-GEM-2026-000896) off # … if the request is sent at 18:00 and you.
P-BASE64 / P-BASE32 mint gems whose purpose is encoding — e.g. tins (GCVE-110-GEM-2026-000897) — and record an empty matchedLine, so the evidence is not auditable. 172 of the 341 rubygems mints in production contain at least one finding with an empty matchedLine.
IOC-STIX-MATCH inherits the shared feed problem described on the pypi-json-processor page: the feed is generated from this database’s own IOC rows, so a mis-extracted indicator mints every gem that mentions it.

Suppression gates

Two independent gates can drop a detection before it mints. Neither can ever add one — both fail open:

Gate	Mechanism
Customer curation (`internal/curationgate`)	`MalwareCurationConsensus` rows marked `fp_cleared` for this `(ecosystem, gem, version)` clear the verdict inside the store transaction
LLM false-positive gate (`internal/aimalgate`)	Before minting, the finding set + source is put to the model configured by `GO_MALWARE_GATE_MODEL` (`@cf/zai-org/glm-5.2` in ECS). Only a confident “benign” verdict drops the detection; a gateway or parse error keeps it. Cleared detections are counted as `aiCleared` in the run stats

Resume

A UNIX-seconds watermark is stored in BulkDataDumpTracker under source rubygems-activity (the sha256 column holds the max version_created_at processed). The first run (no watermark) and --force runs process every gem in both feeds; subsequent runs only process gems published after the saved watermark.

S3 Persistence

Archive path: rubygems/files/{sha256}/{filename} ✓ — the concatenated Ruby/C source set is archived as {gem}/sources.txt after a successful store (cmd/rubygems-json-processor/main.go:371)
Quarantine path: failed-feeds/rubygems-json-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
Failure reasons emitted: store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[RubyGems activity feeds] --> PROC[rubygems-json-processor] PROC -->|success| ARCHIVE[("S3: rubygems/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/rubygems-json-processor/{date}/store-error/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.