pubdev-json-processor

Status: Live Source: pub.dev (/api/packages + per-package /api/packages/{name}) Type: json (recent-packages JSON listing + per-package JSON API + archive fetch) Source slug: pubdev Schedule: Runs hourly (at minute 5) (cron(5 * * * ? *)).

Overview

Continuously watches pub.dev (the Dart/Flutter package registry) for new and updated packages and flags malicious publishes. Each run walks the recent-packages JSON listing (https://pub.dev/api/packages?page=N, paged by following next_url), newest first, stopping once a page yields nothing newer than the stored publish-timestamp watermark. Duplicate package ids within a run are skipped.

For every candidate package it fetches the per-package document (pub.dev/api/packages/<name>), resolves the latest release, and gathers:

Package metadata — version, homepage (pubspec.homepage), repository (pubspec.repository, falling back to pubspec.issue_tracker when it points at GitHub), description, archive URL, and the latest release’s published RFC3339 timestamp.
pubspec manifest — pubspec.yaml content, fed to the install-script / manifest detectors.
Archive Dart sources — the latest release’s .tar.gz archive is downloaded (bounded) and its pubspec.yaml plus lib/**.dart, bin/**.dart, hook/**.dart (build hooks) sources concatenated for deep static detection. Dart packages can run attacker-controlled code via build hooks (hook/build.dart) and dart run, so the archive sources are the prime pub.dev supply-chain execution vector.
Upstream GitHub repo (only when the package’s repository/homepage is a github.com project) — repository, contributors, and license, persisted GitHub-first in its own committed transaction.

Records produced

Condition	Records
Every scanned package	`PackageVersion` (ecosystem `pub`); `updatedAt` = release `published`; version/homepage/repository/archive + non-minting context signals in `metadata` JSON
Upstream is github.com	`GitHubRepository` + `GitHubRepoContributor` + license fields
Malicious (≥1 evidence detection)	`CVEMetadata` (`source="pubdev"`, `GCVE-110-PUB-YYYY-NNNNNN`, `isMaliciousPackage=true`), one `CVEDescription` per detection, `CVEProblemType` (CWE-506 + CWE-94/CWE-200 specifics), `CVEAffected` (vendor `pubdev`, all versions), `CVEMetadataReferences` (pub.dev page + repo + homepage), `PackageVersionCVE`, `GcveIssuance`
Malicious + actor resolved	`ThreatActor` (declared upstream GitHub claimed/victim + embedded contact emails) + `MalwareThreatActor` edges + `MalwareAttribution` (`attributed`, claimed/victim upstream GitHub). Candidates rejected by `internal/legitmaintainer` (known-good maintainer allowlist) are skipped so a legitimate Dart publisher is never branded an actor
Malicious	`MalwareIoc` rows (exfil endpoints, IPs, domains, URLs, emails, install-commands, file hashes), plus ownership-change IOCs for any hijack/ownership trigger on the verdict path and one IOC per known-bad STIX match
Post-batch, every run	`malwareactor.PostPass` attributes still-unattributed `source='pubdev'` malware records via `internal/actorintel`, writing `ThreatActor`, `ThreatActorKey`, `MalwareThreatActor` and `MalwareAttribution`. Records that already carry a `MalwareAttribution` row are skipped, so steady-state runs do ~0 work. Bounded by the same soft deadline as the walk

Detection

Reuses the shared malscan-engine detect engine. The pubspec.yaml manifest is scanned by the install-script / manifest detectors; the full Dart source block (including hook/build.dart build hooks) by the general / shell / source-url detectors. Dart-specific evidence rules target Process.run/Process.start spawning a shell, dart:io HttpClient download-then-execute, base64-decode-then-eval-style code generation, and build hooks that run shell commands (CWE-94, CWE-506, CWE-200). Findings carry one of three classes: evidence (a factual malicious behaviour — mints the advisory on its own), trigger (a weak corroborating signal such as a high-entropy embedded payload or a supply-chain ownership/identity change — never mints alone), and context (reputation/risk, recorded as PackageVersion.metadata only). detect.CombinedVerdict mints on any evidence, a known-bad package owner, a high-entropy payload combined with an ownership/identity change, or two independent identity-change families changing together. The full finding-class model and the per-ecosystem capability config (22 capabilities) live in Malware Detection.

Known-bad IOC matching — the package’s declared source is also matched against the public per-ecosystem known-bad STIX feed (domains, IPs, URLs), loaded once per run and matched in memory. A hit is evidence (CWE-506) folded into the finding set before the combination gate, and is recorded as a MalwareIoc row whose references carry the file/line and STIX provenance.

Known false-positive class — the STIX matcher (internal/malscan/iocscan.go:73-84) runs MatchText over the whole concatenated source blob, so it matches inside doc comments, test fixtures and user-agent strings, and its hits are ClassEvidence that mint on their own (iocscan.go:88-96). Observed in production: an IP extracted from Chrome/120.0.0.0 Safari/537.36, a package’s own vendor homepage (https://www.coralogix.io/), a documentation link (https://datalust.co/seq) and a https://abc123.ngrok.io placeholder inside a /// doc comment. A pubdev mint whose only evidence is IOC-STIX-MATCH should be treated as unconfirmed until the matcher gains comment/fixture awareness.

Suppression gates

Two independent gates can drop a detection before it mints. Neither can ever add one — both fail open:

Gate	Mechanism
Customer curation (`internal/curationgate`)	`MalwareCurationConsensus` rows marked `fp_cleared` for this `(ecosystem, package, version)` clear the verdict inside the store transaction
LLM false-positive gate (`internal/aimalgate`)	Before minting, the finding set + source is put to the model configured by `GO_MALWARE_GATE_MODEL` (`@cf/zai-org/glm-5.2` in ECS). Only a confident “benign” verdict drops the detection; a gateway or parse error keeps it. Cleared detections are counted as `aiCleared` in the run stats

Resume

A publish-timestamp watermark is stored in BulkDataDumpTracker under source pubdev-recent (the sha256 column holds the max published RFC3339 timestamp of the previous run). The first run (no tracker) and --force sweep the full recent listing; subsequent runs resume from the saved watermark and stop once a page yields no releases newer than it.

S3 Persistence

Archive path: pubdev/files/{sha256}/{filename} ✓ — the concatenated Dart source set is archived as {package}/sources.txt after a successful store (cmd/pubdev-json-processor/main.go:401)
Quarantine path: failed-feeds/pubdev-json-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
Failure reasons emitted: store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[pub.dev recent-packages listing] --> PROC[pubdev-json-processor] PROC -->|success| ARCHIVE[("S3: pubdev/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/pubdev-json-processor/{date}/store-error/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.