oss-malware-git-processor

Status: Live Source: ossf/malicious-packages (git) Type: git (runtime clone/pull via pure-Go go-git — no baked data stage) Source slug: oss-malicious-packages Schedule: Runs daily at 03:00 UTC (cron(0 3 * * ? *)). CPU 512 / memory 2048 MB / expected_duration_minutes = 60.

Overview

OpenSSF’s malicious-packages repository is the largest curated public corpus of confirmed malicious registry publishes (npm, PyPI, RubyGems, crates.io, Packagist, NuGet, Go). It is the reference set the package firewall blocks against and the baseline every Vulnetix-minted GCVE-110-* malware advisory is deduplicated against — without it the firewall’s known-bad list collapses to whatever our own scanners have independently found.

Each run:

  1. Refreshes the repo. processor.PullOrClone clones (or pulls) /data/malicious-packages at depth 1. The final image stage is scratch and ships no baked advisory data, so the clone happens at runtime through go-git. --no-pull reads the existing HEAD instead.
  2. Short-circuits on an unchanged HEAD against BulkDataDumpTracker.sha256 for source oss_malicious_packages; a match without --force reports task.nowork and exits 0.
  3. Narrows to changed files via processor.DetectChangedFiles (3-day window) when a tracker exists; first run / --force / un-diffable history → full scan.
  4. Walks osv/ for MAL-*.json files and stores them in transactions of --batch-size (default 200), one SAVEPOINT per file.
  5. Attributes actors — after the walk, malwareactor.PostPass runs for source oss-malicious-packages.

The mapper override is osv.MapMaliciousAdvisory: unlike the default OSV mapper it keeps the MAL-* ID as the row key (CVE aliases are preserved as CVEAlias edges rather than absorbing the record into a CVE row) and forces isMaliciousPackage = true. CWE-506 (Embedded Malicious Code) is added when the upstream cwe_ids list does not already carry it. Files whose advisory has no usable ID are counted unchanged and skipped.

Records produced

ConditionRecords
Every stored advisoryCVEMetadata (source="oss-malicious-packages", cveId = MAL-YYYY-NNNN, isMaliciousPackage=true, sourceFileHash = file SHA1), CVEDescription, CVEMetadataReferences (referenceSource="OSS Malicious Packages"), CVEProblemType (upstream cwe_ids + CWE-506), CVEAffected + CVEAffectedVersion
Advisory carries a CVSS vectorCVEMetric per severity entry
Advisory carries descriptionsone derived CVEMetric (containerType="vulnetix", cvssV4_0)
AlwaysCVEAlias edges via db.InsertAliases (CVE / GHSA aliases, related[], upstream[], plus same-cveId cross-source edges)
Every affected packageDependency, DependencyRegistry, PackageVersion, PackageVersionCVE via db.EnrichAffectedWithDependency — this is what makes an OpenSSF record blockable by package name + version
Post-pass, per attributed recordThreatActor, ThreatActorKey, MalwareThreatActor edges, MalwareAttribution status
Clean runBulkDataDumpTracker row oss_malicious_packages

Threat-actor attribution

malwareactor.PostPass(source="oss-malicious-packages") runs once per execution, after the walk and before the tracker write. It is first-time-only — a (cveId, source) that already has a MalwareAttribution row is never re-enriched, so steady-state runs do almost no work — and bounded to MALWARE_ACTOR_BATCH records (default 150). MALWARE_ACTORS=false disables it.

The shared internal/actorintel engine resolves the malware author from package and repo identity plus registry / GitHub / Docker Hub lookups:

AspectBehaviour
Attribution tablesMalwareThreatActor, MalwareAttribution, shared ThreatActor + ThreatActorKey (never the OSM-only OsmThreat* tables)
Attribution basisrepo owner / container namespace / registry maintainer / Go-module repo / commit author
Hijack handlingcompromised-account advisories mark the maintainer hijack-victim-excluded — the victim is not attributed
Impersonation trapfor non-Go packages a declared repository is usually the dependency-confusion or typosquat target, so it is captured as claimedRepo* and never attributed as an actor
GitHub key harvestpublic SSH-auth, SSH-signing and GPG keys → ThreatActorKey; a reused fingerprint links operators across accounts
Rate limitsGITHUB_PAT / GITHUB_TOKEN; without one GitHub is capped at 60 req/hr and only bare handles resolve

The historical backlog is cleared by the one-time cmd/malware-actor-backfill (just go-malware-actor-backfill), which selects isMaliciousPackage=true AND source<>'osm' across every malware source.

Resume

  • Repo-level — the oss_malicious_packages tracker SHA.
  • Per-file — db.LoadProcessedHashes("oss-malicious-packages") skip set (~230k keys on production; the loader has a 5-minute budget).

The tracker write is skipped when the run context was cancelled mid-way, so a truncated run re-scans next time instead of skipping on a matching SHA.

AI enrichment

The pipeline is wired to aienrich; in ECS each stored record gets the shared post-batch passes (affected routines, ATT&CK, CWE inference, TreeSitter). Local just runs are inference-free unless AIENRICH=true.

Failure modes

SymptomCause
repository unchanged, skipping processingtracker SHA equals HEAD
failed to load resume hashes, processing all filesread-replica scan of ~230k rows exceeded 5 minutes; the run re-upserts everything
soft deadline reached, stopping earlyEXPECTED_DURATION_MINUTES − 10 elapsed; tracker is skipped when the context was cancelled
GITHUB_PAT/GITHUB_TOKEN not setactor attribution degrades to unauthenticated GitHub (60 req/hr)

Flags

FlagDefaultMeaning
--forcefalseIgnore the tracker SHA and the per-file hash skip set
--batch-size200Files per transaction
--repo/data/malicious-packagesClone location
--data-dir(repo + /osv)Override the advisory directory
--no-pullfalseUse the existing clone’s HEAD instead of pulling

Local run: just go-oss-malware-git-backfill (append prod for production).

S3 Persistence

  • Archive path: oss-malicious-packages/files/{sha256}/{filename} ✓
  • Quarantine path: failed-feeds/oss-malware-git-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
  • Failure reasons emitted: parse-error, store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[oss-malware-git-processor] PROC -->|success| ARCHIVE[("S3: oss-malicious-packages/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/oss-malware-git-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.