enrich-malware-exposure Design

Status: Live Source: deps.dev API v3 (/systems/{system}/packages/{name}) Type: fetch (JSON over HTTPS, no auth, no token) Source slug: none — this enrichment writes no CVEMetadata rows Schedule: Hourly (cron(0 * * * ? *)), 256 CPU / 512 MB, EXPECTED_DURATION_MINUTES=45.

Overview

The OSSF “exposure window” is the interval between the moment a malicious package became installable from its registry and the moment the MAL- advisory describing it was published. It is the single most persuasive number in the malware-trends reporting, and it used to be a constant baked into the website.

This processor makes it a live measurement. For every package named by a malicious advisory it asks deps.dev two questions — when was this first published? and is it still on the registry? — and records the answers on the PackageVersion row the ecosystem processors already own. The summary-processor then derives the window (advisory datePublished − min(PackageVersion.publishedAt)) and the “erased” case (publish after the advisory) as a malware_trends stat; nothing about the window itself is stored here.

Only PackageVersion is touched. No CVEMetadata, no MalwareIoc, no ThreatActor.

Target selection

Malicious-package targets come from a single anti-join (cmd/enrich-malware-exposure-fetch/main.go:68-80):

CVEMetadata (isMaliciousPackage = true)
  JOIN CVEAffected ON (cveId, source)
  JOIN url_eco ON rtrim(collectionURL,'/')     -- 10 registries
 WHERE NOT EXISTS (PackageVersion row for that
                   (ecosystem, packageName) with registryStatus IS NOT NULL)
 LIMIT --batch

PackageVersion.registryStatus IS NOT NULL is the resume marker, so each package is checked exactly once. A transient deps.dev failure leaves the target unchecked and it is retried on the next run. The url_eco CTE (main.go:43-53) maps CVEAffected.collectionURL → (deps.dev system, lowercase ecosystem) for npm, pypi, go, maven, cargo, rubygems, nuget, packagist, pub and hex; it is deliberately kept identical to internal/enrichment’s collectionURLToSystem and to the summary-processor exposure query so the PackageVersion join key never drifts.

The batch query runs on its own context.Background() 5-minute timeout so a near-expired task context cannot cancel the scan mid-flight.

Records produced

All three paths go through db.SetPackageVersionRegistryTx, which INSERT … ON CONFLICT (ecosystem, packageName, version) DO UPDATE and COALESCEs publishedAt so a later, less-informed pass can never blank a publish date an earlier one established. discoveredFrom is deps.dev; discoveredAt / lastVerifiedAt / updatedAt are time.Now().UnixMilli().

Lock contention

Every write in this processor targets PackageVersion, whose rows the package-firewall service locks for 13–20 s at a time during its signal refresh. The write pool’s 5 s lock_timeout makes those collisions surface as SQLSTATE 55P03 rather than a stall, so each transaction is wrapped in db.WithTxRetry (main.go:110-137) — the transaction is idempotent, so a retry is always safe. See .repo/PROCESSOR_STABILIZATION.md ISSUE-1B / OBS-B for why the durable fix belongs in package-firewall.

READ_STATEMENT_TIMEOUT=5min is set on the task definition (terraform/go-schedules.tf) so the target query is not cut off by the read pool’s default statement timeout.

Failure modes

Flags

Local development

just go-enrich-malware-exposure-fetch prod BATCH=25

# How far the sweep has got
psql "$DATABASE_URL" -c "
  SELECT \"registryStatus\", count(*)
  FROM \"PackageVersion\" WHERE \"registryStatus\" IS NOT NULL
  GROUP BY 1 ORDER BY 2 DESC;"

S3 Persistence

• Archive: ⚠ Not yet implemented — requires record reconstruction (DB row → canonical JSON).
• Quarantine: ⚠ Not yet implemented — same reason.
• Likely reasons when implemented: enrich-error

This is an enrichment processor; it reads from CVEMetadata rather than ingesting raw feeds, so there is no original payload to archive verbatim. See S3 Persistence Contract § Processors whose unit-of-work is not a file.