OSV File Processor — Design Document

1. Overview

Purpose: Download OSV vulnerability data from Google Cloud Storage public buckets for every ecosystem OSV publishes, deduplicate by SHA256, and process records directly into CVEMetadata and related tables using the shared processing pipeline.

Data source: GCS public bucket — https://osv-vulnerabilities.storage.googleapis.com/<Ecosystem>/all.zip

Ecosystem discovery: The processor lists the GCS bucket at startup (?delimiter=/, with marker pagination) and processes every top-level prefix that has an all.zip. The list is not hardcoded, so any ecosystem OSV adds (e.g. MinimOS, Echo, SwiftURL, future additions) is picked up automatically on the next run. Filtered out:

• Versioned subprefixes (anything containing :, e.g. Alpine:v3.10, Ubuntu:22.04:LTS).
• Sentinel / non-ecosystem folders: all, icons, [EMPTY], Root.

Ecosystems handled by a dedicated processor are not skipped. OSV is the canonical aggregator and we want every OSV record on disk regardless of whether another processor also writes the same advisory. Collision is impossible because every row in CVEMetadata is keyed by (cveId, source) and this processor always writes with source = "osv". A dedicated processor (e.g. chainguard-json-processor) writes its own (cveId, source="chainguard") row from the upstream feed; the OSV row and the dedicated row coexist and are linked through CVEAlias.

Schedule: Daily (cron 0 0 * * *)

Timeout: see ECS task definition (3 hour soft deadline; configurable via EXPECTED_DURATION_MINUTES)

Resources: 256 CPU units, 1024 MB memory

What it reads:

• GCS bucket listing (/?delimiter=/) — for ecosystem discovery
• GCS ZIP archives (one per ecosystem; cached to /tmp/YYYYMMDD_{ecosystem-slug}.zip within the run)
• Read replica: OsvFile per-file SHA256 dedup check
• Read replica: BulkDataDumpTracker per ecosystem for archive-level SHA256 dedup

What it writes:

• CVEMetadata (one row per advisory; source = "osv" uniformly, cveId = adv.ID)
• CVEDescription, CVEMetadataReferences, CVEMetric, CVEProblemType, CVEAffected, CVEAffectedVersion
• CVEAlias (via db.InsertAliases from the central pipeline — see §2 Identifier & Alias Policy)
• OsvFile (per-file tracking record: status, r2Path, r2StoredAt, cveCount, advisoryCount)
• Artifact (one per file if S3 upload succeeded)
• Link (one per file if S3 upload succeeded, with artifactUuid FK)
• CVEMetadata.fileLinkId → set to Link.id if available
• BulkDataDumpTracker (one row per ecosystem: osv_pypi, osv_minimos, osv_github_actions, etc.)

Environment variables:

• DATABASE_URL — required
• DATABASE_URL_READ — optional
• S3_BUCKET_NAME — optional; if absent, S3 upload, Artifact, and Link records are skipped (local dev mode)
• EXPECTED_DURATION_MINUTES — optional; sets soft deadline (default 3 h)

2. Business Logic

Identifier & Alias Policy (CRITICAL)

OSV advisories carry an OSV-native id (e.g. MINI-jhg8-3g7g-q6mp, GHSA-xxxx-xxxx-xxxx, PYSEC-2024-1234, RUSTSEC-2024-0001) and may list aliases[] containing CVE-* identifiers. This processor never uses a CVE-* alias for CVEMetadata.cveId:

Why: The CVE-* identifier (when present) is the cross-DB pivot, not a substitute identifier. If we used the CVE-* as cveId, the OSV record would either collide with a dedicated processor’s row or silently overwrite the native OSV identity. Keeping cveId = adv.ID preserves the OSV identity and makes the CVE-* available as a navigable alias edge.

db.InsertAliases handles cross-source linking automatically: when this processor writes (MINI-jhg8-3g7g-q6mp, source=osv) with alias CVE-2025-5889, it backfills edges to every other source that already carries CVE-2025-5889 (e.g. NIST NVD, GHSA, MITRE). Subsequent runs of any processor that writes the same cveId will likewise backfill an edge to the OSV row. This is what the project means by “the relations carry the canonical source”: the CVEMetadata.source is uniformly "osv" for OSV-written rows, while the alias edges (in CVEAlias.discoveredFrom / peer rows) carry the per-source canonical identity.

This is implemented by calling osv.MapAdvisoryKeepID (which sets mapped.CveID = adv.ID and propagates aliases[]/related[]/upstream[] to mapped.Aliases), and passing the result through processor.StoreCVESourceDataDeferred → db.InsertAliases.

/tmp Download Caching

At startup, files matching /tmp/????????_*.zip that do NOT start with today’s date (YYYYMMDD) are deleted to reclaim space. Each ecosystem ZIP is cached to /tmp/YYYYMMDD_{ecosystem-slug}.zip on first download and reused for the rest of the run. Ecosystem names with spaces or punctuation are slugified (GitHub Actions → github_actions, Azure Linux → azure_linux).

Per-Ecosystem Processing

Ecosystems are processed sequentially. Each ecosystem is independent — failure in one does not stop others. A soft deadline (default 3 h) bounds the total run; if exceeded, the loop stops cleanly between ecosystems.

Two-Level SHA256 Dedup

1. Archive level: Compute SHA256 of the downloaded ZIP bytes. Compare against BulkDataDumpTracker.sha256 for osv_<ecosystem-slug>. If unchanged → skip the entire ecosystem.
2. File level: For each file inside the ZIP, check OsvFile for (archiveSha256, filename) → if fileSha256 matches → skip this file.

Direct Processing (No Queue)

Each file is parsed with osv.ParseJSON, mapped with osv.MapAdvisoryKeepID, and stored directly via processor.StoreCVESourceDataDeferred — all within a database transaction with per-file SAVEPOINTs for rollback isolation. Deferred enrichment (Dependency / PackageVersion upserts) runs in independent short transactions after the main batch commits, to avoid cross-worker deadlocks.

Batch Processing

Files are split into batches of 200 and processed by 4 concurrent workers. Each batch is one transaction with per-file SAVEPOINTs. A 10-minute per-batch context bounds slow batches.

S3 Upload (Conditional)

If S3_BUCKET_NAME is set, each file is uploaded to osv/files/{fileSHA256}/{filename} before the DB transaction writes. S3 keys are content-addressed so re-uploads are idempotent if the DB write fails and the file is re-processed next run.

Tracker Source Key

osv_<slugified ecosystem> — e.g. osv_pypi, osv_minimos, osv_github_actions, osv_azure_linux, osv_bellsoft_hardened_containers.

URL Encoding

Ecosystem names with spaces (GitHub Actions, Azure Linux, BellSoft Hardened Containers) are URL-encoded for the GCS download path via url.PathEscape.

Malicious Package Detection

osv.MapAdvisoryKeepID returns nil for malicious packages (MAL- prefix on the primary id, on any alias/related/upstream, or ossf-malicious-packages source). These are silently skipped (no DB write). Malicious packages are handled by oss-malware-git-processor.

3. OsvFile Status Lifecycle

(new file)    →  processing  →  completed
                              ↘  failed
(sha unchanged)  →  skipped (no DB write)

4. S3 Key Convention

osv/files/{fileSHA256}/{filename}
Example: osv/files/abc123.../MINI-jhg8-3g7g-q6mp.json

GCS source URL stored in the Link record:

https://osv-vulnerabilities.storage.googleapis.com/{Ecosystem}/{filename}

5. Local vs ECS Behaviour

6. Architecture Diagram

7. Deployment Diagram

8. Processing Flow

9. Data Mapping

10. AI Enrichment (aienrich)

Targets are collected inside the batch transaction (one per file that clears RELEASE SAVEPOINT sp_file) into enrichTargets. After the batch tx commits successfully and processor.RunDeferredEnrichment finishes, aienrich.RunBatch(ctx, enrichTargets) fires once with the whole batch. The slice is reset on every retry attempt so retries can’t double-count. Wire-in: cmd/osv-json-processor/main.go, Phase 4 right before processBatch returns.

Four passes fire per target (ghsapoc self-gates to GHSA-shaped ids, which is rare for OSV but kept for GHSA:* records that flow through this processor):

Target.Source = "osv" matches CVEMetadata.source. See the aienrich overview for the full activity contract.

S3 Persistence

Not used. This processor does not currently archive payloads or quarantine failures to S3. Per the S3 Persistence Contract this is non-compliant — see the compliance matrix for the implementation roadmap.

⚠ Not in the compliance matrix — status needs verification.

Expected paths when implemented:

• Archive: osv-file/files/{sha256}/{filename}
• Quarantine: failed-feeds/osv-file-processor/{YYYY-MM-DD}/{reason}/{filename}
• Likely reasons: (none documented)