OSS-Fuzz Advisory Processor — Design Document

1. Overview

Purpose: Process the Google OSS-Fuzz vulnerability database OSV YAML files (OSV-*.yaml) into CVEMetadata and related tables.

Data source: Google OSS-Fuzz vulnerability database — github.com/google/oss-fuzz-vulns (git repository cloned to /data/oss-fuzz-vulns)

Schedule: Runs weekly on Thursdays at 04:00 UTC (cron(0 4 ? * THU *)).

Timeout: 45 minutes

Resources: 256 CPU units, 512 MB memory

What it reads:

  • Git repository at /data/oss-fuzz-vulns/vulns/ — walks files matching OSV-*.yaml
  • Read replica: existing sourceFileHash values for per-file resume
  • Read replica: BulkDataDumpTracker row for ossfuzz_advisory (HEAD SHA dedup)

What it writes:

  • CVEMetadata (upsert, source=ossfuzz)
  • CVEDescription, CVEMetadataReferences, CVEMetric, CVEProblemType
  • CVEAffected, CVEAffectedVersion
  • Dependency, DependencyRegistry, PackageVersion, PackageVersionCVE, GitHubRepoDependency — via db.EnrichAffectedWithDependency, called for every affected entry inside storeAdvisory
  • CVEAlias — always, via db.InsertAliases, even for advisories with no alias list, so the same-cveId cross-source backfill runs on every store
  • A derived Vulnetix CVSS v4 metric (containerType="vulnetix") when the advisory has descriptions and does not suppress derived metrics
  • BulkDataDumpTracker
  • When AI enrichment is enabled (PIX_INFERENCE_ENABLED + gateway token, i.e. ECS only): CVEProblemType (derivedBy="vulnetix"), CVEAttackTechnique / CVEAttackMitigation / CVEAttackDetection / CVEAttackD3fend, CVETreeSitter, and one PixLog row per pass

Environment variables:

  • DATABASE_URL — RDS Write Proxy connection string (required)
  • DATABASE_URL_READ — RDS Read Replica connection string (optional)

2. Business Logic

Pipeline Reuse

OSS-Fuzz uses the identical shared pipeline as PyPI processor (internal/processor). The only differences are constants: source=ossfuzz, trackerSource=ossfuzz_advisory, referenceSource=OSS-Fuzz, file prefix OSV-, and repo path /data/oss-fuzz-vulns.

Git Pull Strategy

processor.PullOrClone attempts git pull --ff-only on the existing clone. Returns HEAD SHA. Exits on error.

SHA-based Freshness Check

Compares HEAD SHA against BulkDataDumpTracker.sha256 for ossfuzz_advisory. Skips on match unless --force.

Per-File Resume

LoadProcessedHashes(source="ossfuzz") loads existing (cveId, sourceFileHash) pairs. SHA1 computed per file; skip if unchanged. The read is given its own 5-minute context because the replica scan is slow at production scale.

Delta Window

processor.DetectChangedFiles narrows the walk to files git reports as recently changed, and processor.FilterPaths applies it. Skipped on the first run (no tracker row) and under --force, where the full tree is walked.

Soft Deadline

When EXPECTED_DURATION_MINUTES is set (ECS only), batches stop dispatching at minutes - 10 (main.go:63-65). Local backfills unset the variable and run to completion.

Known defect — the intended behaviour is that a soft-deadline truncation leaves the tracker un-advanced so the next run resumes. The guard in main.go:159 only tests ctx.Err(), which a soft-deadline break in internal/processor/pipeline.go:161-164 never sets, so the HEAD SHA is written after a truncated run. The next scheduled run then sees an unchanged SHA and exits with NoWork, so any file the truncated run did not reach is skipped until upstream pushes a commit — and even then the 3-day delta window excludes it. Tracked in .repo/efficacy/ossfuzz-git-processor.yaml.

Malicious Package Detection

Same 6-criteria check as PyPI processor. OSS-Fuzz advisories should not contain malicious packages, but the check runs anyway for safety.

CVE ID Derivation

Scans aliases[] for CVE-YYYY-NNNN+ pattern. Otherwise uses OSV-YYYY-NNNNN as the cveId.

Ecosystem Mapping

OSS-Fuzz ecosystem maps to collection URL https://google.github.io/oss-fuzz (defined in osv/mapper.go ecosystemCollectionURLs).

Batch Processing

Batches of 100 files per transaction, with per-file SAVEPOINT for fault isolation.

Tracker Update

Only updates tracker on a clean run (filesErrored == 0). Stores HEAD SHA in sha256.

Exit Semantics

File errors do not by themselves fail the task: notify.Finalize decides, and the process exits 0 unless the failure ratio is systemic (main.go:140-155). The tracker is skipped in that case so the next run retries. A pipeline-level error or a failed tracker write exits 1.

3. Architecture Diagram

graph TD subgraph "cmd/ossfuzz-git-processor/" MAIN[main.go] end subgraph "internal/processor/" GIT[git.go — PullOrClone] PIPE[pipeline.go — Run, processBatch, storeAdvisory] WALK[walker.go — WalkAdvisories] end subgraph "internal/osv/" PARSE[parser.go — ParseYAML] MAP[mapper.go — MapAdvisory, IsMaliciousPackage] TYPES[types.go — Advisory, CVESourceData] end subgraph "internal/db/" POOL[pool.go — Pool] TRACK[tracker.go — GetTracker, UpsertTracker] RESUME[resume.go — LoadProcessedHashes] BATCH[batch.go — WithTx] CVEMETA[cvemetadata.go — UpsertCVEMetadata] CHILDREN[cvedescription, cvereference, cvemetric, cveaffected, cvealias, cveproblemtype] end MAIN --> GIT MAIN --> TRACK MAIN --> POOL MAIN --> PIPE PIPE --> WALK PIPE --> RESUME PIPE --> PARSE PIPE --> MAP PIPE --> BATCH PIPE --> CVEMETA PIPE --> CHILDREN

4. Deployment Diagram

flowchart TD HOOK[Local ECR hook
post-push-ecr.sh] -->|push ARM64 image| ECR[ECR: go-processors
tag: go-ossfuzz-git-processor-sha-xxx] ECR --> TASKDEF[ECS Task Definition
go-ossfuzz-git-processor] TASKDEF --> EB[EventBridge Schedule
go-ossfuzz-git-processor
cron 0 4 ? * THU *] EB -->|trigger| FARGATE[ECS Fargate Task
vdb-scheduler cluster
ARM64 ap-southeast-2] FARGATE --> CW[CloudWatch Logs
/ecs/vdb-scheduler/go-ossfuzz-git-processor] FARGATE --> WRITE[RDS Write Proxy
cf-hyperdrive.proxy-...] FARGATE --> READ[RDS Read Replica
hyperdrive-saas-read...] FARGATE -->|baked into image
ossfuzz-git-data stage| REPO[/data/oss-fuzz-vulns]

5. Processing Flow

flowchart TD START([Start]) --> ENV{DATABASE_URL set?} ENV -->|no| FAIL([Exit 1]) ENV -->|yes| CONNECT[Connect to DB pool] CONNECT --> PULL[PullOrClone /data/oss-fuzz-vulns] PULL -->|error| FAIL PULL -->|ok — headSHA| TRACKER{Tracker sha256
== headSHA?} TRACKER -->|yes, no --force| EXIT0([Exit 0]) TRACKER -->|no or --force| LOAD[LoadProcessedHashes source=ossfuzz] LOAD --> WALK[WalkAdvisories vulns/ OSV-*.yaml] WALK --> BATCH[For each batch of 100 files] BATCH --> TX[Begin transaction] TX --> FILE[For each file] FILE --> SP[SAVEPOINT sp_file] SP --> READ2[ReadFile + SHA1] READ2 --> PARSE[ParseYAML] PARSE -->|error| ROLLBACK[ROLLBACK TO SAVEPOINT
count error] PARSE -->|ok| CVEID[Derive cveId from aliases] CVEID --> SKIP{skipSet match?} SKIP -->|yes, no force| RELEASE[RELEASE SAVEPOINT
count skipped] SKIP -->|no or force| MALICIOUS{IsMaliciousPackage?} MALICIOUS -->|yes| RELEASE MALICIOUS -->|no| STORE[storeAdvisory
UpsertCVEMetadata + children] STORE -->|error| ROLLBACK STORE -->|ok| RELEASE ROLLBACK --> FILE RELEASE --> FILE FILE -->|done| COMMIT[Commit transaction] COMMIT --> BATCH BATCH -->|done| ERRORS{filesErrored > 0?} ERRORS -->|yes| FAIL2([Exit 1 — skip tracker]) ERRORS -->|no| UPSERT_TRACKER[UpsertTracker ossfuzz_advisory
sha256=headSHA] UPSERT_TRACKER --> DONE([Exit 0])

6. Data Mapping

erDiagram BulkDataDumpTracker { string source PK "ossfuzz_advisory" bigint lastProcessedAt string sha256 "git HEAD SHA" int totalCVEs } CVEMetadata { string cveId PK "CVE-YYYY-NNNN or OSV-..." string source PK "ossfuzz" string state int datePublished "Unix seconds" string affectedVendor "OSS project name" string affectedProduct string sourceFileHash "SHA1" bigint lastFetchedAt } CVEMetadataReferences { string uuid PK string cveId FK string url string referenceSource "OSS-Fuzz" } CVEAffected { string uuid PK string cveId FK string collectionURL "https://google.github.io/oss-fuzz" string packageName } CVEMetadata ||--o{ CVEDescription : has CVEMetadata ||--o{ CVEMetadataReferences : has CVEMetadata ||--o{ CVEMetric : has CVEMetadata ||--o{ CVEAffected : has CVEAffected ||--o{ CVEAffectedVersion : has CVEMetadata ||--o{ CVEAlias : aliases

S3 Persistence

  • Archive path: ossfuzz/files/{sha256}/{filename} ✓
  • Quarantine path: failed-feeds/ossfuzz-git-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
  • Failure reasons emitted: parse-error, store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[ossfuzz-git-processor] PROC -->|success| ARCHIVE[("S3: ossfuzz/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/ossfuzz-git-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.