enrich-googleosi Design

Enriches package records from the Google OSI (deps.dev) API for packages referenced in CVE affected data, and for the registry packages the ecosystem processors ingested that no CVE has named yet.

Overview

• Batch size: 1 package per run
• Schedule: Every 15 minutes (cron(*/15 * * * ? *)), 256 CPU / 512 MB, EXPECTED_DURATION_MINUTES=45
• Concurrency: No master lock — the per-package tracker row is claimed before any fetch, so two overlapping runs cannot pick the same package
• State: Per-package rows in BulkDataDumpTracker, keyed enrich_googleosi:{SYSTEM}:{packageName} (e.g. enrich_googleosi:NPM:lodash). Legacy enrich_googleosi:{cveId} rows from an earlier CVE-driven design are skipped as malformed.
• Throughput ceiling: 96 packages/day. Phase 1 (the CVE-driven queue) keeps up; phase 1b is a very long tail — see Queue phases.

Queue phases

findNextPackageBatch tries three sources in strict priority order and returns on the first that yields anything. Each phase runs on its own context.Background() 5-minute timeout, so a near-expired task context cannot cancel a scan mid-flight.

sourceRepoUrl IS NULL is phase 1b’s not-yet-enriched marker because it is the field this enrichment reliably sets. Note that db.SetPackageVersionTEAFields is an UPDATE keyed on (ecosystem, packageName, version) — if deps.dev reports versions that do not exist as PackageVersion rows, nothing is written and sourceRepoUrl stays null. The tracker row still exists, so the package is not re-selected; it is simply not improvable from deps.dev.

Processing Flow

1. Find the next package via the three phases above
2. Claim it (upsert the per-package tracker row) before fetching, so a crash mid-fetch does not re-serve it
3. FetchDepsDevPackage: GET https://api.deps.dev/v3/systems/{SYSTEM}/packages/{packageName} — the host is api.deps.dev, not deps.dev/api, which serves HTML
4. For each version of the package: walkDependencyTree(depth=0, maxDepth=3)
   • Fetch version info → upsert Dependency, DependencyLink, DependencyRegistry, DependencySLSAProvenance, DependencyAttestation
   • Write back to PackageVersion (writePackageVersionTEAFields): publishedAt and sourceRepoUrl via db.SetPackageVersionTEAFields. Every argument is optional and nil leaves the stored value alone, so an enrichment pass can add to a package record but never erode one. deps.dev carries no licence expression on the version endpoint, so licenseExpression is always passed nil.
   • If a SOURCE_REPO or REPOSITORY link points to github.com (first match only):
     • CreateOrUpdateGitHubRepository → upsert GitHubRepository
     • UpsertGitHubBranches, UpsertGitHubContributors, UpsertGitHubLanguages
     • UpsertOpenSSFScorecard via api.scorecard.dev
     • UpsertGitHubRepoPackageManager → UpsertGitHubRepoDependency
   • Fetch the dependency graph → recurse on every non-SELF node. Cycle prevention uses an in-memory visited set plus a persistent BulkDataDumpTracker check that skips any (SYSTEM:name:version) enriched within the last 12 hours.
5. MarkPackageCVEsEnriched: update CVEMetadata.lastEnriched for every CVE whose CVEAffected row names this package on one of the system’s collection URLs

FetchDepsDevAdvisory / EnrichCVEWithDepsDevData still exist in internal/enrichment/googleosi.go but are not on this path — no CVEMetadataReferences rows with referenceSource=GOOGLE_OSI are produced.

Ecosystem ↔ system mapping

deps.dev, the registry processors, and the GitHub package-manager vocabulary all spell ecosystems differently (PYPI / pypi / pip). Matching on the wrong one silently updates nothing, which is indistinguishable from a package with no extra data, so the mapping is explicit in depsDevSystemToEcosystem and exported once via EcosystemSystemPairs() for the phase-1b SQL: NPM, PYPI, GO, MAVEN, CARGO, RUBYGEMS, NUGET, PACKAGIST, PUB, HEX.

orgId Resolution

GitHubRepoDependency.orgId and GitHubRepoPackageManager.orgId are required String fields. For VDB enrichment, orgId = "CNA-2019-0009" is a constant (internal/enrichment/github_enrichment.go), and the matching CVENumberingAuthority row is upserted on demand — no manual seed is required.

Tables Written

• PackageVersion — publishedAt, sourceRepoUrl, lastVerifiedAt on the row the ecosystem processors own (UPDATE only, COALESCE’d — never inserts, never blanks)
• Dependency — package from deps.dev version info
• DependencyLink — HOMEPAGE/SOURCE_REPO/REPOSITORY links
• DependencyRegistry — registry URLs
• DependencySLSAProvenance — SLSA provenance from deps.dev
• DependencyAttestation — attestation records
• GitHubRepository — GitHub repo metadata (if SOURCE_REPO/REPOSITORY link found)
• GitHubBranch — branches via GitHub API
• GitHubRepoContributor — contributors via GitHub API
• GitHubRepoLanguage — language breakdown via GitHub API
• GitHubRepoPackageManager — package manager record (orgId=CNA-2019-0009)
• GitHubRepoDependency — repo↔dependency link (orgId=CNA-2019-0009)
• OpenSSFScorecard — scorecard from api.scorecard.dev
• CVENumberingAuthority — the ('CNA-2019-0009','GitHub') seed row, upserted on demand
• CVEMetadata.lastEnriched — enrichment timestamp
• BulkDataDumpTracker — per-package state + the 12-hour per-(SYSTEM:name:version) dedup markers

Environment Variables

S3 Persistence

• Archive: ⚠ Not yet implemented — requires record reconstruction (DB row → canonical JSON).
• Quarantine: ⚠ Not yet implemented — same reason.
• Likely reasons when implemented: enrich-error

This is an enrichment processor; it reads from CVEMetadata rather than ingesting raw feeds, so there is no original payload to archive verbatim. See S3 Persistence Contract § Processors whose unit-of-work is not a file.