gemnasium-git-processor

Status: Live Source: gitlab-org/security-products/gemnasium-db (git) Type: git (runtime clone/pull via pure-Go go-git — no baked data stage) Source slugs: gitlab (provenance) plus the upstream source implied by each advisory identifier — cve.org, github, gms, and others Schedule: Runs daily at 03:00 UTC (cron(0 3 * * ? *)). CPU 256 / memory 512 MB / expected_duration_minutes = 45.

Overview

Gemnasium is the advisory database behind GitLab Dependency Scanning. It independently curates affected ranges for npm, PyPI, Maven, Go, RubyGems, NuGet, Packagist, Cargo, Pub, Conan and Swift, and it carries GMS-* advisories that exist nowhere else. Its practical value here is range quality: Gemnasium frequently states a tighter, GitLab-verified affected_range than the CVE record does, and it is the second independent opinion the alias graph reconciles against GHSA and NVD.

Each run:

1. Refreshes the repo. processor.PullOrClone clones (or pulls) /data/gemnasium-db at depth 1 through go-git — the final image stage is scratch and ships no baked advisory data. --no-pull reads the existing HEAD instead.
2. Short-circuits on an unchanged HEAD against BulkDataDumpTracker.sha256 for source gemnasium_advisory; a match without --force reports task.nowork and exits 0.
3. Loads the resume skip set with db.LoadProcessedHashes("gitlab") (30-second budget — this is deliberately short; on timeout the run proceeds without resume and re-upserts).
4. Walks the repo root recursively for .yml files, skipping .gitlab-ci.yml, then applies the 3-day changed-file filter when a tracker exists.
5. Stores in transactions of --batch-size (default 200) with one SAVEPOINT per file; each batch runs under db.WithBatchTx with its own 5-minute context so an overtime cancel cannot poison an in-flight commit.
6. Attributes actors — malwareactor.PostPass for source gitlab.

The two-row write — read this before querying source='gitlab'

Gemnasium is a republisher, not an issuing authority, so the processor deliberately splits each advisory across two CVEMetadata rows:

1. Provenance row — db.UpsertCVEMetadata with source='gitlab', carrying only cveId, title, sourceAdvisoryRef, datePublished, dateUpdated, isMaliciousPackage, aliases and sourceFileHash. No descriptions, metrics, affected rows or references hang off it. Its job is to record that gemnasium-db has this advisory and to carry the per-file resume hash.
2. Full row — mapped.Source is then reassigned to sourceident.FromPrefix(adv.Identifier) and the record is stored through processor.StoreCVESourceData. So a CVE-* advisory writes its full payload under source='cve.org', a GHSA-* advisory under source='github', a GMS-* advisory under source='gms', and an unrecognised prefix falls back to gitlab.

Consequences worth knowing:

• count(*) WHERE source='gitlab' counts advisories gemnasium knows about, not rows with data. Most gitlab rows have a null affectedProduct by design.
• The bulk of this processor’s CVEDescription / CVEMetadataReferences / CVEAffected output lands under source='cve.org', where it shares rows with mitre-cve-json-processor. Because UpsertCVEMetadata overwrites title, vectorString, rawDataJSON, affectedVendor/affectedProduct and state from the incoming record, whichever of the two processors ran last owns those columns for a given CVE.
• Reference rows written by this processor are identifiable by referenceSource='GitLab' regardless of which source row they hang off.

datePublished for the provenance row is the mapper’s upstream date when present, else derived from the CVE ID’s year (cveid.DateFromID) — never time.Now(), so the UpsertCVEMetadata earliest-wins guard can preserve a better value from another source.

Records produced

Parsing

Gemnasium YAML is not OSV, so internal/gemnasium has its own schema: identifier, identifiers[], package_slug, affected_range, fixed_versions, cvss_v2, cvss_v3, cwe_ids, urls.

• ParsePackageSlug normalises ecosystem/package to the OSV ecosystem vocabulary: npm/lodash → ("npm","lodash"), pypi/requests → ("PyPI","requests"), maven/org.example/artifact → ("Maven","org.example:artifact"), go/github.com/u/r → ("Go","github.com/u/r").
• ParseAffectedRange expands constraint strings into version rows: >=X,<Y, >=X <Y, ||-combined sub-ranges, =X, bare <X/<=X (lower bound assumed 0), and Maven interval notation (,X] / [X,Y).
• ExtractCVEID prefers a CVE-YYYY-NNNN+ match in identifier, then in identifiers[], and falls back to the primary identifier — this is the value used both as the row key and for the resume lookup.

Threat-actor attribution

malwareactor.PostPass(source="gitlab") runs after the walk. It is first-time-only (a (cveId, source) with a MalwareAttribution row is skipped) and bounded to MALWARE_ACTOR_BATCH (default 150) records per run; MALWARE_ACTORS=false disables it. It writes to the generic MalwareThreatActor / MalwareAttribution / ThreatActor tables. Note the pass selects on source='gitlab', i.e. the provenance rows.

Failure modes

Flags

Local run: just go-gemnasium-git-backfill (append prod for production).

S3 Persistence

• Archive path: gitlab/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/gemnasium-git-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: store-error, tx-rollback

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.