gitlab-json-processor

Status: Live Source: gitlab.com/gitlab-org/cves — tarball …/-/archive/master/cves-master.tar.gz Type: json (single gzipped tar of CVE Record Format 5.0 files) Source slug: gitlab — shared with gemnasium-git-processor, see the warning below Schedule: Runs daily at 08:00 UTC (cron(0 8 * * ? *)), ECS command ["/app/gitlab-json-processor", "--emit-crit=true"].

⚠ source='gitlab' has two writers, and this one owns the small share. gemnasium-git-processor (cmd/gemnasium-git-processor/main.go:32, source = "gitlab") writes the GitLab gemnasium-db advisory corpus under the same slug. Measured in production, of 39,191 source='gitlab' rows:

dataVersion	Rows	Writer	Shape
1.0	37,935	gemnasium-git-processor	UUID sourceAdvisoryRef, gemnasium YAML-derived payload (8,176 carry rawDataJSON)
5.0 / 5.1 / 5.2	1,256	this processor	sourceAdvisoryRef = cveId, CVE Record Format payload

Consequences:

• Any per-source field-coverage measurement of gitlab mixes both writers. Split on dataVersion LIKE '5.%' to isolate this processor’s output. (rawDataJSON IS NOT NULL does not work — gemnasium populates it for 8,176 of its own rows.)
• db.LoadProcessedHashes("gitlab") (step 2 below) loads both writers' sourceFileHash values into one keyspace. For a CVE both processors know about, each run sees the other’s hash, the resume check misses, and the record is needlessly re-stored — and whichever runs last owns the row, because db.UpsertCVEMetadata assigns rawDataJSON, dataVersion and vectorString unconditionally from EXCLUDED (internal/db/cvemetadata.go:59,72). The last full run of this processor stored 1,288 records (BulkDataDumpTracker.gitlab.totalCVEs) yet only 1,256 rows still carry a 5.x dataVersion — the difference is records gemnasium has since overwritten.

The trackers are distinct (gitlab vs gemnasium_advisory), so the freshness gates do not interfere. Fixing the collision means giving one writer its own slug.

1. Overview

• Purpose: ingest GitLab’s first-party CVE database (one CVE 5.0 JSON per advisory, year-organised) and emit CVEMetadata rows under source="gitlab" plus optional CRIT candidates for the GitLab.com SaaS and self-managed (CE/EE) SKUs.
• Source URL: https://gitlab.com/gitlab-org/cves (public repo). Tarball: https://gitlab.com/gitlab-org/cves/-/archive/master/cves-master.tar.gz.
• Owner: GitLab Security (CNA assignerShortName="GitLab").
• Licence: repo licence (CC0/CC-BY — confirm at runtime; no scrape restrictions imposed).
• Schedule: cron(0 8 * * ? *) daily 08:00 UTC. GitLab publishes security releases on Tuesdays; daily run keeps cadence within 24 h while avoiding the 06:00–07:00 slot already occupied by twilio/fastly.
• ECS resources: cpu_units=512, memory_mb=1024, expected_duration_minutes=30 (per blueprint sizing for full-dataset JSON tarball download + ~1500 records).
• Reads: GitLab archive endpoint (one HTTP fetch), BulkDataDumpTracker.gitlab for freshness, LoadProcessedHashes for per-record resume.
• Writes: CVEMetadata, CVEDescription, CVEMetadataReferences, CVEMetric, CVEAffected, CVEAffectedVersion, CVEProblemType, CVEAlias, BulkDataDumpTracker, S3 archive, optional CritRecord via crit-candidates/pending/.

2. Source contract (Phase 0 recon)

The schema match means we can reuse internal/cvelistv5 for parsing and storage end-to-end — no new mapper code.

3. Architecture

4. Source → DB field mapping

Handled entirely by cvelistv5.StoreRecordWithSource(ctx, tx, rec, fileHash, "gitlab", cveID, logger). The cvelistv5 mapper persists every field of the CVE 5.0 record into the appropriate row (descriptions, references, metrics, affected, problem-types). No bespoke mapping table is required for this processor.

The only override versus default cvelistv5 ingestion: source="gitlab" instead of the default "cvelistv5". This drives downstream attribution and ensures GitLab’s CVE rows are distinguishable from copies that may also exist in the broader cvelistv5 corpus.

5. Identifier policy

• CVEMetadata.cveId: native CVE-YYYY-NNNN from cveMetadata.cveId. GitLab is a CNA — every advisory has a CVE identifier; no minted IDs needed.
• CVEAlias: written automatically by cvelistv5.StoreRecordWithSource via db.InsertAliases (mapper.go:279). Empty alias list still triggers same-cveId cross-source backfill so a gitlab-sourced row is automatically aliased to peer nist-nvd/cve.org rows when those exist.
• No bundle suppression required — GitLab CVEs are 1:1 with their published CVE IDs.

6. CRIT / VEX

Service taxonomy

The six services below ship in the CRIT spec dictionary (github.com/Vulnetix/ietf-crit-spec → dictionaries/gitlab.json), loaded by critutil.LoadSpecDictionaries(). There is no internal/critutil/dictionaries/extended/ directory; the local extended layer is internal/critutil/dictionaries/local/ and carries no GitLab entries. A triple with no dictionary template resolves to nil and stages nothing, so the dictionary is the hard gate on emission. A record that stages zero envelopes is offered to CRIT inference via critprep.Hook instead.

Default emission: two envelopes per CVE (gitlab_saas + gitlab_self_managed). Service-specific keywords extend with additional envelopes; they don’t replace the default pair.

VEX status derivation

• All emitted candidates use vex_status="fixed". GitLab only publishes CVEs against shipped releases — by the time the CVE-*.json file appears in the public repo, fixed versions are out.
• temporal.provider_fix_date = cveMetadata.datePublished (parsed to YYYY-MM-DD).
• temporal.vuln_published_date = canonical lookup (db.LoadCanonicalDatePublished) with fallback to cveMetadata.datePublished.
• temporal.service_available_date = "2014-03-01" (GitLab.com GA).

Provenance

• provenance.primary_source.kind = "gitlab", advisory_id = cveID, advisory_url = "https://gitlab.com/gitlab-org/cves/-/blob/master/{year}/{cveID}.json", s3_key = gitlab/files/{sha256}/{path}.
• Evidence rules:
  • source-scoped-provider: “GitLab CNA-issued advisory implies provider=gitlab”
  • service-text-match: hint keyword → service triple
  • dictionary-resolved: (gitlab,…) ∈ extended dict
  • cve-publish-date-lookup: canonical-date trace

7. S3 / source-file archive layout

• Bucket: vdb-manager-artifacts (or whatever S3_BUCKET_NAME resolves at runtime).
• Archive prefix (success): gitlab/files/{sha256}/{year}/{CVE-YYYY-NNNN}.json — content-addressed; the {path} is the tarball-relative path (preserves year directory).
• Quarantine prefix (failure): failed-feeds/gitlab-json-processor/{YYYY-MM-DD}/{reason}/{path}.
• Quarantine reasons: parse-error (cvelistv5 unmarshal failed), store-error (DB transaction failed after retries), decode-error (gzip/tar extraction failed — terminates the run with a fatal).

8. Error handling & Slack

• Per-record failures are non-fatal: log warn, increment failed counter, quarantine the bytes, continue to next record.
• Tarball fetch failure is fatal: exit 1 with Errored Slack event.
• Retry profile: HTTP fetch retries 3 times with exponential backoff via the standard httpclient.New client. Per-record DB writes are attempted once — failures go to quarantine.
• Slack events: Started at process entry; Errored on fatal exits (DB pool init, tarball fetch); NoWork when tracker is fresh and not --force; Completed with stats {fetched, stored, skipped, failed, critStaged} on clean exit.

9. Performance

• Concurrency: sequential walk of tarball entries; the bottleneck is per-record DB write. ~1,290 records at ~10 ms each is comfortably under the 30-min soft deadline once resume is warm, but a --force cold pass is the run most likely to hit it.
• Per-request timeout: httpclient default (30 s) for tarball fetch with internal retries.
• Soft deadline: only when EXPECTED_DURATION_MINUTES env var is set; backfill runs unset it (per blueprint “Backfill must not have a deadline”).
• Resume strategy:
  1. BulkDataDumpTracker.gitlab checked at startup; if fresh and not --force, exit NoWork.
  2. LoadProcessedHashes(source="gitlab") returns {cveId → sourceFileHash}. Records whose tarball-entry SHA1 matches the tracker’s prior hash are skipped (cvelistv5 already stored them).

10. Backfill

• cmd/gitlab-json-processor itself is the backfill — same binary, same flow. The justfile recipe go-gitlab-json-backfill adds:
  • unset EXPECTED_DURATION_MINUTES so the soft deadline does not fire.
  • --force={{FORCE}} to bypass tracker freshness.
  • --limit={{LIMIT}} for sampling during development.
  • --emit-crit={{EMIT_CRIT}} — the recipe’s EMIT_CRIT default is "true", even though the binary’s own flag default is false. The ECS command also passes --emit-crit=true.
• Expected runtime: a few minutes on prod data once resume is warm; most entries skip via sourceFileHash after the first run.

11. Tests

• internal/gitlab/parse_test.go walks a checked-in mini-tarball fixture (testdata/cves-mini.tar.gz, 3 records) and asserts each entry parses with cvelistv5.ParseRecord.
• cmd/gitlab-json-processor/crit_mapper_test.go covers:
  • dual SaaS+self-managed emission for a generic GitLab CVE
  • Runner extra envelope when affected.product="GitLab Runner"
  • Dependency Proxy extra envelope when description mentions it
  • Container Registry extra envelope
  • empty CVE id rejection
  • S3 key shape

12. Definition of done

See plan §17. Every box must tick before merge:

• Phase 0 recon documented (this doc §2).
• go vet + go test clean across internal/gitlab/ and cmd/gitlab-json-processor/.
• All 6 registration files updated (Containerfile, terraform, task-manager.toml, post-push-ecr.sh, ecr-build targets.go, justfile) with the locked name gitlab-json-processor.
• cvelistv5.StoreRecordWithSource is the only path to CVEMetadata.
• S3 archive on success + quarantine on failure wired.
• Slack events confirmed for at least one local backfill run.
• §16 verification executed end-to-end against .env.production.
• One CloudWatch firing observed with exit code 0.

S3 Persistence

• Archive path: gitlab/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/gitlab-json-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: parse-error, store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.