ocaml-git-processor

Status: Live Source: ocaml/security-advisories (git) Type: git (repo baked by the ocaml-git-data stage; the final image is alpine with a git binary so PullOrClone uses system git at startup) Source slug: ocaml Schedule: Runs weekly on Mondays at 03:00 UTC (cron(0 3 ? * MON *)). CPU 256 / memory 512 MB / expected_duration_minutes = 45.

Overview

The opam ecosystem has no OSV feed and no CNA. ocaml/security-advisories is the only authoritative record of OCaml package vulnerabilities, and it publishes them in a bespoke DSL that no generic importer understands. Without this processor an opam lockfile has zero vulnerability coverage in the VDB and the OSEC↔CVE alias edges never exist.

The catalogue is small (tens of advisories), so this is a completeness feed, not a volume feed: its value is that an OCaml consumer gets any answer at all.

Each run:

Refreshes the repo with processor.PullOrClone on /data/security-advisories. The image is alpine with git installed (--no-pull skips the pull and reads the baked HEAD).
Short-circuits on an unchanged HEAD against BulkDataDumpTracker.sha256 for source ocaml_advisory; a match without --force reports task.nowork and exits 0.
Loads the resume skip set — db.LoadProcessedHashes("ocaml").
Walks advisories/ recursively for OSEC-*.md, then applies the 3-day changed-file filter when a tracker exists.
Loads known CVE IDs — db.LoadKnownCVEIDs reads every (cveId, source) pair matching CVE-% from the read replica, so ExploitCVE junctions can be attached to the sources that actually hold the CVE. This is a large scan and dominates start-up time.
Stores in transactions of --batch-size (default 50), one SAVEPOINT per file.

Advisory format

Advisories are Markdown with a fenced code block holding a custom DSL parsed by internal/ocaml/parser.go (not OSV):

Key	Meaning
`id`	`OSEC-YYYY-NN` identifier
`published` / `modified`	ISO 8601 timestamps
`aliases`	bracketed, space-separated CVE / GHSA IDs
`severity`	CVSS vector string
`severity_score`	numeric score with optional label
`affected`	package name + version constraints
`events`	git repository URLs with introduced/fixed commit hashes
`references`	typed URLs (report, fix, advisory, article)
`credits`	contributor credits with roles
`cwe`	bracketed, space-separated CWE identifiers

The Markdown body supplies the title (H1), the description, and any code blocks.

Affected constraints use affected: "package" {< "version" | (>= "v1" & < "v2")} with operators <, <=, >, >=, =, !=, | for OR, & for AND and parentheses for grouping.

ocaml.ExtractCVEID keys the row on the first CVE-YYYY-NNNN+ alias when the advisory has one, otherwise on the OSEC-* ID — so a CVE-numbered OCaml advisory merges into the shared CVE row while an OCaml-only one keeps its native ID.

Records produced

Condition	Records
Every stored advisory	`CVEMetadata` (`source="ocaml"`, `sourceFileHash` = file SHA1), `CVEDescription`, `CVEMetadataReferences` (`referenceSource="OCaml"`), `CVEProblemType` from `cwe`, `CVEAffected` + `CVEAffectedVersion` (collection URL `https://opam.ocaml.org`)
`severity` present	`CVEMetric` with the upstream vector and parsed score
Descriptions present	one derived `CVEMetric` (`containerType="vulnetix"`, `cvssV4_0`)
Always	`CVEAlias` edges via `db.InsertAliases` (CVE / GHSA aliases + same-`cveId` cross-source edges)
Every affected package	`Dependency`, `DependencyRegistry`, `PackageVersion`, `PackageVersionCVE` via `db.EnrichAffectedWithDependency`
Advisory contains a code block in a PoC language (`c`, `ocaml`, `python`, `bash`, `sh`, `ruby`, `go`, `rust`, `java`, `javascript`, `ml`)	`Exploit` (`source="ocaml"`, `exploitId` = the OSEC ID, `category="poc"`, `originalURL` = the advisory’s GitHub blob URL) + `ExploitCVE` junctions for every CVE alias, resolved against the known-CVE map
End of run	`BulkDataDumpTracker` row `ocaml_advisory`

The exploit heuristic is language-based, not behaviour-based: any fenced block tagged with one of those languages marks the advisory as carrying a PoC, so a reproducer snippet or a patch fragment can produce an Exploit row. Treat source='ocaml' exploit rows as “advisory contains code”, not as verified weaponisation.

Resume

Repo-level — the ocaml_advisory tracker SHA.
Per-file — db.LoadProcessedHashes("ocaml") skip set on CVEMetadata.sourceFileHash.

--force bypasses both.

Failure modes

Symptom	Cause
`repository unchanged, skipping processing`	tracker SHA equals HEAD — normal most weeks
`failed to load known CVE IDs`	the `LoadKnownCVEIDs` scan failed; the run continues but `ExploitCVE` junctions are limited
`skipping unmappable advisory`	the DSL block was absent or unparseable into an advisory
`soft deadline reached, stopping early`	`EXPECTED_DURATION_MINUTES − 10` elapsed. The tracker still advances, so unreached files wait for the next `--force`
`%d files errored` + exit 1	any errored file fails the run after the tracker has advanced

Flags

Flag	Default	Meaning
`--force`	`false`	Ignore the tracker SHA and the per-file hash skip set
`--batch-size`	`50`	Files per transaction
`--repo`	`/data/security-advisories`	Clone location
`--data-dir`	(repo + `/advisories`)	Override the advisory directory
`--no-pull`	`false`	Use the existing clone’s HEAD instead of pulling

Local run: just go-ocaml-git-backfill (append prod for production).

S3 Persistence

Archive path: ocaml/files/{sha256}/{filename} ✓
Quarantine path: failed-feeds/ocaml-git-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
Failure reasons emitted: store-error, tx-rollback

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[ocaml-git-processor] PROC -->|success| ARCHIVE[("S3: ocaml/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/ocaml-git-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.