hsec-git-processor

Status: Live Source: haskell/security-advisories (git) Type: git (repo baked by the hsec-git-data stage; the final image is alpine with a git binary so PullOrClone uses system git at startup) Source slug: haskell Schedule: Runs weekly on Mondays at 04:00 UTC (cron(0 4 ? * MON *)). CPU 256 / memory 512 MB / expected_duration_minutes = 45.

Overview

Hackage has no CNA and no OSV export. haskell/security-advisories (the HSEC-* namespace, maintained by the Haskell Security Response Team) is the only authoritative record of Haskell package and GHC-component vulnerabilities. It is also the source of the HSEC↔CVE alias edges, without which an HSEC- identifier quoted in a Haskell advisory or cabal audit output resolves to nothing in the VDB.

Volume is small (tens of advisories), so this is a coverage-completeness feed: it exists so a cabal.project.freeze gets a real answer instead of silence.

Each run:

  1. Refreshes the repo with processor.PullOrClone on /data/haskell-advisories (alpine image, system git). --no-pull reads the baked HEAD instead.
  2. Short-circuits on an unchanged HEAD against BulkDataDumpTracker.sha256 for source hsec_advisory; a match without --force reports task.nowork and exits 0.
  3. Loads the resume skip set — db.LoadProcessedHashes("haskell").
  4. Walks advisories/published/ recursively for HSEC-*.md, then applies the 3-day changed-file filter when a tracker exists.
  5. Loads known CVE IDs — db.LoadKnownCVEIDs reads every (cveId, source) pair matching CVE-% so ExploitCVE junctions attach to the sources that actually hold the CVE. This scan dominates start-up time.
  6. Stores in transactions of --batch-size (default 50), one SAVEPOINT per file.

Advisory format

Each advisory is a Markdown file whose leading fenced block is TOML:

SectionFields
[advisory]id (HSEC-YYYY-NNNN), cwe (integers), capec, keywords, aliases, related
[[references]]type, url
[[affected]]package, ghc-component, os, cvss, declarations, nested [[affected.versions]] ranges

The Markdown body supplies the title (H1), the description, body links, and any code blocks.

hsec.ExtractCVEID keys the primary row on the first CVE-YYYY-NNNN+ alias when present, otherwise on the HSEC-* ID.

datePublished comes from git. The TOML carries no publication date, so the processor calls processor.FileFirstCommitDate and uses the advisory file’s first-commit timestamp. When git history is unavailable (shallow clone boundary) the field is left unset and the UpsertCVEMetadata earliest-wins guard preserves whatever another source already knows.

One row per affected package

hsec.MapAdvisories returns one CVESourceData per [[affected]] entry, not one per file:

  • affected index 0 takes the primary ID (CVE alias, or the HSEC-* ID);
  • every further index is minted as a synthetic clone id "{HSEC-ID}-{n+1}" (e.g. HSEC-2025-0006-2);
  • all clone ids are cross-linked to each other as aliases.

So a multi-package Haskell advisory produces several CVEMetadata rows under source='haskell', and identifiers of the form HSEC-YYYY-NNNN-2 are Vulnetix-minted, not upstream identifiers.

Records produced

ConditionRecords
Every stored recordCVEMetadata (source="haskell", sourceFileHash = file SHA1), CVEDescription, CVEMetadataReferences (referenceSource="Haskell"), CVEProblemType from [advisory].cwe, CVEAffected + CVEAffectedVersion (collection URL https://hackage.haskell.org)
[[affected]].cvss presentCVEMetric with the upstream vector
Descriptions presentone derived CVEMetric (containerType="vulnetix", cvssV4_0)
AlwaysCVEAlias edges via db.InsertAliases (aliases, related, sibling clone ids, plus same-cveId cross-source edges)
Every affected packageDependency, DependencyRegistry, PackageVersion, PackageVersionCVE via db.EnrichAffectedWithDependency
Row keyed on an HSEC- idGcveIssuance (gcveId = "GCVE-110-" + cveId, GNA 110, year/sequenceNumber parsed from the HSEC id) + GcveAlias rows for every alias and for the upstream HSEC id
Advisory contains a code block in a PoC language (c, ocaml, python, bash, sh, ruby, go, rust, java, javascript, ml, haskell)Exploit (source="haskell", exploitId = the HSEC id, category="poc", originalURL = the advisory’s GitHub blob URL) + ExploitCVE junctions for every CVE alias
End of runBulkDataDumpTracker row hsec_advisory

The exploit heuristic is language-based, not behaviour-based — any fenced block tagged with one of those languages marks the advisory as carrying a PoC. Treat source='haskell' exploit rows as “advisory contains code”, not as verified weaponisation.

Resume

  • Repo-level — the hsec_advisory tracker SHA.
  • Per-file — db.LoadProcessedHashes("haskell") skip set, matched against the primary id only.

--force bypasses both.

Failure modes

SymptomCause
repository unchanged, skipping processingtracker SHA equals HEAD — normal most weeks
git date extraction failedfirst-commit lookup failed; datePublished is left to the DB guard
failed to load known CVE IDsthe scan failed; ExploitCVE junctions are limited
gcve issuance failedlogged as a warning — the CVE rows still land, only the GCVE identifier is missing
soft deadline reached, stopping earlyEXPECTED_DURATION_MINUTES − 10 elapsed. The tracker still advances, so unreached files wait for the next --force
%d files errored + exit 1any errored file fails the run after the tracker has advanced

Flags

FlagDefaultMeaning
--forcefalseIgnore the tracker SHA and the per-file hash skip set
--batch-size50Files per transaction
--repo/data/haskell-advisoriesClone location
--data-dir(repo + /advisories/published)Override the advisory directory
--no-pullfalseUse the existing clone’s HEAD instead of pulling

Local run: just go-hsec-git-backfill (append prod for production).

S3 Persistence

  • Archive path: haskell/files/{sha256}/{filename} ✓
  • Quarantine path: failed-feeds/hsec-git-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
  • Failure reasons emitted: store-error, tx-rollback

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[hsec-git-processor] PROC -->|success| ARCHIVE[("S3: haskell/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/hsec-git-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.