Shared Business Logic Library

The processor business logic lives in Go under scripts/go-processors/internal/. Each cmd/<name>-processor binary is thin; it wires together the shared packages below. The earlier TypeScript library (src/shared/*.ts) has been fully removed — the whole pipeline is Go now.

Two kinds of package live under internal/:

  • Per-source parsers — one package per data source (alpineapk, atlassian, certfr, cisco, euvd, aur, cargo, hex, …). Each fetches and maps one feed into the shared osv.CVESourceData shape. There are ~110 of these; each is documented on its own processor page, not here.
  • Shared / cross-cutting packages — the infrastructure every processor depends on, catalogued below.

Persistence & data model

  • db — the largest shared package: pgxpool-based read/write pools (pool.go), the WithTx / WithBatchTx / WithTxRetry transaction helpers, and every Upsert* / Insert* / Load* query used to store CVEMetadata and its child tables. WithTxRetry replays a transaction that lost a row-lock race (SQLSTATE 55P03/40001/40P01), escalating lock_timeout across attempts. Also owns the canonical alias write path InsertAliases and the UTF-8 write-boundary scrubber SanitizeUTF8.
  • processor — the pipeline driver (Run, processBatch, storeAdvisory) that git/JSON processors reuse: walk advisories, resume by sourceFileHash, store in batched savepoints, then run post-commit S3 archive + AI enrichment.
  • osv — the shared CVESourceData advisory shape plus MapAdvisory, the OSV YAML/JSON parsers, and the malicious-package skip rules.
  • cvelistv5 — parses and emits CVE List v5 JSON records, including the x_vulnetix extension objects written into the exported archives.
  • export — builds a complete cvelistv5 record from database rows; shared by the cveprefix and gcve archive processors.
  • recordcache — filesystem-backed keyed JSON cache used for per-record resume between archive runs.
  • s3client — rotating S3 client factory and the Uploader.Archive / Uploader.Quarantine helpers that implement the S3 persistence contract.

AI enrichment & intelligence

  • aienrich — Cloudflare AI Gateway inference passes (affected routines, ATT&CK mapping, CWE inference, TreeSitter queries, GHSA PoC). Best-effort and time-boxed per run — see aienrich.
  • aimalgate — an LLM “second opinion” gate that asks whether a flagged package is genuinely malicious, used to suppress false positives.
  • malscan — adapts the malscan-engine IOC-scan capability for the ecosystem malware processors.
  • actorintel / malwareactor — discover and persist malware-author threat actors and their attributions (MalwareThreatActor / MalwareAttribution).
  • hijack — wires the malscan-engine cross-registry ownership/hijack rules into the ecosystem processors.
  • legitmaintainer — cross-ecosystem allowlist of known-legitimate maintainers, a guard against branding real maintainers as attackers.
  • iocfilter — drops indicators that are not real IOCs (source-maps, test fixtures, dist bundles) so they never reach the IOC tables.
  • pkgregistry — resolves an (ecosystem, packageName) pair to its upstream source repository and fetches source snippets for LLM grounding.
  • enrichment — shared logic for the standalone enrich-* ECS tasks (EPSS, CESS, Google OSI, Nuclei, references, GitHub repository enrichment).

Run lifecycle & infrastructure

  • notify — publishes the processor lifecycle events (task.started, task.completed, task.errored, task.overtime, task.no_work, task.zombie_detected) to SNS, and Finalize, the systemic-vs-partial error policy that keeps a run green for a handful of transient per-record failures.
  • emf — emits CloudWatch Embedded Metric Format lines so monitoring can alarm on per-outcome counts without extra infrastructure.
  • rundeadline — derives a processor’s soft deadline from EXPECTED_DURATION_MINUTES (budget minus a margin of 10 %, clamped to [2 min, 10 min]). Processors stop starting new work at the soft deadline so they finish before the overtime watcher fires.
  • zombieguard — startup detection of stale / previous-date sibling ECS tasks of the same family; stops them and reports task.zombie_detected.
  • httpclient — shared browser-like HTTP client and header helpers (HTTP/2 by default, an HTTP/1.1 variant for servers that reset HTTP/2 streams).
  • githubapi — thin google/go-github wrapper for “fetch a file at a ref”; its rate-limit backoff refuses to sleep past the run’s budget (ErrRateLimitBudget) so a rate-limited call cannot cause an overtime.
  • retry — shared exponential-backoff-with-jitter retry helper (Do, DoValue) used to absorb transient upstream failures.

Scoring, identifiers & CRIT

  • cvss — CVSS v2/v3.0/v3.1/v4.0 vector parsing, validation, and severity.
  • cveid — helpers for recognising and normalising CVE-YYYY-N identifiers.
  • Cloud resource locators, CPE, and PURL generation are covered in the schemas pages.
  • critmatch / critpublisher / critutil — the CRIT ranking, in-process draining, and shared building blocks.