canonical-git-processor

Status: Live Source: canonical/ubuntu-security-notices (osv/usn/USN-*.json) Type: git (partial clone + osv/usn sparse checkout baked into the container image at /data/ubuntu-security-notices, refreshed at runtime with the real git binary) Source slug: canonical Schedule: Runs daily at 03:30 UTC (cron(30 3 * * ? *)), 1024 CPU / 7168 MB / 30 GiB ephemeral, expected_duration_minutes = 120.

Overview

Canonical publishes Ubuntu Security Notices (USN) in OSV format. A single USN usually fixes several CVEs across several Ubuntu releases, and the useful payload is the per-release deb package name and fixed version — the authoritative answer to “is this Ubuntu host affected”, which the upstream CVE record cannot give.

The binary is a thin main over the shared pipeline (internal/processor.Run) with the shared OSV mapper (internal/osv.MapAdvisory):

processor.PullOrClone refreshes /data/ubuntu-security-notices. Unlike the other git processors this image is Alpine (not scratch) and ships the real git binary, so the baked partial-clone / sparse-checkout config survives and the runtime pull only transfers the osv/usn delta. The full checkout would be ~9 GB; the osv/usn cone is ~220 MB.
HEAD is compared to BulkDataDumpTracker.sha256 for source canonical_advisory. Unchanged and no --force → NoWork, exit 0.
processor.DetectChangedFiles(..., "osv/usn") restricts delta detection to the same cone, so git log does not have to consider osv/cve/. Full scan on first run (no tracker) or --force.
osv/usn/USN-*.json is walked, parsed as OSV JSON, skipped when CVEMetadata.sourceFileHash equals the file’s SHA-1.
Batches of 200 files per transaction, one SAVEPOINT sp_file per file.

cveId is the first CVE-YYYY-NNNN in aliases[]; when the USN references no CVE the USN-… id is used verbatim. Every other CVE in the alias list becomes a CVEAlias edge, which is why source='canonical' also holds a large number of placeholder CVEMetadata shells — the alias table’s foreign key mints a row under the aliased (cveId, source) pair. Those shells are expected and must be excluded from any field-coverage measurement.

Records produced

Condition	Records
Every stored advisory	`CVEMetadata` (`source="canonical"`, `sourceAdvisoryRef` = USN id, `sourceFileHash` = SHA-1 of the file)
USN has `summary`/`details`	`CVEDescription`
USN has `references[]`	`CVEMetadataReferences` (`referenceSource="Canonical"`)
USN has `severity[]`	`CVEMetric`; plus one derived `containerType="vulnetix"` `cvssV4_0` metric from the description text. Most USNs carry no CVSS — Canonical publishes its own priority label, not a vector.
USN has CWE ids	`CVEProblemType`
USN has `affected[]`	`CVEAffected` + `CVEAffectedVersion` (one group per Ubuntu release)
Always (even with an empty alias list)	`CVEAlias` via `db.InsertAliases` — USN↔CVE edges, every CVE-to-CVE edge in the notice, plus the same-`cveId` cross-source edges
Affected package resolves to an ecosystem	`Dependency`, `DependencyRegistry`, `PackageVersion`, `PackageVersionCVE`, and `GitHubRepoDependency` when the advisory names a known GitHub repo (`db.EnrichAffectedWithDependency`)
Clean run (no file errors, not cancelled)	`BulkDataDumpTracker` row `canonical_advisory` — `sha256` = git HEAD
`PIX_INFERENCE_ENABLED` set (ECS only)	`CVEAttackTechnique` / `CVEAttackMitigation` / `CVEAttackDetection` / `CVEAttackD3fend`, `CVEProblemType` (`derivedBy="vulnetix"`), `CVETreeSitter`, `PixLog` via `internal/aienrich`

Resume

Repository level — BulkDataDumpTracker.sha256 (source canonical_advisory) holds the last cleanly processed HEAD. It is not advanced when any file errored or when the run was cancelled, so a run that always errors on at least one file leaves the tracker permanently stale.
File level — CVEMetadata.sourceFileHash (SHA-1), loaded once via db.LoadProcessedHashes with a 5-minute budget. This is the largest single source in the catalogue, so the resume load is the run’s dominant startup cost.

Failure modes

Symptom	Cause / guard
`repository unchanged, skipping processing`	HEAD matches the tracker.
`failed to load resume hashes, processing all files`	Read replica exceeded the 5-minute budget; the run re-stores everything it walks.
`file errors present, skipping tracker update to retry next run`	The tracker stays stale until a fully clean run. Check the tracker’s `lastProcessedAt` against `max(lastFetchedAt)` on `CVEMetadata` to see whether this is chronic.
`soft deadline reached, stopping early`	120-minute budget nearly spent; partial progress commits, tracker not advanced.
OOM with no summary line	The sparse cone plus resume map is the memory driver; this is why the task is sized 7168 MB rather than the 512 MB default.

Flags

Flag	Default	Purpose
`--force`	`false`	Ignore the tracker SHA and the per-file hash skip set.
`--batch-size`	`200`	Files per transaction.
`--repo`	`/data/ubuntu-security-notices`	Clone path.
`--data-dir`	`<repo>/osv/usn`	Override the walked directory.

Local: just go-canonical-git-backfill (unsets EXPECTED_DURATION_MINUTES, and the AI Gateway vars unless AIENRICH=true).

S3 Persistence

Archive path: canonical/files/{sha256}/{filename} ✓
Quarantine path: failed-feeds/canonical-git-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
Failure reasons emitted: parse-error, store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[canonical-git-processor] PROC -->|success| ARCHIVE[("S3: canonical/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/canonical-git-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.