rustsec-git-processor

Status: Live Source: rustsec/advisory-db Type: git (shallow clone baked into the container image at /data/advisory-db) Source slug: rustsec Schedule: Runs daily at 04:00 UTC (cron(0 4 * * ? *)), 256 CPU / 512 MB, expected_duration_minutes = 45.

Overview

RustSec is the Rust ecosystem’s own advisory database and the upstream that cargo audit consumes. Its advisories are Markdown files with a TOML frontmatter block, not OSV YAML, so they cannot go through the shared internal/osv mapper — internal/rustsec parses the frontmatter and the Markdown body separately.

Only files whose path contains /crates/ or /rust/ are processed; root-level Markdown (README, CONTRIBUTING) is filtered out (cmd/rustsec-git-processor/main.go:148-154).

The advisory id is RUSTSEC-YYYY-NNNN. When the frontmatter aliases list carries a CVE-YYYY-NNNN, that CVE becomes the primary cveId and the RUSTSEC id is demoted to a CVEAlias edge; otherwise the RUSTSEC id is the primary id (internal/rustsec/mapper.go:33-36).

Two things RustSec provides that the OSV mirror of the same data does not:

• Rust-specific weakness mapping — internal/rustsec/cwe.go derives CWE ids from the advisory categories and keywords (mapper.go:92-102), so advisories that never received an NVD CWE still get one.
• Proof-of-concept code — RustSec advisories frequently inline a PoC in the Markdown body. rustsec.ExtractPoCs lifts those fenced code blocks into an Exploit row (main.go:336-344), which is what promotes a Rust advisory from “known” to “weaponised” in exploit-maturity scoring.

Records produced

Scheduling and resume

Three independent resume layers:

1. Repo-level — HEAD SHA is compared to BulkDataDumpTracker.sha256 for rustsec_advisory; an unchanged repo exits 0 with NoWork (main.go:104-116).
2. Delta window — processor.DetectChangedFiles narrows the walk to files git reports as recently changed. Skipped on the first run and under --force (main.go:158-162).
3. Per-file — db.LoadProcessedHashes(source="rustsec") gives cveId → sourceFileHash; a file whose SHA-1 matches is counted unchanged (main.go:318-321).

Batches of 200 files share one transaction, each file inside its own SAVEPOINT sp_file so a single bad advisory cannot poison the batch (main.go:292-361).

S3 path deviation

The generated S3 section below states the contract path. In practice the {sha256} slot of the archive key is filled with the SHA-1 file digest already computed for resume, not a SHA-256 (main.go:303-304, 368), so real keys look like rustsec/files/{sha1}/crates/foo/RUSTSEC-2024-0001.md. Quarantine reasons are store-error (per-file failure) and tx-rollback (batch transaction failed after files were staged) — a parse failure is reported as store-error because the per-file closure does not distinguish it.

Failure modes

• LoadKnownCVEIDs startup cost — the Exploit→CVE junction needs the set of known CVE ids, loaded with SELECT "cveId","source" FROM "CVEMetadata" WHERE "cveId" LIKE 'CVE-%' (main.go:133). On production that is a multi-million row scan and adds a minute or more before the first batch starts.
• Tracker is written even on a partial run — db.UpsertTracker runs before the errored-file check (main.go:235-254), so a run that errored on some files still records the HEAD SHA. Those files are retried only when the repo next changes; the per-file sourceFileHash mismatch is what picks them up.
• Tracker is also written after a soft-deadline truncation — the batch loop breaks at main.go:196-199 and falls straight through to UpsertTracker, with no equivalent of the ctx.Err() guard the shared pipeline has. The next scheduled run sees an unchanged SHA and exits NoWork, and because the delta window only looks back 3 days the unreached files can be missed entirely. Only ~1,200 advisories exist, so a 35-minute budget makes this unlikely today, but the guard is absent.
• No AI enrichment — this processor stores through processor.StoreCVESourceData directly and never constructs an aienrich.Enricher, so CWE/ATT&CK/TreeSitter inference does not run here.

Flags

Local run: just go-rustsec-git-backfill (add prod to target production). The recipe unsets EXPECTED_DURATION_MINUTES, so a backfill carries no soft deadline.

S3 Persistence

• Archive path: rustsec/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/rustsec-git-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: store-error, tx-rollback

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.