cisco-cvrf-backfill

Status: Live (local-only — no ECS task, no EventBridge schedule, no Containerfile target) Source: Cisco Security Center CVRF via the getCvrfPaginationRecords.x JSON enumeration API Type: cvrf (CSAF-CVRF v1.2 XML per advisory) Source slug: cisco — the same rows cisco-cvrf-processor maintains Schedule: none. Run by hand: just go-cisco-cvrf-backfill prod.

Overview

The scheduled cisco-cvrf-processor short-circuits on an unchanged listing-page SHA256, which is what keeps the daily run cheap — but it also means a mapper change, a schema addition or a partially-failed run never gets retroactively applied to the ~5,200 historical Cisco advisories. This binary is the escape hatch: it skips the listing tracker entirely and walks every advisory URL, relying only on the per-advisory content hash for resume, so --force re-imports the full Cisco corpus.

It shares the parser and mapper with the scheduled processor (internal/cisco), so the record shape is identical; only the driving loop and the resume policy differ.

Property	Scheduled processor	This backfill
Listing-page SHA256 tracker	Consulted; exits early when unchanged, and updated on completion	Not read, not written
Per-advisory `sourceFileHash` resume	Yes	Yes (bypass with `--force`)
Deadline	Soft deadline from `EXPECTED_DURATION_MINUTES`, else 90 min (4 h with `--force`)	None — runs to completion, per `scripts/go-processors/AGENTS.md`
Write shape	One advisory per transaction, up to 3 retries	Batches of 50 in one transaction, per-item `SAVEPOINT sp_item` + up to 3 retries
Parse-failure handling	Stores a `state="REJECTED"` hash row so the advisory is not refetched	Quarantines the XML and counts an error; no hash row
S3 archive key	`cisco/advisories/{sha256}/{advisoryID}.xml` (non-canonical)	`cisco/files/{sha256}/{advisoryID}.xml` (canonical)

Records produced

Identical to the scheduled processor — see cisco-cvrf-processor § Data Mapping: CVEMetadata (cveId = Cisco advisory ID, source="cisco"), CVEDescription (document notes + remediations), CVEMetadataReferences, CVEAffected (vendor Cisco), CVEAlias (CVE IDs and Cisco Bug IDs — the bug IDs mint placeholder CVEMetadata rows), plus Artifact + Link + CVEMetadata.fileLinkId when the S3 upload succeeds.

The same CVSS gap applies: internal/cisco/types.go:113-118 decodes ScoreSet/BaseScore/Vector, while Cisco emits ScoreSetV3/BaseScoreV3/VectorV3, so no CVEMetric row is produced from upstream data. Re-running this backfill will not fill in Cisco CVSS until that struct is fixed — worth knowing before committing to a multi-hour run.

Failure modes

Failure	Behaviour
Listing enumeration fails	`task.errored` + `exit 1`
Advisory fetch fails	`RecordError`, counted, advisory skipped (no quarantine — there are no bytes)
`ParseCVRF` fails	XML quarantined to `failed-feeds/cisco-cvrf-backfill/{date}/parse-error/{file}.xml`, counted as an error
`MapToSourceData` returns nil	Counted as skipped
Item store fails after 3 attempts (with UTF-8 sanitisation retry)	Rolled back to `sp_item`, XML quarantined under `store-error`, counted; batch continues
Batch transaction fails	Entire batch counted as errored
Any error at end of run	`task.errored` to SNS and `exit 1` (so a partially successful backfill still exits non-zero)

Flags

Flag	Default	Description
`--force`	false	Ignore the per-advisory content hash and reprocess everything
`--limit`	0	Cap advisories processed (0 = all)

# Import everything new since the last run, against production
just go-cisco-cvrf-backfill prod

# Full re-import (all ~5.2k advisories)
just go-cisco-cvrf-backfill prod true

# Smoke test: 5 advisories against the local DB
just go-cisco-cvrf-backfill local false 5

Key files

File	Purpose
`cmd/cisco-cvrf-backfill/main.go`	Enumeration, batching, savepoint/retry logic, S3 archive + `Artifact`/`Link`
`internal/cisco/parser.go`	`FetchListingEntries`, `FetchAdvisory`, `ParseCVRF`
`internal/cisco/mapper.go`	`MapToSourceData` — CVRF document → `osv.CVESourceData`
`internal/processor/pipeline.go`	`StoreCVESourceData` — shared write path for all child tables

S3 Persistence

Archive path: cisco/files/{sha256}/{filename} ✓
Quarantine path: failed-feeds/cisco-cvrf-backfill/{YYYY-MM-DD}/{reason}/{filename} ✓
Failure reasons emitted: parse-error, store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[cisco-cvrf-backfill] PROC -->|success| ARCHIVE[("S3: cisco/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/cisco-cvrf-backfill/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.