S3 Persistence Contract

Every processor MUST persist the raw feed payload to S3 — both on success and on failure — so we never lose source data and can audit/replay any processing decision.

This rule applies to every processor in scripts/go-processors/cmd/, regardless of source type (json, csaf, cvrf, rss, fetch, git, snort, gsheet, csv, enrich-*). It applies to scheduled ECS runs and to local backfill runs alike. Skip the upload only when S3_BUCKET_NAME is unset (local development without AWS credentials).

See the Compliance Matrix for the current status of every processor.

Two paths, one bucket

The same S3_BUCKET_NAME bucket holds both:

• {source} is the canonical short slug used in CVEMetadata.source (e.g. osv, mitre-cve, vulncheck-nvd, ghsa, siemens, certfr).
• {processor} is the cmd/ directory name (e.g. osv-json-processor).
• {sha256} is the lowercase hex content hash of the raw payload.
• {filename} is the original feed-relative path (preserve nested directories from the archive — vulns/PYSEC-2024-123.yaml, not just the basename).

Reason tags (kebab-case)

Use these tags consistently so the quarantine bucket stays queryable:

Helpers

Use the shared methods on s3client.Uploader — do not hand-roll either path in a processor.

u, err := s3client.NewUploader(ctx, os.Getenv("S3_BUCKET_NAME"), "application/json", logger)
// ...
u.Archive(ctx, "osv", fileSHA256, zf.name, zf.data)                            // success path
u.Quarantine(ctx, "osv-json-processor", "schema-violation", zf.name, zf.data)  // failure path

Both methods are no-ops when the receiver is nil, so processors can hold an *Uploader field that is nil when S3 is not configured without scattering nil checks at every call site.

Helpers live in:

• scripts/go-processors/internal/s3client/uploader.go — Uploader.Archive, Uploader.Quarantine
• scripts/go-processors/internal/s3client/recordsink.go — MarshalRecord, ArchiveRecord, QuarantineRecord for record-based (non-file) processors
• scripts/go-processors/internal/s3client/feedsink.go — NewFromEnv, PipelineHooks for the processor pipeline
• scripts/go-processors/internal/enrichment/s3upload.go — enrichment-specific helpers

Idempotency and batching

• The archive path is content-addressed by {sha256} so re-uploading the same payload is a free no-op on S3.
• Successful archive uploads SHOULD be recorded on the per-source file tracker row (e.g. OsvFile.r2Path, OsvFile.r2StoredAt). Resume runs consult these columns and skip re-uploading already-archived files.
• Archive uploads run inside the existing batch-processing concurrency (typically 4 workers, 200 files per batch). Do not introduce a separate uploader goroutine pool — reuse the batch worker that just stored the record.
• Quarantine uploads run synchronously at the failure site, before logging the warn/error line. Best-effort: a failed Quarantine call MUST NOT block the pipeline.

Processors whose unit-of-work is not a file

Some processors operate on records that did not arrive as standalone files (RSS feed entries, paginated JSON arrays, CSV rows, individual records from a single bulk download). In that case:

1. Synthesise a stable {filename} from the record identity: {advisoryId}.json or {cveId}-{source}.json — never a UUID, never an index. The same record on a re-run MUST produce the same filename.
2. Re-serialise the record to canonical JSON (encoding/json with sorted keys via the standard library) and use that as the payload bytes.
3. Compute SHA256 over those payload bytes for the archive path.

The contract is the same: archive on success, quarantine on failure.

Compliance audit

The Compliance Matrix audits every processor. Every PR that adds a new processor MUST add a row. Every PR that adds a new failure path in an existing processor MUST keep the matrix accurate.