S3 Persistence Compliance Matrix
This matrix audits the processors in cmd/ against the S3 Persistence Contract; the catalogue now spans ~168 processor commands. The contract requires: (1) archive successfully processed payloads to S3, (2) quarantine failed payloads to failed-feeds/{processor}/{date}/{reason}/, (3) maintain raw payload bytes at each failure point.
Compliance Status:
- Compliant: Both S3 archive and quarantine fully implemented (1 processor)
- Partial (Archive): S3 implemented, quarantine missing (17 processors — need failure wiring)
- Non-compliant: Neither implemented (95 processors)
Key Findings:
- Only osv-json-processor is fully compliant
- 17 processors upload to S3 but lack quarantine wiring
- 28 enrichment processors need record reconstruction (no raw file bytes available)
- 95 processors require both S3 and quarantine from scratch
Note — re-audit in progress: The catalogue has since grown to ~168 processor commands. The per-processor compliance breakdown below (and the compliant / partial / non-compliant / enrichment-only sub-counts above) predates those recent additions; newer processors are not yet classified and are being re-audited. Treat the counts as covering only the originally-audited set, not the full ~168.
Processor Compliance Audit
Grouped by type and sorted alphabetically within each group. Columns: S3 (archive active), Quarantine (quarantine implemented).
JSON Processors (29)
| Processor | Source | Unit | S3 | Q | Failure Paths |
|---|
| alpine-json-processor | alpine | file | ✓ | ✓ | parse-error |
| atlassian-json-processor | atlassian | record | ✓ | ✓ | store-error |
| aws-security-bulletins-json-processor | aws-security-bulletins | record | ✓ | ✓ | store-error |
| bugcrowd-json-processor | bugcrowd | record | ✓ | ✓ | store-error |
| cargo-json-processor | cargo | record | ✓ | ✓ | store-error |
| gpz-0day-itw-gsheet-processor | gpz | record | ✓ | ✓ | fetch-error, parse-error, store-error, map-error |
| certcc-json-processor | certcc | record | ✓ | ✗ | store-error |
| chainguard-json-processor | chainguard | file | ✓ | ✓ | parse-error |
| cisa-kev-json-processor | cisa_kev | record | ✓ | ✓ | parse-error, store-error |
| circl-json-processor | circl | record | ✓ | ✓ | store-error |
| confluent-json-processor | confluent | record | ✓ | ✓ | parse-error, store-error |
| cran-json-processor | cran | record | ✓ | ✓ | store-error |
| crowdsec-json-processor | crowdsec | record | ✓ | ✓ | parse-error, fetch-error |
| eukev-json-processor | eukev | record | ✓ | ✓ | parse-error, store-error |
| gitlab-json-processor | gitlab | file | ✓ | ✓ | parse-error, store-error |
| go-json-processor | go | record | ✓ | ✓ | store-error |
| hackerone-json-processor | hackerone | record | ✓ | ✓ | store-error |
| hex-json-processor | hex | record | ✓ | ✓ | store-error |
| maven-json-processor | maven | record | ✓ | ✓ | store-error |
| mitre-cve-json-processor | cve.org | file | ✓ | ✗ | schema-violation, parse-error, store-error |
| npm-json-processor | npm | record | ✓ | ✓ | store-error |
| nuget-json-processor | nuget | record | ✓ | ✓ | store-error |
| osm-json-processor | osm | file | ✓ | ✓ | parse-error, store-error |
| osv-json-processor | osv | enrich | ✓ | ✓ | schema-violation, parse-error |
| ovhcloud-json-processor | ovhcloud | enrich | ✓ | ✓ | parse-error |
| rocky-linux-json-processor | rocky_linux | file | ✓ | ✗ | parse-error, schema-violation |
| vulncheck-kev-json-processor | vulncheck | file | ✗ | ✗ | parse-error |
| vulncheck-nvd-json-processor | vulncheck | enrich | ✓ | ✗ | schema-violation, store-error |
| wolfi-json-processor | wolfi | file | ✗ | ✗ | parse-error |
CSAF/CVRF Processors (11)
| Processor | Source | Unit | S3 | Q | Failure Paths |
|---|
| abb-csaf-processor | abb | file | ✓ | ✓ | parse-error |
| certbund-csaf-processor | certbund | file | ✓ | ✓ | parse-error, schema-violation |
| cisco-cvrf-backfill | cisco | file | ✓ | ✓ | parse-error, store-error |
| cisco-cvrf-processor | cisco | file | ✓ | ✓ | parse-error, schema-violation |
| msrc-csaf-processor | msrc | file | ✓ | ✗ | parse-error, schema-violation |
| ncsc-nl-csaf-processor | ncsc-nl | file | ✓ | ✓ | parse-error |
| oracle-cpu-csaf-processor | oracle | file | ✓ | ✗ | (none wired) |
| ox-csaf-processor | open-xchange | file | ✓ | ✓ | parse-error |
| redhat-csaf-processor | redhat | file | ✓ | ✓ | parse-error |
| siemens-csaf-processor | siemens | file | ✓ | ✓ | parse-error, schema-violation |
| suse-csaf-processor | suse | file | ✓ | ✓ | parse-error, schema-violation |
| Processor | Source | Unit | S3 | Q | Failure Paths |
|---|
| acsc-rss-processor | acsc | file | ✗ | ✗ | parse-error, fetch-error |
| alas-rss-processor | alas | file | ✗ | ✗ | parse-error |
| binarly-rss-processor | binarly | file | ✗ | ✗ | parse-error |
| cert-at-rss-processor | cert-at | record | ✓ | ✓ | parse-error, store-error |
| cert-au-rss-processor | cert-au | record | ✓ | ✓ | parse-error, store-error |
| cert-ca-rss-processor | cert-ca | record | ✓ | ✓ | parse-error, store-error |
| cert-eu-rss-processor | cert-eu | record | ✓ | ✓ | parse-error, enrich-error |
| certfr-json-processor | cert-fr | file | ✓ | ✓ | parse-error |
| cert-it-rss-processor | cert-it | record | ✓ | ✓ | parse-error, store-error |
| cert-jp-rss-processor | cert-jp | record | ✓ | ✓ | parse-error, store-error |
| cert-lv-rss-processor | cert-lv | record | ✓ | ✓ | parse-error, store-error |
| cert-pt-rss-processor | cert-pt | record | ✓ | ✓ | parse-error, enrich-error |
| cert-se-rss-processor | cert-se | record | ✓ | ✓ | parse-error, store-error |
| cert-tw-rss-processor | cert-tw | record | ✓ | ✓ | parse-error, store-error |
| cert-ua-rss-processor | cert-ua | record | ✓ | ✓ | parse-error, store-error |
| cert-us-rss-processor | cert-us | record | ✓ | ✓ | parse-error, store-error |
| cxsecurity-rss-processor | cxsecurity | enrich | ✓ | ✗ | parse-error |
| gentoo-rss-processor | gentoo | enrich | ✗ | ✗ | parse-error |
| ghsa-rss-processor | ghsa | enrich | ✗ | ✗ | parse-error |
| isc-rss-processor | isc | enrich | ✗ | ✗ | parse-error |
| ncsc-fi-rss-processor | ncsc_fi | file | ✗ | ✗ | parse-error |
| nozomi-rss-processor | nozomi | file | ✗ | ✗ | parse-error |
| rhsa-rss-processor | rhsa | enrich | ✗ | ✗ | parse-error |
| sick-psirt-rss-processor | sick_psirt | file | ✗ | ✗ | parse-error |
| tailscale-rss-processor | tailscale | file | ✗ | ✗ | parse-error |
| tenable-rss-processor | tenable | file | ✗ | ✗ | parse-error |
| zdi-rss-processor | zdi | file | ✓ | ✗ | parse-error |
Git/Repository Processors (18)
| Processor | Source | Unit | S3 | Q | Failure Paths |
|---|
| almalinux-git-processor | almalinux | file | ✓ | ✓ | parse-error, store-error |
| anchore-adp-git-processor | anchore_adp | file | ✓ | ✓ | parse-error, store-error, tx-rollback |
| bitnami-git-processor | bitnami | file | ✓ | ✓ | parse-error, store-error |
| canonical-git-processor | canonical | file | ✓ | ✓ | parse-error, store-error |
| cleanstart-git-processor | cleanstart | file | ✓ | ✓ | parse-error, store-error |
| cnvd-git-processor | cnvd | file | ✓ | ✓ | parse-error |
| conan-git-processor | conan | record | ✓ | ✓ | store-error |
| docker-git-processor | docker | file | ✓ | ✓ | parse-error, store-error |
| drupal-git-processor | drupal | file | ✓ | ✓ | parse-error, store-error |
| fstec-git-processor | fstec | file | ✓ | ✓ | store-error |
| gemnasium-git-processor | gitlab | file | ✓ | ✓ | store-error, tx-rollback |
| ghsa-git-processor | github | file | ✓ | ✓ | parse-error, store-error |
| hsec-git-processor | haskell | file | ✓ | ✓ | store-error, tx-rollback |
| julia-git-processor | julia | record | ✓ | ✓ | store-error |
| ocaml-git-processor | ocaml | file | ✓ | ✓ | store-error, tx-rollback |
| ossfuzz-git-processor | ossfuzz | file | ✓ | ✓ | parse-error, store-error |
| pypi-git-processor | pypi | file | ✓ | ✓ | parse-error, store-error |
| rustsec-git-processor | rustsec | file | ✓ | ✓ | store-error, tx-rollback |
| vulnrichment-git-processor | cisa_adp | file | ✓ | ✓ | parse-error, store-error, tx-rollback |
| wizsec-git-processor | opencloud | file | ✓ | ✓ | parse-error, store-error, tx-rollback |
Fetch/Backfill/Archive Processors (33)
| Processor | Source | Unit | S3 | Q | Failure Paths |
|---|
| 0day-today-fetch-backfill | 0day-today | file | ✓ | ✓ | parse-error |
| adobe-security-fetch-processor | adobe | file | ✓ | ✓ | store-error |
| akamai-fetch-processor | akamai | file | ✓ | ✓ | fetch-error, parse-error, store-error |
| alpine-apk-fetch-processor | alpine-apk | record | ✓ | ✓ | store-error |
| aur-fetch-processor | aur | record | ✓ | ✓ | store-error |
| homebrew-fetch-processor | homebrew | record | ✓ | ✓ | store-error |
| broadcom-vmware-security-fetch-processor | broadcom-vmware | record | ✓ | ✓ | store-error |
| cert-be-fetch-processor | cert-be | file | ✓ | ✓ | parse-error, store-error |
| cert-il-fetch-processor | cert-il | record | ✓ | ✓ | store-error |
| cert-tw-fetch-processor | cert-tw | record | ✓ | ✓ | store-error |
| cloudflare-advisories-fetch-processor | cloudflare | record | ✓ | ✓ | store-error |
| cvelistv5-json-backfill | cve.org | file | ✓ | ✓ | parse-error, store-error |
| databricks-fetch-processor | databricks | file | ✓ | ✓ | parse-error, store-error |
| digitalocean-fetch-processor | digitalocean | file | ✓ | ✗ | fetch-error, parse-error |
| eol-json-processor | eol | record | ✓ | ✓ | parse-error, store-error |
| eol-fetch-processor | eol | record | ✓ | ✓ | store-error |
| epss-csv-backfill | epss | file | ✓ | ✓ | parse-error, store-error |
| euvd-json-backfill | euvd | record | ✓ | ✓ | store-error |
| euvd-json-processor | euvd | record | ✓ | ✓ | store-error |
| fastly-fetch-processor | fastly | record | ✓ | ✗ | fetch-error, parse-error |
| github-yara-fetch-processor | ? | enrich | ✓ | ✗ | fetch-error |
| hashicorp-discuss-fetch-processor | hashicorp-discuss | record | ✓ | ✓ | store-error |
| pwno-fetch-processor | ? | enrich | ✗ | ✗ | fetch-error |
| sap-patch-day-fetch-processor | sap | record | ✓ | ✓ | store-error |
| servicenow-kb-fetch-processor | servicenow | record | ✓ | ✓ | store-error |
| snowflake-fetch-processor | snowflake | record | ✓ | ✓ | store-error |
| snyk-fetch-processor | snyk | record | ✗ | ✗ | fetch-error, parse-error, store-error |
| tencent-blade-fetch-processor | tencent-blade | record | ✓ | ✓ | parse-error, store-error |
| twilio-fetch-processor | twilio | record | ✓ | ✓ | parse-error, store-error |
| variot-json-backfill | variot | file | ✗ | ✗ | parse-error |
| variot-json-processor | variot | file | ✗ | ✗ | parse-error |
| vercel-security-bulletins-fetch-processor | vercel | record | ✓ | ✓ | store-error |
| zoom-fetch-processor | zoom | record | ✓ | ✓ | store-error |
Enrichment-Only Processors (23)
⚠️ Special: Record reconstruction required (no raw file bytes). Database row serialization needed for quarantine.
| Processor | Source | S3 | Q | Failure Paths |
|---|
| cveprefix-json-processor | cve.org | ✗ | ✗ | enrich-error |
| enrich-coalition-cess-json | coalition-cess | ✗ | ✗ | enrich-error |
| enrich-docker-vex | docker | ✗ | ✗ | enrich-error |
| enrich-first-epss-json | (no CVEMetadata) | ✗ | ✗ | enrich-error |
| enrich-googleosi-json | (no CVEMetadata) | ✗ | ✗ | enrich-error |
| enrich-malware-exposure-fetch | (no CVEMetadata) | ✗ | ✗ | enrich-error |
| enrich-nse-fetch | (no CVEMetadata) | ✓ | ✗ | fetch-error, decode-error |
| enrich-nuclei-fetch | (no CVEMetadata) | ✓ | ✗ | fetch-error, enrich-error |
| enrich-references-fetch | (no CVEMetadata) | ✓ | ✗ | fetch-error, enrich-error |
| gcve-json-processor | ? | ✗ | ✗ | enrich-error |
| github-poc-processor | ? | ✗ | ✗ | fetch-error, enrich-error |
| mageia-json-processor | mageia | ✗ | ✗ | enrich-error |
| vulnetix-kev-processor | ? | ✗ | ✗ | enrich-error |
| wiz-json-processor | wiz | ✗ | ✗ | enrich-error |
Dashboard/Monitor/Other Processors (17)
| Processor | Source | Unit | S3 | Q |
|---|
| ai-discovered-vulns-runner | ? | file | ✗ | ✗ |
| ai-models-processor | ? | file | ✗ | ✗ |
| community-snort-processor | community-snort | record | ✓ | ✓ |
| community-suricata-snort-processor | community-suricata | record | ✓ | ✓ |
| emergingthreats-snort-processor | emerging_threats | file | ✗ | ✗ |
| misp-galaxy-json-processor | misp_galaxy | file | ✗ | ✗ |
| nist-nvd-modified-json-processor | nvd | file | ✗ | ✗ |
| nist-nvd-recent-json-processor | nvd | file | ✗ | ✗ |
| nist-nvd-year-json-processor | nvd | file | ✗ | ✗ |
| oss-malware-git-processor | oss-malicious-packages | file | ✓ | ✓ |
| scanner-processor | ? | enrich | ✗ | ✗ |
| scan-watchdog | ? | file | ✗ | ✗ |
| shadowserver-dashboard-processor | ? | enrich | ✗ | ✗ |
| summary-processor | ? | enrich | ✗ | ✗ |
| nvd-deepdive-processor | ? | enrich | ✗ | ✗ |
| vulncheck-nvd-cleanup | vulncheck | file | ✗ | ✗ |
| epss-score-prune | epss | n/a | n/a | n/a |
Summary
| Status | Count | Details |
|---|
| Compliant | 1 | osv-json-processor (S3 + quarantine fully wired) |
| Partial (Archive) | 17 | S3 active; quarantine wiring needed (abb, certbund, certfr, cert-jp, cisco, cxsecurity, mitre-cve, msrc, ncsc-nl, redhat, rocky, siemens, suse, zdi, 0day-today, github-yara, vulncheck-nvd) |
| Non-compliant | 95 | Neither S3 nor quarantine |
| Enrich-only | 23 | Require record reconstruction (database serialization for quarantine) |
Implementation Roadmap
- Phase 1 - Wire Quarantine (17 processors): Add
quarantineFailedFeed() calls at parse/schema/store errors - Phase 2 - Implement S3+Quarantine (95 processors): Copy patterns from osv-json-processor, establish consistent path format
- Phase 3 - Record-Based Enrichment (23 processors): Implement CVEMetadata row serialization for quarantine when database enrichment fails