Anchore ADP Git Processor

Status: Live Source: anchore/cve-data-enrichment (data/anchore/{YYYY}/CVE-*.json) Type: git (cloned at runtime — the image is FROM scratch with no baked data) Source slug: anchore_adp (underscore, not hyphen) Schedule: Runs daily at 07:00 UTC (cron(0 7 * * ? *)), 256 CPU / 512 MB, expected_duration_minutes = 45.

Overview

Processes Anchore ADP (Authorized Data Publisher) CPE enrichment JSON files into CVEMetadata and related tables. Anchore provides CPE configurations for CVEs not yet analyzed by NVD, enabling earlier vulnerability matching against software inventories.

Data Source

  • Repository: https://github.com/anchore/cve-data-enrichment.git
  • Format: Custom JSON (NOT CVE Record v5 — uses additionalMetadata + adp top-level keys)
  • Structure: data/anchore/{YYYY}/CVE-{YYYY}-{N}.json
  • Volume: ~20K+ files (~88 MB)

ID Scheme

Records are stored with source="anchore_adp" (the constant in internal/anchoreadp/mapper.go) and remapped IDs to avoid collision with other sources:

  • Stored cveId: ANCHORE-{YYYY}-{N} (e.g., ANCHORE-2024-0072)
  • Alias: Original CVE-{YYYY}-{N} stored in CVEAlias table
  • Rationale: Same CVE ID may exist in cvelistV5, vulnrichment, and other sources with different enrichment data

Processing Flow

  1. Clone/pull the anchore-adp repository via processor.PullOrClone() — the image bakes no data, so the first run in a fresh task clones ~88 MB
  2. Check BulkDataDumpTracker (source anchore_adp_advisory) — skip if HEAD SHA unchanged
  3. Walk data/anchore/ directory for CVE-*.json files
  4. processor.DetectChangedFiles narrows to files touched in the last 3 days (full scan on first run or --force)
  5. Load existing sourceFileHash map for resume (db.LoadProcessedHashes)
  6. Local runs only: resume from the .repo/anchore-adp statefile checkpoint
  7. Batch-process files (500 per transaction, one SAVEPOINT sp per record):
    • Parse custom JSON format via anchoreadp.ParseRecord()
    • Remap CVE ID to ANCHORE ID
    • Store metadata, descriptions, references, affected products + versions
    • Insert CVE alias linking ANCHORE → CVE via db.InsertAliases
  8. Archive each stored record’s raw JSON to S3, quarantine failures
  9. Update tracker with HEAD SHA (written even when some files errored, so a transient failure cannot force a full re-scan next run)

Unlike the OSV-pipeline git processors this command does not initialise internal/aienrich, so none of the LLM-backed passes (CWE inference, ATT&CK mapping, TreeSitter queries) run for anchore_adp records.

Data Mapping

Anchore ADP FieldTarget TableNotes
additionalMetadata.cveIdCVEAliasOriginal CVE ID as alias
ANCHORE-YYYY-NCVEMetadata.cveIdPrimary key
additionalMetadata.descriptionCVEDescriptioncontainerType=“adp”, lang=“en”
additionalMetadata.references[]CVEMetadataReferencesreferenceSource=“anchore_adp”, type=“url”
upstream.datePublishedCVEMetadata.datePublishedRFC3339/date layouts; falls back to dateUpdated, then to the year encoded in the CVE id, then to 0
upstream.dateUpdatedCVEMetadata.dateUpdated
additionalMetadata.descriptionCVEMetadata.titleTruncated to 500 runes
adp.affected[0].vendor / .productCVEMetadata.affectedVendor / affectedProductFirst affected entry only
adp.affected[]CVEAffectedcontainerType=“adp”
adp.affected[].cpes[]CVEAffected.cpesJSON array
adp.affected[].versions[]CVEAffectedVersionversion, lessThan, status, versionType
Full JSONCVEMetadata.rawDataJSONComplete file contents
SHA1(file)CVEMetadata.sourceFileHashResume tracking

Key Differences from Vulnrichment

  • Not CVE Record v5: Custom JSON format with additionalMetadata + adp top-level keys
  • No CNA container: Only ADP enrichment data (CPEs, versions)
  • No metrics/problemTypes: Anchore provides CPE configurations, not CVSS/CWE
  • Standalone parser: Cannot reuse cvelistv5.StoreRecordWithSource()

Deployment

  • ECS: go-anchore-adp-git-processor task, daily at 07:00 UTC
  • Resources: 256 CPU / 512 MB (smaller repo than vulnrichment)
  • Soft deadline: expected_duration_minutes = 45 (terraform/go-schedules.tf and scripts/task-manager.toml); the run stops dispatching new batches 10 minutes before it
  • Local: just go-anchore-adp-git-backfill

S3 path deviations

Two things about the archive key differ from the generated block below (which is derived from the compliance matrix, not from the code):

  • The digest segment is the SHA-1 returned by anchoreadp.FileHash (internal/anchoreadp/parser.go:23), reused from sourceFileHash — not the SHA-256 the S3 Persistence Contract specifies.
  • The prefix is the literal source slug anchore_adp (underscore), because main.go passes the source constant. scripts/build-docs-s3-sections.py:120 hyphenates underscore slugs, so the generated block below shows anchore-adp/…, which is not the key actually written.

Actual key: anchore_adp/files/{sha1}/{repo-relative-path}.

S3 Persistence

  • Archive path: anchore-adp/files/{sha256}/{filename} ✓
  • Quarantine path: failed-feeds/anchore-adp-git-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
  • Failure reasons emitted: parse-error, store-error, tx-rollback

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[anchore-adp-git-processor] PROC -->|success| ARCHIVE[("S3: anchore-adp/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/anchore-adp-git-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.