Wiz Security Open CVDB Git Processor — Design

Overview

Processes YAML cloud vulnerability advisories from the Wiz Security Open Cloud Vulnerability Database (github.com/wiz-sec/open-cvdb) and creates or enriches CVEMetadata rows (source=opencloud) with minted CLOUD-YYYY-N identifiers.

This processor complements the existing opencloud-rss-processor which ingests the same dataset via RSS feed. YAML files contain richer structured data: severity, affected platforms/services, discoverer metadata, Piercing Index vectors, CVE aliases, and remediation guidance.

When an advisory slug matches an existing GcveAlias entry (from the RSS processor), the existing CLOUD record is enriched rather than duplicated.

Data Source

Property	Value
Repository	`https://github.com/wiz-sec/open-cvdb.git`
Auth	None — fully public
Format	YAML files in `vulnerabilities/*.yaml`
Items	~282 cloud vulnerability advisories
Baked	Yes — shallow clone baked into container image (~37 MB)

Parsing

Field	YAML Key	Type
Title	`title`	string (required)
Slug	`slug`	string (required)
CVE IDs	`cves`	[]string (may be empty)
Affected Platforms	`affectedPlatforms`	[]string
Affected Services	`affectedServices`	[]string
Image	`image`	string (optional)
Severity	`severity`	enum: critical/high/medium/low/null
Piercing Index	`piercingIndexVector`	map (optional)
Discoverer	`discoveredBy`	object: name/org/domain/twitter
Published	`publishedAt`	“YYYY/mm/dd” → Unix seconds
Disclosed	`disclosedAt`	“YYYY/mm/dd” (optional)
Exploitability	`exploitabilityPeriod`	string (optional)
Known ITW	`knownITWExploitation`	bool/null
Summary	`summary`	string (multiline)
Remediation	`manualRemediation`	string (optional)
Detection	`detectionMethods`	string (optional)
Contributor	`contributor`	string (GitHub URL)
References	`references`	[]string (URLs)
Status	`entryStatus`	enum: Stub/Draft/Finalized

ID Generation

Shares the CLOUD-{year}-{seq} namespace with opencloud-rss-processor:

On startup, query all GcveAlias entries where aliasSource='opencloud' to build slug → cveID mapping
For each YAML file, if the slug exists in the map, reuse that CLOUD ID (enrichment mode)
If the slug is new, mint a fresh CLOUD-YYYY-N using the shared GcveIssuance sequence

Storage

No new tables or columns. All tables already exist. Everything below goes through processor.StoreCVESourceData, so each affected platform entry additionally feeds db.EnrichAffectedWithDependency (Dependency, DependencyRegistry, PackageVersion, PackageVersionCVE).

wizsec.MapToSourceData always sets DatePublished to wizsec.ParseDate(publishedAt), which returns 0 for a missing or non-YYYY/MM/DD date — so an advisory without a parseable publishedAt lands as datePublished = 0 rather than NULL (internal/wizsec/parser.go:29-39, mapper.go:118).

Table	Rows inserted
`CVEMetadata`	One per advisory; `source="opencloud"`, `cveId="CLOUD-YYYY-N"`
`CVEDescription`	1-3 per advisory: summary, remediation, detection
`CVEMetadataReferences`	One per `references[]` entry + canonical advisory URL
`CVEAffected`	One per `affectedPlatforms[]` entry
`GcveIssuance`	One per new advisory; `gcveId="GCVE-110-CLOUD-YYYY-N"`
`GcveAlias`	One per new advisory; links slug to GCVE ID
`CVEAlias`	One per `cves[]` entry; links CLOUD ID to external CVE IDs
`Artifact`	One per advisory (S3 upload of raw YAML)
`Link`	One per advisory (links Artifact to CVEMetadata)
`CVEMetric`	One derived Vulnetix CVSS v4 row per advisory that has a description (`containerType="vulnetix"`), emitted by the shared `storeAdvisory` because `MapToSourceData` sets neither `Metrics` nor `SuppressDerivedMetrics` (`internal/processor/pipeline.go:528-555`)
`Dependency`, `DependencyRegistry`, `PackageVersion`, `PackageVersionCVE`	Per affected platform, via `db.EnrichAffectedWithDependency`
`BulkDataDumpTracker`	`source="wizsec_advisory"`, `sha256` = git HEAD SHA, written only when the run had zero errors (`main.go:305-311`)

The advisory’s own severity enum and its piercingIndexVector are not mapped to a CVEMetric — they survive only inside rawDataJSON (internal/wizsec/mapper.go:132-155), which is why CVEMetadata.vectorString is NULL for every opencloud row.

Incremental Strategy

Resume uses per-file SHA1 hashes stored in CVEMetadata.sourceFileHash. On startup, load all hashes for source='opencloud' via LoadProcessedHashes(). Per file: if the slug maps to an existing cveID and the file hash matches, skip. Tracker (BulkDataDumpTracker) stores the git HEAD SHA to skip entire runs when the repo is unchanged.

Flags

Flag	Default	Description
`--repo`	`/data/open-cvdb`	Path to git repository
`--data-dir`	``	Override: process files from this directory
`--no-pull`	false	Skip git pull, use baked data
`--force`	false	Reprocess all advisories
`--batch-size`	200	Files per transaction batch
`--limit`	0	Maximum advisories to process (0 = unlimited)
`--emit-crit`	false (`true` in ECS)	Stage CRIT candidate envelopes to `crit-candidates/pending/`

CRIT staging

With --emit-crit (which terraform/go-schedules.tf:3078 passes on every scheduled run) the processor loads the CRIT spec + extended dictionaries and the canonical-pubdate lookup at startup (loadCritCtx, crit_ctx.go), then calls stageCRITForWizSec per advisory after the YAML has been archived so provenance.s3_key points at the stored copy (main.go:415-422). Cloud advisories are the natural CRIT population — an open-cvdb entry is already (provider, service, resource_type)-shaped — so this is the primary CRIT producer for the cloud namespace.

ECS Schedule

Runs weekly on Thursdays at 03:00 UTC (cron(0 3 ? * THU *)), 256 CPU units, 512 MB memory.

Key Files

File	Purpose
`cmd/wizsec-git-processor/main.go`	Main processor
`cmd/wizsec-git-processor/s3.go`	S3 uploader
`internal/wizsec/types.go`	YAML advisory Go structs
`internal/wizsec/parser.go`	YAML parsing, date parsing, file hashing
`internal/wizsec/mapper.go`	Advisory → CVESourceData mapping
`schemas/wizsec_advisory.schema.json`	JSON Schema Draft 7 for rawDataJSON

S3 path deviations

Two uploaders coexist:

Legacy, in-transaction (s3.go) — wizsec/advisories/{fileHash}/{slug}.yaml, written inside the per-file transaction so the resulting Artifact + Link rows and CVEMetadata.fileLinkId can reference the key (main.go:378-413). This is what backs the Artifact / Link rows in the storage table above.
Contract-conformant, post-transaction (s3client.Uploader) — the paths in the generated section below, except that the {sha256} slot carries wizsec.ComputeFileHash, which is SHA-1 (main.go:296), and the filename is the repo-relative vulnerabilities/<slug>.yaml.

S3 Persistence

Archive path: opencloud/files/{sha256}/{filename} ✓
Quarantine path: failed-feeds/wizsec-git-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
Failure reasons emitted: parse-error, store-error, tx-rollback

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[wizsec-git-processor] PROC -->|success| ARCHIVE[("S3: opencloud/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/wizsec-git-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.