cnvd-git-processor

Status: Live but storing nothing — see Known defect Source: CNVD-Dump — CIRCL’s git mirror of the China National Vulnerability Database Type: git (runtime clone/pull via pure-Go go-git — no baked data stage) Source slug: cnvd Schedule: Runs weekly on Tuesdays at 03:00 UTC (cron(0 3 ? * TUE *)). CPU 256 / memory 512 MB / expected_duration_minutes = 45.

Overview

CNVD is China’s national vulnerability catalogue. A large share of its entries (CNVD-YYYY-NNNNN) are Chinese-vendor products and CNVD-only findings that never reach MITRE, and the entries that do carry a CVE frequently name affected products the CVE record omits. This feed is the only route by which a CNVD identifier resolves in the VDB at all, and the only source of the CNVD↔CVE alias edges that let a CNVD-numbered report be reconciled with a CVE.

CNVD publishes no API we can consume directly; CIRCL’s git mirror of the weekly JSON dumps is the practical ingest path, which means the feed is only as fresh as that mirror.

Each run:

  1. Refreshes the repo. processor.PullOrClone clones (or pulls) /data/CNVD-Dump at depth 1 through go-git — the final image stage is scratch and ships no baked dump data.
  2. Short-circuits on an unchanged HEAD against BulkDataDumpTracker.sha256 for source cnvd_advisory; a match without --force reports task.nowork and exits 0.
  3. Walks dump_json/ for every .json file, sorted by name, then applies the 3-day changed-file filter when a tracker exists (--force / first run / un-diffable history → full scan).
  4. Parses each dump file with cnvd.ParseFile, which accepts a bare JSON array, a wrapped object (data, vulnerabilities, items, records), or a single record — one dump file holds many advisories.
  5. Stores records in transactions of 200 per dump file.

Known defect: the parser never unwraps a dump file

Every file in the mirror is shaped {"vulnerabilitys": {"vulnerability": [ … ]}} — note the upstream spelling of the wrapper key. cnvd.ParseFile (internal/cnvd/cnvd.go:162) only tries data, vulnerabilities, items and records, so it falls through to its “single record” branch and hands the entire file to cnvd.MapRecord, which finds no number field and returns nil.

The result is a run that exits 0 and looks healthy while storing nothing. A full production backfill on 2026-08-06 reported dumpFilesProcessed: 598, recordsTotal: 598, recordsUpserted: 0, unchanged: 598 with exactly 598 skipping unmappable record … empty CNVD number warnings — one per file. This binary has never written a CNVD row.

Everything below the fold describes the intended contract, which becomes real once the wrapper key is handled.

Where production’s CNVD rows actually came from

The 128,752 source='cnvd' rows in production were loaded by the one-off .repo/backfill_cnvd.py from a larger historical export, not by this processor. They carry no sourceAdvisoryRef, no affectedProduct and no vectorString.

Two of those three gaps are mapper defects rather than upstream gaps — the dump does carry products.product and the CNVD id — but the third is upstream: CNVD publishes a Chinese severity label (低/中/高/超危) and no CVSS vector, so vectorString will always be null on CNVD rows.

Note also the scale mismatch: the git mirror is a rolling window (598 dump files at HEAD 6797ad5), so even a fixed parser can only ever act as an incremental catcher. It cannot reconstruct the 128,752 historical rows, and it cannot repair their empty columns, because those records are not in the mirror.

cnvd.MapRecord always keys the row on the CNVD number (number); CVE identifiers found in the cves[] block are emitted as CVEAlias edges, never as the row key. Records with an empty number are counted unchanged and dropped with a warning.

Records produced

ConditionRecords
Every stored recordCVEMetadata (source="cnvd", cveId = CNVD-YYYY-NNNNN, sourceAdvisoryRef = the same CNVD ID, sourceFileHash = SHA1 of the record’s raw JSON)
Record text presentCVEDescription rows (lang="zh") for description, formalWay (remediation), patchDescription/patchName, and discovererName
referenceLink / cves[].cveUrl / bids[].bidUrlCVEMetadataReferences (type="url", referenceSource = cnvd, cnvd-cve-xref, cnvd-bid-xref)
products[] presentCVEAffected (vendor cnvd, product = the product string) with no version rows — CNVD product strings are free text, not versioned ranges
Descriptions presentone derived CVEMetric (containerType="vulnetix", cvssV4_0) from the Chinese prose
AlwaysCVEAlias edges via db.InsertAliases (CVE cross-references + same-cveId cross-source edges)
Every affected productDependency, DependencyRegistry, PackageVersion, PackageVersionCVE via db.EnrichAffectedWithDependency
End of runBulkDataDumpTracker row cnvd_advisory (written unconditionally — see Failure modes)

CNVD publishes a severity label (低/中/高/超危) but no CVSS vector, so no upstream CVEMetric vector is stored; CVEMetadata.vectorString stays null on these rows, and affectedVendor / affectedProduct are deliberately left empty by the mapper even when products[] is populated.

Resume

  • Repo-level — the cnvd_advisory tracker SHA.
  • Per-record — db.LoadProcessedHashes("cnvd") builds a {cnvdId → sourceFileHash} skip set; the hash is the SHA1 of the record’s raw JSON bytes, so a record is re-upserted only when its own bytes change.
  • Statefile (local only) — outside ECS, .repo/cnvd.state.json records the last completed dump file so an interrupted local backfill resumes from it. The statefile is deleted on a clean finish and ignored under --force.

Failure modes

SymptomCause
repository unchanged, skipping processingtracker SHA equals HEAD
no CNVD dump files foundthe changed-file filter removed every dump file — the run exits before writing the tracker
soft deadline reached, stopping earlyEXPECTED_DURATION_MINUTES − 10 elapsed. The tracker is still advanced to the new HEAD, so dump files the run never reached are not revisited until the next --force; a truncated run should be followed by just go-cnvd-git-backfill FORCE=true
CNVD batch transaction failedone record’s store error aborted the surrounding transaction; the whole 200-record batch is lost and counted as errored
%d records errored + exit 1any errored record fails the run after the tracker has already been advanced

Flags

FlagDefaultMeaning
--forcefalseIgnore the tracker SHA, the per-record hash skip set, and the statefile
--repo/data/CNVD-DumpClone location
--data-dir(repo + /dump_json)Override the dump directory
--state-dir.repoStatefile directory (local runs only)

Local run: just go-cnvd-git-backfill (append prod for production).

S3 Persistence

  • Archive path: cnvd/files/{sha256}/{filename} ✓
  • Quarantine path: failed-feeds/cnvd-git-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
  • Failure reasons emitted: parse-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[cnvd-git-processor] PROC -->|success| ARCHIVE[("S3: cnvd/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/cnvd-git-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.