CERT-CC JSON Processor Design

Status: Live Source: CERT/CC Vulnerability Notes API Type: json (per-month index endpoint, one record per vulnerability note) Source slug: certcc Schedule: Daily at 06:30 UTC (cron(30 6 * * ? *)), gated by a 24 h freshness tracker.

Overview

Property	Value
Source	CERT Coordination Center (Carnegie Mellon)
API	`https://kb.cert.org/vuls/api/{YYYY}/{MM}/`
Schedule	Once daily (cron 30 6 * * ? *)
Soft deadline	`EXPECTED_DURATION_MINUTES − 10` when set (60 min in `task-manager.toml` ⇒ 50 min), otherwise a 4-hour fallback. `--backfill` runs carry no deadline (`cmd/certcc-json-processor/main.go:70-77`)
Resources	256 CPU / 512 MB
Source field	`certcc`
VVD format	`CERTCC-{YYYY}-{idnumber}`
GCVE format	`GCVE-110-CERTCC-{YYYY}-{idnumber}`

Business Logic

Freshness Check

Uses BulkDataDumpTracker with source=certcc, frequency=86400s (24h)
Skipped in --backfill mode

Processing Flow

Fetch — GET current year/month endpoint (or iterate all months for backfill)
Parse — JSON array of advisory objects
Deduplicate — Skip VUIDs already in CVEMetadata (unless --all)
Map — Each advisory → CERTCC-{year}-{idnumber} + CVESourceData
Store — In a single transaction per advisory:
- UpsertCVEMetadata (source=“certcc”)
- InsertDescriptions (overview or legacy fields)
- InsertReferences (self-link + public[] URLs)
- InsertMetrics (CVSS v2 base + CAM + VRDA + temporal/environmental as “other” types)
- UpsertGcveIssuance + InsertGcveAlias (VUID + CVE IDs)
- InsertAliases (CVEAlias for extracted CVE IDs)
- S3 artifact upload → Link → fileLinkId

VVD ID Generation

Deterministic: CERTCC-{year}-{idnumber} where year = year from datecreated
No sequence allocation needed (idnumber is the VUID numeric portion)
Example: VU#772695, datecreated=2004-01-01 → CERTCC-2004-772695

Backfill Mode

--backfill iterates from current month back to September 2000
500ms delay between month fetches (rate limiting)
Continues on error (logs and proceeds to next month)

Architecture

graph TB EB[EventBridge: daily 06:30 UTC] --> ECS[ECS Fargate Task] ECS --> main[cmd/certcc-json-processor/main.go] main --> certcc[internal/certcc/] certcc --> parser[parser.go: Parse JSON] certcc --> mapper[mapper.go: Map to CVESourceData] certcc --> types[types.go: Advisory struct] main --> db[internal/db/] db --> cvemeta[UpsertCVEMetadata] db --> refs[InsertReferences] db --> desc[InsertDescriptions] db --> metrics[InsertMetrics] db --> gcve[UpsertGcveIssuance + InsertGcveAlias] db --> artifact[InsertArtifact + InsertLinkWithArtifact] db --> RDS[(PostgreSQL RDS)] main --> S3[S3: certcc/{year}/{idnumber}.json]

Data Mapping

CERT-CC Field	Target Table/Column	Notes
`vuid`	CVEMetadata.sourceAdvisoryRef	e.g. “VU#772695”
`name`	CVEMetadata.title	Vulnerability name
`overview` / `clean_desc+impact+resolution+workarounds`	CVEDescription.value	Modern vs legacy format
`cveids[]`	GcveAlias + CVEAlias	Extracted CVE IDs as aliases
`public[]`	CVEMetadataReferences	Each URL as type=“url”
`cvss_basevector` + `cvss_basescore`	CVEMetric (cvssV2_0)	Vector string + parsed score
`cvss_temporal` + `cvss_environmental` + decomposed fields	CVEMetric (other/certcc-cvss-temporal-env)	JSON object
`cam_*` fields (10)	CVEMetric (other/certcc-cam)	JSON object with all CAM dimensions
`vrda_*` fields (3)	CVEMetric (other/certcc-vrda)	JSON object with VRDA assessments
Full advisory JSON	S3 artifact + Link	`certcc/{year}/{idnumber}.json`
`publicdate` → `datecreated` fallback	CVEMetadata.datePublished	Unix seconds
`dateupdated`	CVEMetadata.dateUpdated	Unix seconds

Records produced

Condition	Records
Every note not already seen	`CVEMetadata` (`cveId="CERTCC-{year}-{id}"`, `source="certcc"`, `sourceAdvisoryRef="VU#{id}"`), `CVEDescription` (`lang="en"`), `CVEMetadataReferences` (self-link + `public[]` URLs + `cert_advisory`), `GcveIssuance` + `GcveAlias` (VUID and each `cveids[]` entry)
`cvss_basevector` present	`CVEMetric` (`cvssV2_0`) with the parsed base score and a derived severity
CAM / VRDA / temporal-environmental fields present	`CVEMetric` rows with `metricType="other"` and `otherType` in {`certcc-cam`, `certcc-vrda`, `certcc-cvss-temporal-env`}
`cveids[]` non-empty	`CVEAlias` edges from the `CERTCC-…` row to each CVE that already exists in `CVEMetadata`
S3 upload succeeded	`Artifact` + `Link` at `certcc/{year}/{idnumber}.json`, and `CVEMetadata.fileLinkId`

Known deviations

InsertAliases is conditional. It is only called when the note carries at least one CVE ID (cmd/certcc-json-processor/main.go:303-307). scripts/go-processors/AGENTS.md requires direct writers (those bypassing processor.StoreCVESourceData) to call it on every store so the same-cveId cross-source backfill always runs. Harmless in practice for the private CERTCC-… id space, but it is a deviation from the documented contract.
Empty strings instead of NULL. This processor’s local strPtr always returns a non-nil pointer (cmd/certcc-json-processor/main.go:508), unlike the shared pipeline’s helper which maps "" → NULL. Notes without a CVSS vector therefore get vectorString = '', not NULL — 2,819 of 3,713 production rows. Any coverage query must use NULLIF("vectorString",'') IS NOT NULL.
Non-canonical archive key. The S3 object goes to certcc/{year}/{idnumber}.json, not {source}/files/{sha256}/{filename}, and there is no quarantine call on the failure path.

Schema

See schemas/certcc_advisory.schema.json for the full JSON Schema definition of the API response format.

S3 Persistence

Archive path: certcc/files/{sha256}/{filename} ✓
Quarantine path: failed-feeds/certcc-json-processor/{YYYY-MM-DD}/{reason}/{filename} ⚠ not yet wired
Failure reasons emitted: store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[certcc-json-processor] PROC -->|success| ARCHIVE[("S3: certcc/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/certcc-json-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.