CERT-FR Processor — Design Document

Overview

Fetches CERT-FR (French national CERT, ssi.gouv.fr) security advisories from the public alerte and avis JSON feeds. Parses full advisory content — including French descriptions, affected systems, and vendor references — and stores complete CVE records in CVEMetadata with source="cert-fr". No API key required.

Processes two advisory types:

• alerte — active alerts (high-severity, time-sensitive)
• avis — informational notices

Data Sources

Processing Logic

1. Fetch alerte + avis index feeds (shallow JSON arrays: reference + json_url + last_revision_date)
2. Load per-advisory resume set: cveId → sourceFileHash from CVEMetadata where source='cert-fr'
3. For each advisory (up to --workers in parallel, rate-limited to 5 req/s):
   • Fetch full advisory JSON from the per-item json_url
   • Compute SHA256 of raw JSON bytes
   • Skip if primary CVE’s stored hash matches (unchanged) — unless --force
   • Parse advisory into certfr.Advisory struct
   • Skip if cves[] is empty
   • Map to []*osv.CVESourceData via certfr.MapAdvisory
   • Store via processor.StoreCVESourceData in its own transaction (with retries)
4. Exit non-zero if every advisory errored

Package Layout

internal/certfr/types.go

Data structures: IndexItem, Advisory, AdvisoryCVE, AdvisoryRisk, Revision, VendorAdvisory, AffectedSystem, Product, Vendor.

internal/certfr/mapper.go

• MapAdvisory(adv *Advisory, rawData []byte, advURL string) []*osv.CVESourceData
• Dates from revisions[] (index 0 = published, last = updated) — format "2006-01-02T15:04:05.000000" UTC
• State: "CLOSED" if closed_at is non-empty, else "PUBLISHED"
• Descriptions (lang="fr"): summary preferred; falls back to content; then appends non-duplicate entries from risks[].description
• References: advisory URL + all vendor_advisories[].url (type=“advisory”)
• Affected: one entry per unique vendor + product from affected_systems[]
• Multi-CVE bundle handling: a CERT-FR bulletin listing more than one CVE is bundling co-listed CVEs, not asserting them as aliases of each other (commit 56bdb9b):
  • The first CVE is the primary (full record: title, refs, descriptions, affected).
  • Each remaining CVE is upserted as its own minimal CVEMetadata row sharing the same sourceAdvisoryRef, title, state, dates, and references — but no Descriptions, no Affected, no inter-CVE CVEAlias edges.
  • Cross-source linkage (cert-fr ⇄ cve.org / nist-nvd / certbund / …) for every CVE in the bundle is still emitted automatically by db.InsertAliases via the same-cveId cross-source backfill.
• Bundle-level scoring suppression (SuppressDerivedMetrics): CERT-FR summary / content / risks[].description is bulletin-level prose. The shared processor.StoreCVESourceData pipeline normally derives a Vulnetix CVSS v4 vector from Descriptions (cvss.DeriveV4FromDescription). For multi-CVE bulletins this produces a single vector that does not correspond to any specific CVE in the bundle yet would be attributed to the primary’s (cveId, source='cert-fr') row. To prevent this, the cert-fr mapper sets osv.CVESourceData.SuppressDerivedMetrics = true on the primary record whenever len(adv.CVEs) > 1. Descriptions are still persisted to CVEDescription (lang='fr'); only the synthetic v4 metric is skipped. Single-CVE bulletins still receive the derived metric.

cmd/certfr-json-processor/main.go

Main binary with flags:

scripts/task-manager.toml still declares an all flag for this task, which the binary does not define — a dashboard run that passes --all exits 2 with flag provided but not defined. The just go-certfr-json-backfill recipe also defaults LIMIT=200, so a plain backfill invocation only touches the first 200 index entries.

cmd/certfr-json-processor/s3.go

buildS3Uploader — S3 PutObject closure using default AWS credential chain.

Storage Mapping

Through the shared processor.StoreCVESourceData path, each affected entry also feeds db.EnrichAffectedWithDependency, producing Dependency, DependencyRegistry, PackageVersion and PackageVersionCVE rows.

Incremental / Resume Strategy

• Per-advisory resume: LoadProcessedHashes loads (cveId → sourceFileHash) for source='cert-fr' at startup. If the primary CVE’s stored hash matches the advisory JSON SHA256, the advisory is skipped.
• Re-processing: --force bypasses the resume set (e.g. after description improvements).
• Idempotent: CVEMetadata uses ON CONFLICT DO UPDATE; InsertDescriptions deletes then re-inserts, so lang is always accurate on re-process. InsertMetrics likewise deletes existing rows scoped to the (cveId, source, containerType, adpOrgId, metricType) tuples being written before inserting, so re-runs replace rather than accumulate duplicate metric rows.
• Not idempotent: db.InsertReferences issues a bare ON CONFLICT DO NOTHING against a table whose only unique index is the uuid primary key, so every re-store of an advisory appends duplicate CVEMetadataReferences rows. The same applies to CVEProblemType.

Schedule

Runs daily at 01:00 UTC (cron(0 1 * * ? *) — terraform/go-schedules.tf:851). ECS Fargate, 256 CPU / 512 MB RAM, expected_duration_minutes = 60.

The soft deadline is EXPECTED_DURATION_MINUTES - 10, but the fallback is a hardcoded 55 minutes applied even when the variable is unset (main.go:69-73), so a local backfill is also truncated at 55 minutes. Workers that observe the deadline stop fetching and report the remaining advisories as skipped rather than errored.

Verification Queries

-- CERT-FR CVE records
SELECT "cveId", "title", "state", "datePublished", "sourceAdvisoryRef"
FROM "CVEMetadata"
WHERE source = 'cert-fr'
ORDER BY "datePublished" DESC
LIMIT 20;

-- French descriptions
SELECT d."cveId", d.lang, LEFT(d.value, 120) AS description
FROM "CVEDescription" d
WHERE d.source = 'cert-fr'
LIMIT 20;

-- Vendor advisory references
SELECT "cveId", url, title
FROM "CVEMetadataReferences"
WHERE "referenceSource" = 'cert-fr'
LIMIT 20;

-- Affected products
SELECT "cveId", vendor, product
FROM "CVEAffected"
WHERE source = 'cert-fr'
LIMIT 20;

-- Lang distribution check
SELECT lang, COUNT(*) FROM "CVEDescription" WHERE source = 'cert-fr' GROUP BY lang;

S3 path deviation

The generated section below states the contract path. The archive is actually written by a raw PutObject closure (s3.go:41-47) from inside the store transaction, at certfr/advisories/{sha256}/{cveId}.json, and is paired with an Artifact + Link row and CVEMetadata.fileLinkId (main.go:312-346) — not by s3client.Uploader.Archive, and not under the cert-fr/files/ prefix. The quarantine path is contract-conformant: failed-feeds/certfr-json-processor/{YYYY-MM-DD}/parse-error/{reference}.json (s3.go:28-29). Fetch failures and store-after-retry failures are counted but their payloads are not quarantined.

S3 Persistence

• Archive path: cert-fr/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/certfr-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: parse-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.