NCSC-NL CSAF Processor

Data Source

• Publisher: Nationaal Cyber Security Centrum (NCSC-NL), Netherlands
• Format: CSAF 2.0 JSON advisories
• Index: https://advisories.ncsc.nl/csaf/v2/index.txt
• Advisory base: https://advisories.ncsc.nl/csaf/v2/
• Volume: Hundreds of advisories per year (multi-vendor coordinator), each containing 1–20+ CVEs
• Update frequency: Multiple advisories per week; existing advisories may be revised
• Language: Dutch (nl)

Index Format

Plain text file with one relative path per line:

2026/ncsc-2026-0098.json
2026/ncsc-2026-0097.json
2025/ncsc-2025-0412.json

Each path is appended to the base URL to construct the full advisory URL.

CSAF Advisory Structure

Standard CSAF 2.0 with these NCSC-NL-specific patterns:

• Publisher role: coordinator (NCSC-NL coordinates across vendors, not a vendor itself)
• Language: Advisory titles and notes are in Dutch (lang: "nl")
• Product status: Uses known_affected (standard CSAF)
• CWE: Singular cwe object per vulnerability
• Product tree: Hierarchical vendor/product/version branches; vendor extracted dynamically
• Scores: CVSS v3.1 is the primary scoring system
• Multi-vendor: A single advisory may cover products from multiple vendors (Apple, Microsoft, etc.)

Primary-key Modelling: advisory-keyed rows

NCSC-NL is a coordinator, not a vendor: one bulletin routinely bundles 1–20+ CVEs across several vendors. Keying CVEMetadata on the CVE would collapse every bulletin covering a CVE into one last-writer-wins row and lose the bulletin’s own product list. So, like redhat-csaf-processor and siemens-csaf-processor:

• One CVEMetadata row per advisory, keyed on NCSC-{YYYY}-{N} — derived from document.tracking.id by ncscnl.AdvisoryToVVDID (internal/ncscnl/parser.go:55), which strips leading zeros from the sequence (NCSC-2026-0098 → NCSC-2026-98).
• Every vulnerabilities[].cve in the bulletin becomes an alias on CVESourceData.Aliases, written by db.InsertAliases inside the shared pipeline.
• A GcveIssuance row GCVE-110-NCSC-{YYYY}-{N} is minted per advisory, plus one GcveAlias row per constituent CVE (main.go:338-374).

Consumers asking “which NCSC-NL bulletins cover CVE-X?” must traverse CVEAlias; there is no CVE-*-keyed ncsc-nl row.

Data Mapping

Relations

• CVEAlias: every vulnerabilities[].cve in the bulletin, written via db.InsertAliases
• GcveIssuance + GcveAlias: GCVE-110-NCSC-{YYYY}-{N} plus one alias row per CVE
• CVEDescription: Vulnerability notes (category: description/summary/general), fallback to document notes; lang: "nl"
• CVEMetadataReferences: Advisory URL + document refs + vulnerability refs + remediation URLs
• CVEMetric: CVSS v3.1/v3.0/v2.0/v4.0 scores from scores[]
• CVEProblemType: CWE from singular cwe field and cwes array
• CVEAffected: Products from product_status.known_affected, vendor extracted from product tree branches
• Artifact + Link: raw CSAF uploaded to ncsc-nl/advisories/{sha256}/{ncscId}.json, type='OTHER', bomFormat='ncsc-nl', contentType='PLAIN_JSON'

DEFERRED advisories

An index entry whose advisory URL returns 403 / 404 / 410 is stored as a placeholder row (storeDeferred, main.go:381): state='DEFERRED', sourceAdvisoryRef set, no title, no raw JSON, and datePublished = 0 because the path supplies no date. Production currently holds zero DEFERRED ncsc-nl rows.

Incremental Processing

1. Fetch index.txt, compute SHA256
2. Compare against BulkDataDumpTracker key ncsc_nl_csaf_index
3. If unchanged and --force not set, exit early (reports task.nowork)
4. Load per-advisory sourceFileHash resume set from DB (keyed on NCSC-YYYY-N)
5. Fetch every advisory in the index sequentially (200 ms courtesy sleep between advisories; no worker pool), then compute its SHA256 and skip the store when the hash is unchanged. The resume check happens after the fetch (main.go:148 fetch, main.go:182 resume check), so an index change forces a re-fetch of the entire index — currently ~6 000 advisories — not just the new entries.
6. Update tracker on completion

Soft deadline

softDeadline = now + (EXPECTED_DURATION_MINUTES − 10), defaulting to now + 60 min when the env var is unset; the loop then breaks at softDeadline − 10 min (main.go:140). With the ECS budget of 60 minutes the effective work window is 40 minutes, and a full sequential pass over the index can exceed it — the tracker is still advanced, so unreached advisories wait for the next index change. The unconditional 60-minute fallback also truncates just go-ncsc-nl-csaf-backfill at the 50-minute mark.

Flags

Files

Local Development

# Direct Go run (fastest for testing)
just go-ncsc-nl-csaf-backfill

# With production DB
just go-ncsc-nl-csaf-backfill TARGET=prod

# Force reprocess all
just go-ncsc-nl-csaf-backfill FORCE=true

# Limit to 10 advisories
just go-ncsc-nl-csaf-backfill LIMIT=10

# Container build (mirrors ECS)
just ncsc-nl-csaf-processor

Deployment

• ECS schedule: Every 6 hours (cron(0 */6 * * ? *))
• Resources: 256 CPU / 512 MB memory
• ECR tag: go-ncsc-nl-csaf-processor-latest
• Log group: /ecs/vdb-scheduler/go-ncsc-nl-csaf-processor

S3 Persistence

• Archive path: ncsc-nl/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/ncsc-nl-csaf-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: parse-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.