Design: Siemens CSAF Processor

Overview

Fetches Siemens ProductCERT security advisories published in CSAF 2.0 (Common Security Advisory Framework) JSON format from the public TLP:WHITE feed index and stores the parsed vulnerability data into the VDB PostgreSQL database.

Source identifier: siemens Data type: csaf ECS task name: go-siemens-csaf-processor Schedule: Runs weekly on Fridays at 03:00 UTC (cron(0 3 ? * FRI *)).

Data Source

Feed Index Structure

{
  "feed": {
    "id": "siemens-security-advisories-csaf-feed-tlp-white",
    "entry": [
      {
        "id": "SSA-027652",
        "title": "...",
        "published": "2025-09-09T00:00:00Z",
        "updated": "2026-03-10T00:00:00Z",
        "link": [{ "rel": "self", "href": "https://cert-portal.siemens.com/productcert/csaf/ssa-027652.json" }],
        "summary": { "content": "..." }
      }
    ]
  }
}

CSAF Advisory Structure

Each advisory JSON follows the CSAF 2.0 standard:

• document.tracking.id — Advisory ID (e.g. SSA-027652)
• document.tracking.initial_release_date / current_release_date — Publication dates
• document.title — Advisory title
• document.references[] — Document-level references
• vulnerabilities[] — One or more CVE entries, each with:
  • cve — CVE identifier
  • scores[] — CVSS v2/v3/v4 scoring
  • notes[] — Descriptions and summaries
  • cwes[] — CWE identifiers
  • product_status.known_affected[] — Affected product IDs
  • references[] — Per-vulnerability references
  • remediations[] — Fix/workaround information with URLs
• product_tree — Hierarchical product catalog (branches/full_product_names)

Data Mapping

Primary-key Modelling: SSA-keyed rows

Siemens publishes the same CVE in multiple SSA bulletins (different products, different disclosure cycles). Keying CVEMetadata on the CVE collapses every SSA covering that CVE into a single last-writer-wins row, losing per-advisory product/affected lists. Following the Red Hat pattern (see redhat-csaf-processor.design.md §4):

• One CVEMetadata row per advisory, keyed on document.tracking.id (e.g. SSA-027652) with source='siemens'.
• Every vulnerabilities[] entry in the advisory is aggregated into the one row — descriptions, metrics, references, CWEs, and affected products are unioned (deduped).
• Each vulnerabilities[].cve becomes a CVEAlias row via data.Aliases. db.InsertAliases canonicalises edge direction and emits same-cveId cross-source edges so a consumer querying for CVE-X can resolve to every Siemens SSA covering it.
• Multi-SSA-per-CVE is now expressible: two SSAs each have their own row and both alias the same CVE-*.

CVEMetadata (one row per advisory)

CVEAlias

For every advisory:

• Every vulnerabilities[].cve is added to CVESourceData.Aliases.
• Any vulnerabilities[].ids[].text that begins with CVE- is also added (defensive — some Siemens advisories duplicate the CVE there).
• db.InsertAliases writes one edge per matching (aliasCveId, aliasSource) row found in CVEMetadata, and additionally emits same-cveId cross-source edges (e.g. (SSA-..., siemens) ↔ (CVE-..., nist-nvd) is created indirectly via the alias chain).

CVEDescription

• Source: vulnerabilities[].notes where category is description, summary, or general
• Falls back to document.notes if vulnerability has no matching notes
• containerType = "cna", lang = "en"

CVEMetadataReferences

All references are deduplicated by URL across document + vulnerability + remediation levels:

ReferenceSource = "siemens" on all rows.

CVEMetric

One row per unique CVSS vector string:

CVEProblemType

CVEAffected

Products from product_status.known_affected across every vulnerability in the advisory are unioned into the single SSA row, deduped by affectedHash:

• vendor = "Siemens"
• product = resolved from product_tree by product ID
• affectedHash = MD5(vendor|##|product|##||##|)
• containerType = "cna"

Artifact + Link

• Artifact: type = "OTHER", bomFormat = "siemens", R2 bucket/key set
• S3 key: siemens/advisories/{sha256}/{ssaId}.json
• Link: contentType = "PLAIN_JSON", URL = advisory self-link
• CVEMetadata.fileLinkId points to the Link record

DEFERRED advisories — the dominant row class

A feed entry whose advisory URL returns a permanent HTTP error (403 / 404 / 410) is not retried. storeDeferred (main.go:412) instead writes a placeholder row keyed on the SSA id with state='DEFERRED', dataVersion='5.0' and sourceAdvisoryRef set — no title, no rawDataJSON, no metrics, no affected products. Because that path leaves DatePublished nil, storeAdvisory (internal/processor/pipeline.go:430-434) passes 0 and the row lands with datePublished = 0.

Siemens’ TLP:WHITE feed lists a large tail of advisories that the portal no longer serves anonymously, so this is not a rare path: as of 2026-08 production holds 494 DEFERRED rows against 327 PUBLISHED ones — 60 % of the siemens footprint carries nothing but an id and a URL, and 444 of those have datePublished = 0.

Worse, 50 of those DEFERRED rows carry a non-zero datePublished, which means they were ingested successfully at some point and later downgraded: UpsertCVEMetadata sets title, vectorString and rawDataJSON from EXCLUDED on conflict, so a re-run that gets a 403 for a previously-good advisory replaces its content with NULL while keeping the earlier datePublished. Their fetchCount values (5–8) show the overwrite repeating on every run. This is a defect, not intended behaviour — see the efficacy record for siemens-csaf-processor.

Incremental Strategy

1. Feed-level: SHA256 of the full feed index JSON stored in BulkDataDumpTracker with key siemens_csaf_listing. Run is skipped entirely if unchanged (unless --force).
2. Advisory-level: SHA256 of each advisory JSON stored in CVEMetadata.sourceFileHash, keyed by (SSA, siemens). Advisory is skipped if hash matches existing record (unless --force).

This means ECS runs are O(1) network + O(changed) DB writes on most days.

Flags

Processing Pipeline

main()
 ├─ Fetch feed index (5-attempt retry, 30 s per attempt)
 ├─ Parse feed → []FeedEntry
 ├─ Check feed SHA256 vs BulkDataDumpTracker → skip if unchanged (NoWork)
 ├─ db.LoadProcessedHashes(source="siemens") → resume map (SSA id → sha256)
 ├─ Dispatcher goroutine → workCh
 ├─ N=--workers fetch workers, each:
 │   ├─ stop dispatching once (softDeadline − 10 min) has passed
 │   ├─ wait for the shared 200 ms ticker
 │   ├─ Fetch advisory JSON (5 attempts; 403/404/410 → DEFERRED, no retry)
 │   ├─ siemens.ParseAdvisory()  (failure → S3 quarantine `parse-error`)
 │   ├─ siemens.MapAdvisory() → *osv.CVESourceData — ONE record per advisory
 │   │     (nil → S3 quarantine `schema-violation`)
 │   └─ Skip when resumeSet[ssaID] == advisory SHA256
 ├─ Single consumer goroutine, per advisory:
 │   └─ storeItem: one transaction per advisory, up to 3 attempts
 │       ├─ S3 upload → InsertArtifact → InsertLinkWithArtifact
 │       ├─ processor.StoreCVESourceData() [shared pipeline]
 │       └─ UpdateCVEMetadataFileLinkID() [best-effort]
 └─ db.UpsertTracker(siemens_csaf_listing, newSHA256, processedCount)

siemens.MapAdvisory returns a single *osv.CVESourceData for the whole advisory (internal/siemens/mapper.go:24), consistent with the SSA-keyed modelling above. There is no 50-item batch and no per-item savepoint — every advisory gets its own short transaction with a 2-minute context.

Soft deadline

softDeadline = now + (EXPECTED_DURATION_MINUTES − 10), defaulting to now + 90 min when the env var is unset; workers then stop dispatching at softDeadline − 10 min. Under the 60-minute ECS budget the effective work window is 40 minutes. The unconditional 90-minute fallback also caps just go-siemens-csaf-backfill at 80 minutes.

Files

Local Development

# Incremental run (default — processes only new/changed advisories)
just go-siemens-csaf-backfill

# Full backfill of all advisories
just go-siemens-csaf-backfill FORCE="true"

# Limit to first 10 advisories (quick test)
just go-siemens-csaf-backfill LIMIT="10"

# Run in prod environment
just go-siemens-csaf-backfill TARGET="prod" LIMIT="5"

# Run in local ARM64 container (mirrors ECS)
just siemens-csaf-processor

Deployment

The processor is built as part of Containerfile.go-processors (target: siemens-csaf-processor) and pushed to ECR as go-siemens-csaf-processor-latest. The ECS task definition and EventBridge schedule are managed in terraform/go-schedules.tf under module "siemens_csaf_processor".

Known Limitations

• Multi-CVE advisories aggregate every vulnerabilities[] entry into a single SSA-keyed row. Per-CVE provenance within an advisory is recoverable only from the raw CSAF JSON in S3 / CVEMetadata.rawDataJSON.
• Consumers querying by CVE-* must follow CVEAlias to discover Siemens advisories — the canonical primary key is the SSA ID, not the CVE.
• Some older advisories may not follow strict CSAF 2.0 (notes/references may differ). The parser is tolerant of missing fields.
• Product names resolved from product_tree may be verbose (full platform strings). The affectedHash deduplicates by vendor+product pair.

S3 Persistence

• Archive path: siemens/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/siemens-csaf-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: parse-error, schema-violation

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.