IBM Security Bulletins JSON Processor — Design Document

Overview

Ingests security advisories from the IBM Support public JSON API and writes one CVEMetadata row per CVE ID found in each bulletin under source = "ibm".

API Endpoint: https://www.ibm.com/support/pages/securityapp/api/site/datalist

The API returns (bulletin × CVE) rows — one row per CVE per bulletin. The processor groups rows by bulletin nid, then writes one CVEMetadata row per CVE. CVE identifiers are linked as aliases via db.InsertAliases. Raw payloads are archived to S3.

No API key is required.

Source

The API returns:

• top_records: array of (bulletin × CVE) rows
• cve_ids: flat array of all CVE IDs
• product_names: product code mappings
• isCountNearLimit: boolean indicating if more records exist

Each row in top_records contains:

• nid — bulletin node ID
• title — bulletin title
• field_cve_id — CVE identifier
• field_product — affected product name
• field_cvss_base_score — textual severity (Low/Medium/High/Critical)
• field_pub_date — YYYY-MM-DD publication date
• field_x_force_url — CVE.org link

Vulnerability Identifier Prefixes

Architecture

Data Flow

Source-to-DB Mapping

CVEMetadata

CVEMetric

CVEAlias

Aliases include all other CVEs from the same bulletin. Cross-source backfill is performed automatically by db.InsertAliases.

Business Rules

B1 — One row per CVE

When a bulletin contains multiple CVEs, each CVE gets its own CVEMetadata row. All other CVEs from the same bulletin are stored as aliases.

B2 — Grouping by NID

The API returns one row per (bulletin × CVE) pair. The processor groups by nid before mapping to collapse multi-CVE bulletins.

B3 — Idempotency

A stable MD5 hash of nid|title|pubDate|cveCount|cveId1|severity1|... is stored in SourceFileHash. On subsequent runs, unchanged bulletins are skipped.

B4 — CVSS from textual severity

The API provides textual severity only (Low/Medium/High/Critical). These are mapped to representative CVSS v3.1 scores:

• Critical → 9.0
• High → 7.5
• Medium → 5.5
• Low → 3.5

B5 — No soft deadline for backfill

EXPECTED_DURATION_MINUTES is only read when set (ECS scheduled runs). Local backfill runs via justfile have no time limit.

B6 — Slack notifications

• notifier.RecordError() at every failure point
• notifier.HasErrors() checked at end-of-run to decide between Completed() and Errored()

Flags

ECS Schedule

• Cron: cron(0 7 * * ? *) — daily at 07:00 UTC
• CPU: 256 units
• Memory: 512 MB
• Expected duration: 60 minutes
• Container image: go-processors:go-ibm-security-bulletins-json-processor-{tag}

Verification Queries

-- IBM CVEMetadata rows
SELECT COUNT(*) FROM "CVEMetadata" WHERE source = 'ibm';

-- By severity
SELECT m."baseSeverity", COUNT(*)
FROM "CVEMetric" m
JOIN "CVEMetadata" cm ON cm."cveId" = m."cveId" AND cm.source = m.source
WHERE cm.source = 'ibm'
GROUP BY m."baseSeverity";

-- Recent IBM bulletins
SELECT m."cveId", m.title, m."datePublished"
FROM "CVEMetadata" m
WHERE m.source = 'ibm'
ORDER BY m."datePublished" DESC
LIMIT 20;

-- Affected products
SELECT DISTINCT a.product, COUNT(*) as cnt
FROM "CVEAffected" a
WHERE a.source = 'ibm'
GROUP BY a.product
ORDER BY cnt DESC
LIMIT 20;

Files

New files

• scripts/go-processors/cmd/ibm-security-bulletins-json-processor/main.go
• scripts/go-processors/internal/ibm/types.go
• scripts/go-processors/internal/ibm/client.go
• scripts/go-processors/internal/ibm/mapper.go
• scripts/go-processors/internal/ibm/service_map.go (Phase 1.10)
• scripts/go-processors/internal/ibm/crit_mapper.go (Phase 1.10)
• scripts/go-processors/internal/ibm/service_map_test.go (Phase 1.10)
• scripts/go-processors/internal/ibm/crit_mapper_test.go (Phase 1.10)
• scripts/go-processors/internal/critutil/dictionaries/extended/ibm.json (Phase 1.10)
• scripts/go-processors/ibm-security-bulletins-json-processor.design.md

Modified files

• Containerfile.go-processors — IBM final-target stanza.
• scripts/task-manager.toml — [tasks.ibm-security-bulletins-json-processor] block (cron 0 7 * * *, daily 07:00 UTC).
• scripts/task-dashboard/cmd/ecr-build/targets.go — append to simpleTargets.
• terraform/go-schedules.tf — module "ibm_security_bulletins_json_processor" + image local. Schedule: cron(0 7 * * ? *), command includes --emit-crit=true.
• justfile — removed the misleading go-ibm-security-bulletins-json-backfill recipe (the API is fixed-rolling-window, no backfill possible).
• .claude/hooks/post-push-ecr.sh — append IBM to TARGETS array.
• scripts/go-processors/internal/critutil/serviceavail/data.json — IBM block with 37 services and GA dates.

Upstream spec-lib changes (Phase 1.10 prerequisite)

github.com/Vulnetix/ietf-crit-spec commit 1a8210e adds:

• critvector.go — "ibm": "IB" in providerToCode (unblocks ComputeVector for provider="ibm").
• schemas/crit-record-v0.2.0.schema.json — "ibm" in both provider enums and "ibm_url" in template_format enum.
• schemas/crit-dictionary-v0.2.0.schema.json — "ibm" provider enum, "ibm_url" template_format enum, if provider=ibm then template_format=ibm_url constraint.
• schemas/crit-samples-v0.1.0.schema.json — "ibm" provider enum + "ibm_url" template_format enum.

vdb-manager/scripts/go-processors/go.mod bumped to github.com/Vulnetix/ietf-crit-spec v0.2.1-0.20260506213551-1a8210e645b7 to pick up the change.

Phase 1.10 — bringing the producer up to CRIT parity

The original scaffolding wrote CVEMetadata rows via processor.StoreCVESourceData but emitted no CRIT envelopes. Phase 1.10 closes the gap, bringing IBM into parity with alas/gcp/cloudflare/msrc/oracle/aws/salesforce/servicenow/sap.

Service mapping (internal/ibm/service_map.go)

Two-stage resolution:

1. field_oc_code lookup — IBM’s opaque product code (e.g. SSGMG2 → Db2). The most precise resolution path; populated empirically as codes surface in production. Bootstrap with ~30 codes covering the highest-volume products.
2. field_product substring fallback — case-insensitive match against ~70 synonyms covering Cloud Pak / Watson / watsonx / Db2 / WebSphere / MQ / CICS / AIX / Power / Z/OS / IBM i / QRadar / Guardium / Maximo / Cognos / Tivoli / Netcool / OpenShift / etc.

When neither path resolves, the producer logs "crit skipped: unmapped product" with oc_code + product so operators can grow the table iteratively.

Extended dictionary (internal/critutil/dictionaries/extended/ibm.json)

37 entries spanning IBM’s product surface. All use template_format=ibm_url (added upstream in spec-lib 1a8210e). Slot names kebab-case for ABNF compliance.

CRIT mapper (internal/ibm/crit_mapper.go)

MapBulletinToCRIT produces one envelope per (CVE × ServiceMatch):

• fix_propagation heuristic (no body fetched; defaults driven by service shape):
  • SaaS / cloud-managed (cloud_pak, ibm_cloud, watson, watsonx, instana, apptio, red_hat_openshift, verify, maas360, api_connect, app_connect) → automatic.
  • Storage / hardware (storage, bladecenter) → rebuild_and_redeploy.
  • On-prem databases / middleware / OS / asset-mgmt → version_update.
• resource_lifecycle mirrors the cloud/customer split.
• vex_status defaults to fixed (every IBM advisory IS the fix per the source’s publishing model).
• confidence ceiling at medium — IBM advisory body content (full description / remediation paragraph) lives in the ibm.com/support/pages/node/{nid} HTML which we don’t fetch; the API row gives only metadata. Operators see “API row only” in the evidence string.

Schedule & deployment

Verified API cadence: distinct bulletins/day from a live sample = 2 / 88 / 53 / 14 (~40/day). Hourly was considered but rejected — 23/24 runs would be near-zero-write no-ops. Daily 07:00 UTC plus the 6-day API window provides 24-hour outage tolerance.

No backfill mode

API parameters offset, page, start/end, from are all ignored — the endpoint always returns the most recent ~1000 rows × ~6 days. Historical IBM CVEs that fell off the window before this processor existed must come from a different feed (out of scope). The misleading go-ibm-security-bulletins-json-backfill justfile recipe has been removed; manual one-shots use go run directly or aws ecs run-task.

S3 Persistence

Not used. This processor does not currently archive payloads or quarantine failures to S3. Per the S3 Persistence Contract this is non-compliant — see the compliance matrix for the implementation roadmap.

⚠ Not in the compliance matrix — status needs verification.

Expected paths when implemented:

• Archive: ibm-security-bulletins/files/{sha256}/{filename}
• Quarantine: failed-feeds/ibm-security-bulletins-json-processor/{YYYY-MM-DD}/{reason}/{filename}
• Likely reasons: (none documented)