confluent-json-processor

Status: Live Source: support.confluent.io Zendesk Help Center JSON API Type: json (paginated Zendesk section listing) Source slug: confluent Schedule: Runs weekly on Thursdays at 05:00 UTC (cron(0 5 ? * THU *)).

Overview

Apache Kafka does not publish GHSA advisories, so CONFSA bulletins are the only first-party feed that maps upstream Apache CVEs onto Confluent-specific fixed versions (CP 7.x.y) and states whether Confluent Cloud is affected at all. NVD carries the upstream CVE; only this feed carries the Confluent applicability and remediation detail that a Confluent customer can act on, and it is the sole input to the confluent/* CRIT envelopes. Without it, a Confluent user gets the generic Kafka CVE with no answer to “is my managed cluster affected, and which CP release fixes it”.

Each run pages the Zendesk section listing following next_page, keeps only articles whose title is CONFSA-YYYY-NN (the section also contains plain Security Release Notes, which are filtered out), and stores each changed article.

Records produced

--limit is checked against stored records inside the loop but against fetched articles when deciding whether to stop paging, so a limited run may fetch one page more than it needs.

Identifier policy

• Primary cveId: a.CVEIDs[0] — the first CVE reference extracted from the title and body. Every CONFSA observed so far carries at least one.
• Fallback: a minted CONFSA-{year}-{NNNN} sequence, where year comes from the article’s created_at (falling back to updated_at, then the current year) and the sequence continues from db.LoadMaxGcveSequence. On a failed store the sequence counter is decremented so a retry does not burn ids.
• The native CONFSA-YYYY-NN is always written as an alias.

When the primary id is the CVE, the native CONFSA-YYYY-NN id exists in CVEMetadata only as an alias-FK placeholder row under source='confluent' — content-free, datePublished = 0, written by db.InsertAliases → db.EnsureMinimalCVEMetadata for the non-CVE alias. 39 of the 258 production rows are these placeholders; they are not processor output and must be excluded from any field-coverage measurement of this source.

In practice minting is not rare: only 25 of the 258 rows are CVE-keyed. Most CONFSAs name their CVE in linked NVD URLs rather than in a form the title/body regex extracts, so the fallback path — and with it the duplicate-mint defect below — is the common case.

Resume

A time-based tracker gate (frequencySecs = 86400) short-circuits the whole run unless --force. On the weekly schedule the 24-hour window never blocks a scheduled fire.

🛑 Broken — the per-article resume check never matches, and minting duplicates on every run. db.LoadProcessedHashes is keyed by CVEMetadata.cveId (internal/db/resume.go:42), but the check is seen[a.ConfsaID] == a.ContentHash (main.go:236) against the native CONFSA-YYYY-NN id. The stored cveId is either the referenced CVE or a minted CONFSA-YYYY-NNNN — never the native id, and the native id’s alias-placeholder row carries no sourceFileHash, so LoadProcessedHashes excludes it. seen[a.ConfsaID] is therefore always "" and no article is ever skipped.

For CVE-keyed articles that only wastes a re-upsert. For minted articles it is destructive: each run takes a fresh sequence number from db.LoadMaxGcveSequence and writes a new record for the same advisory. Measured in production: 103 minted rows for 14 distinct articles — CONFSA-2025-08 alone has 20 rows (CONFSA-2025-0001 … CONFSA-2025-0020), and CONFSA-2026-03..06 have 17 each — plus 194 GcveIssuance rows for those same 14 advisories. The row count grows by one per article per run, and the stored == 0 && skipped > 0 → no_work branch (main.go:345) can never fire.

A further 91 rows carry the historical VVD-CONFSA-YYYY-NNNN prefix (pre-prefix-strip mints) alongside their stripped CONFSA-YYYY-NNNN twins, so the same advisory can appear under three id shapes.

Fix: key the resume map on what is actually stored. Either look up the article by its sourceAdvisoryRef (the Zendesk URL, which is stable and unique per article), or resolve the existing minted id for a ConfsaID before minting a new one — minting must be conditional on “this article has no record yet”, not on “the resume check missed”.

Anti-bot triage

HTTP 403 from Zendesk is a hard exit: the identifying-UA filter rejected us, no DB writes happen, and the run reports task.errored. (The recon-era concern that Zendesk would block us did not materialise — the feed has been served to the standard crawler UA since launch.)

Deployment

• Source:
  • Primary: https://support.confluent.io/api/v2/help_center/sections/360008413952/articles.json (Zendesk Help Center JSON API; standard schema)
  • Per-advisory HTML at https://support.confluent.io/hc/en-us/articles/{id}-CONFSA-YYYY-NN-...
• Owner: Confluent Security
• Licence: Public (subject to Zendesk UA filter)
• Schedule cadence: Runs weekly on Thursdays at 05:00 UTC (cron(0 5 ? * THU *)). CONFSAs publish at irregular cadence, so a weekly check is generous.
• ECS resources: cpu_units=256, memory_mb=512, expected_duration_minutes=15
• ECS command: ["/app/confluent-json-processor", "--emit-crit=true"] — CRIT staging is on in production even though the binary’s own --emit-crit default is false.

2. Source contract (Phase 0 recon — 2026-05-09)

Recon-protocol grid:

URLs checked (all 403 to default-UA WebFetch but indexable / known via search):

• https://www.confluent.io/security/ — 404 (no public-marketing security page)
• https://support.confluent.io/hc/en-us/sections/360008413952-Security-Advisories-and-Security-Release-Notes — 403
• https://support.confluent.io/hc/en-us/articles/39740309244180-CONFSA-2025-04-CVE-2025-27817-... — 403; sample advisory exists per indexed snippet

Source contract:

Sample advisory: https://support.confluent.io/hc/en-us/articles/39740309244180-CONFSA-2025-04-CVE-2025-27817-Confluent-Platform-and-Confluent-Cloud-Arbitrary-File-Read-and-Server-Side-Request-Forgery-SSRF-Vulnerability-via-unauthorized-changes-to-Kafka-Client-SASL-OAUTHBEARER-configuration

3. Architecture diagram

[Zendesk JSON API] ── HTTP GET (identifying UA) ──> 403?
                                                       │
                                                       ├─ yes → log + exit non-zero
                                                       │       (DEFERRED until UA negotiation)
                                                       │
                                                       └─ no → confluent.ParseSection
                                                                   │
                                                                   ▼
                                       [Article{id, title, body, updated_at}]
                                                                   │
                                                                   ▼
                                       confluent.ExtractCONFSA: regex CONFSA-YYYY-NN + CVE-YYYY-NNNN
                                                                   │
                                                                   ▼
                                       confluent.MapToSourceData → osv.CVESourceData
                                                                   │
                                                                   ▼
                                       processOneCVE (main.go:422) — bespoke, NOT
                                       processor.StoreCVESourceData:
                                         db.UpsertCVEMetadata
                                         db.InsertAliases
                                         db.InsertDescriptions
                                         db.InsertReferences
                                                                   │
                                                                   ▼
                                       optional CRIT staging (Confluent Cloud / Platform)
                                                                   │
                                                                   ▼
                                       S3 archive (canonical JSON re-serialise)

4. Source → DB field mapping

5. Identifier policy

• Primary cveId: First CVE alias by canonical sort. Every CONFSA carries at least one CVE so minting is rarely needed; fall back to CONFSA-{YYYY}-{NN} if a future CONFSA omits CVE refs.
• Aliases via db.InsertAliases: CONFSA-YYYY-NN + every CVE-YYYY-NNNN.
• source="confluent".

6. CRIT / VEX

One envelope per detected product (inferConfluentServices may match several, so a CONFSA affecting both Cloud and Platform stages two). An article that matches no product keyword stages nothing and is instead offered to CRIT inference via critprep.Hook. Because the feed is Confluent-scoped by identity, the matcher is built with ImplicitProviders: ["confluent"] so short service tokens (ksqldb, connect) count without an in-text “Confluent”.

VEX semantics:

• vex_status="fixed" when CONFSA names a fixed version
• shared_responsibility="provider_only" for Confluent Cloud, customer_action_required for Confluent Platform (customer-installed)
• fix_propagation="automatic" for Cloud, version_update for Platform

Dictionary: the five services above ship in the CRIT spec dictionary (github.com/Vulnetix/ietf-crit-spec → dictionaries/confluent.json), loaded by critutil.LoadSpecDictionaries(). There is no internal/critutil/dictionaries/extended/ directory — the local extended layer is internal/critutil/dictionaries/local/, and it carries no Confluent entries. A (provider, service, resource_type) triple with no dictionary template resolves to nil and stages no envelope, so the dictionary is the hard gate on emission.

7. S3 layout

Per AGENTS.md:

• Archive: confluent/files/{sha256}/{CONFSA-id}.json
• Quarantine: failed-feeds/confluent-json-processor/{date}/{reason}/{filename}

8. Error handling

Anti-bot triage flow: if the Zendesk JSON API returns 403 on any page, log "confluent zendesk fetch 403 — UA filter rejected" and exit 1 with no DB writes.

There is no per-article HTML fetch, so the fetch-error reason never fires, and no schema-violation path exists.

9. Performance

CPU 256, memory 512 MB. Expected duration <5 minutes per scheduled run (Zendesk JSON returns 100 articles per page; full section <1 page). Full backfill: <10 minutes.

10. Backfill

just go-confluent-json-backfill TARGET="local" FORCE="false" LIMIT="0" \
    EMIT_CRIT="true" SOURCE_URL=""

Recipe mirrors go-ovhcloud-json-backfill since both consume Zendesk-shaped JSON. It unsets EXPECTED_DURATION_MINUTES, and the binary leaves its soft deadline at the zero value when that var is absent, so a backfill genuinely runs to completion. --source-url overrides the section listing URL for targeted re-ingest.

11. Code map

12. Why this feed earns its slot

GHSA / NVD overlap is partial, and the non-overlapping part is the whole point:

• Apache Kafka does not use GitHub GHSA, so CONFSAs are the only first-party feed for Kafka security CVEs from the Confluent angle.
• CONFSAs map upstream Apache CVEs to Confluent-Platform-specific fixed versions (CP 7.x.y) — the actionable remediation a Confluent customer needs, and the input to the confluent/platform CRIT envelopes.
• The Confluent Cloud applicability flag exists nowhere else: it is what tells a managed-cluster customer whether they need to act at all.

S3 Persistence

• Archive path: confluent/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/confluent-json-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: parse-error, store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.