databricks-fetch-processor

Status: Live Source: Databricks Knowledge Base (sitemap-driven) Type: fetch (HTML scraping) Source slug: databricks Schedule: Runs weekly on Saturdays at 06:00 UTC (cron(0 6 ? * SAT *)).

Overview

Ingests Databricks KB-hosted security bulletins from kb.databricks.com. Databricks publishes no listing endpoint, no RSS feed, no JSON API, and no org-level GitHub Security Advisories tab — kb.databricks.com/sitemap.xml is the sole enumerable surface. The processor filters that sitemap to URLs whose path matches security-bulletin-*-cve-YYYY-NNNN+ and scrapes each resulting KB article for CVE / CVSS / affected-version / fixed-version metadata.

Each CVE in a multi-CVE bulletin (none have appeared yet, but the mapper handles them) becomes its own CVEMetadata row under source="databricks", sharing sourceAdvisoryRef, title, affectedProduct, and page hash. The volume is intentionally low — historically 1–2 bulletins per year — but discovery is deterministic, so a newly-published bulletin lands in the next daily run.

Value-bearing Databricks-vendored OSS (mlflow, delta, dbt-databricks, etc.) continues to be ingested via per-repo GHSA through ghsa-git-processor; this processor exists for first-party advisories on proprietary components (JDBC driver, ODBC driver, future runtime issues) that GHSA does not cover.

Source

Property	Value
Discovery URL	`https://kb.databricks.com/sitemap.xml`
Auth	None — fully public
Format	XML sitemap → server-rendered HTML detail pages
Pagination	None — single ~1,000-URL sitemap, no sub-sitemap index
Discovery filter	Path regex `security-bulletin-[a-z0-9-]+-cve-\d{4}-\d{4,}`
Current bulletin count	1 (`CVE-2024-49194`, JDBC driver, as of 2026-05-15)
Cadence	Irregular — no Patch Tuesday equivalent
Identifier shape	Slug-based (`security-bulletin-databricks-jdbc-driver-vulnerability-advisory-cve-YYYY-NNNN`); no KB-side numeric advisory ID

Page Structure

Detail pages are Helpjuice-rendered (<h1 data-helpjuice-element="Header Article Title">…</h1>) with a stable Problem / Cause / Solution structure. Each section heading is an <h1 id="problem-N"> / <h1 id="cause-N"> / <h1 id="solution-N"> element — sub-sections inside Solution use <h2>. The article header carries a Helpjuice prelude paragraph:

Bulletin ID: DB-2024-01<br>Publication Date: 2024-DEC-11<br>Last Updated: 2024-DEC-11

CVE / Affected / Fixed / CVSS metadata lives in a four-column inline table; the score cell uses <br><br>  padding (the NBSP is normalised to ASCII space before parsing).

Element	Source
Bulletin title	`<h1 data-helpjuice-element="Header Article Title">`, fallback `<title>` (with " - Databricks" suffix trimmed)
Bulletin ID	`Bulletin ID: DB-YYYY-NN` line in the header prelude
Primary CVE	URL path suffix (`-cve-YYYY-NNNN+`)
Additional CVEs	Any `CVE-YYYY-NNNN+` matches in body
`Problem` section	`<h1 id="problem-N">` body
`Cause` section	`<h1 id="cause-N">` body
`Solution` section	`<h1 id="solution-N">` body, bounded by `Contact Information` / `Acknowledgments` / `Changelog` siblings
Affected Versions	Column whose `<th>`-like cell text contains “Affected” + “Version”
Fixed Versions	Column whose cell text contains “Fixed” + “Version”
CVSS	Header cell `CVSSvN.N` + data row cell whose stripped text parses as a 0–10 base score
Published date	`Publication Date: YYYY-MMM-DD` (label) or `Published: Month DD, YYYY` (fallback)

Parsing

internal/databricks/parser.go uses compiled regexp — no DOM tree. The volume is too small to justify the dependency.

Pattern / helper	Purpose
`bulletinURLRe`	Filters sitemap `<loc>` entries to security-bulletin-CVE paths
`cveRe`	`CVE-YYYY-NNNN+` extraction (case-insensitive)
`cvssHeaderRe`	Locates the CVSS-bearing table by matching the `CVSSvN.N` header cell
`extractTableCVSS`	Walks that table for a `<td><p>X.Y</p></td>` cell whose stripped value parses as a 0–10 base score
`extractTableVersions`	Walks the same table’s header + data row to extract the “Affected” and “Fixed” version columns
`cvssVectorRe`	Prose-form `CVSS:N.N/...` vector capture (fallback when the table is missing)
`articleTitleRe`	Helpjuice `<h1 data-helpjuice-element="Header Article Title">`
`bulletinIDRe`	`Bulletin ID: DB-YYYY-NN` extraction from the header prelude
`sectionStartRe` / `anySectionRe`	Section delimiter scan keyed on `<h1 id="(problem
`publicationDateRe`	`Publication Date: YYYY-MMM-DD` Helpjuice convention
`publicationDateAltRe` / `anyLongDateRe`	`Month DD, YYYY` fallback (older or future bulletin formats)

The product name is derived from the URL slug between security-bulletin- and -vulnerability-advisory- (or -cve-), title-cased with JDBC / ODBC / SQL / API / CLI / SDK preserved in uppercase. Example:

.../security-bulletin-databricks-jdbc-driver-vulnerability-advisory-cve-2024-49194
                     └──────product slug──────┘
                       → "Databricks JDBC Driver"

Published-date strings parse to Unix seconds via time.Parse. Two layouts are tried: the Helpjuice convention 2006-Jan-2 / 2006-Jan-02 (used for the live 2024-DEC-11 JDBC bulletin; the month token is case-normalised before parsing because Go’s layout matcher is case-sensitive), and the long-form January 2, 2006 family as a fallback for older or future bulletin formats. Unparseable dates leave DatePublished nil (the osv int4 convention).

Non-breaking spaces (U+00A0) inside table cells — Helpjuice appends them as visual padding after <br><br> — are normalised to ASCII space in ParseDetailPage before the cell regexes run, so \s* matches them.

Storage

processor.StoreCVESourceData writes every record inside one transaction per CVE.

Table	Rows inserted
`CVEMetadata`	One per CVE; `source="databricks"`, `sourceAdvisoryRef=KB URL`, `affectedVendor="Databricks"`, `affectedProduct=<slug-derived label>`, `sourceFileHash=sha256(page HTML)`
`CVEDescription`	One per CVE; `lang="en"`, synthesised from `Problem` + `Cause:` + `Solution:` blocks
`CVEMetadataReferences`	KB URL (`type=advisory`) + MITRE CVE record link (`type=technical`)
`CVEMetric`	One `cvssV3_1` row when a vector or score parses; `BaseScore` + `BaseSeverity` derived locally
`CVEAffected` / `CVEAffectedVersion`	One affected-product row with `Affected Versions` (`status=affected`) and `Fixed Versions` (`status=fixed`) when present
`CVEAlias`	Same-cveId cross-source edges via `db.InsertAliases` — links `(CVE-YYYY-NNNN, databricks)` to every peer source (`nist-nvd`, `cve.org`, `vulncheck-nvd`, `circl`, `coalition_cess`, `gitlab`, …) carrying the same CVE. The `Bulletin ID: DB-YYYY-NN` identifier is also passed in as an alias candidate; it is currently a no-op because no processor publishes `CVEMetadata` rows keyed `cveId='DB-YYYY-NN'`, but it lets the edge materialise automatically if one ever does

Identifier Policy

Case	`cveId`	`Aliases` passed to `InsertAliases`
Bulletin has a primary CVE in the URL	`CVE-YYYY-NNNN+`	Other CVEs from the bulletin (multi-CVE case) + `DB-YYYY-NN` Bulletin ID
Bulletin without a CVE in the URL	not ingestible — skipped by the discovery filter	—

Databricks KB pages do carry an internal Bulletin ID: DB-YYYY-NN identifier inside the article body prelude. The mapper extracts it and appends it to Aliases, but no existing processor publishes CVEMetadata rows keyed on DB-YYYY-NN, so the central db.InsertAliases pass treats it as “alias target not in DB” and logs it at debug level. The behaviour is forward-compatible: if a Databricks ledger processor ever appears, the alias edge materialises on the next ingest cycle without any code change here.

Multi-CVE bulletins emit one CVEMetadata row per CVE, all sharing the same sourceAdvisoryRef, title, affectedProduct, and page hash. Each row’s Aliases list contains the other CVEs from the bulletin.

Incremental Strategy

Single-tier resume: the sitemap has no <lastmod>, so there is no feed-level freshness gate. Instead, on startup the processor loads SELECT DISTINCT "sourceAdvisoryRef" FROM "CVEMetadata" WHERE source='databricks' into a set; any URL already in that set is skipped unless --force / --all is passed. New bulletins land in the next daily run.

Rate limit: 500 ms between detail fetches. Sitemap is fetched with a 3-attempt retry budget; each detail page has a 20-second per-request timeout.

S3 Persistence

Per the S3 Persistence Contract, both paths use the canonical helpers — no hand-rolled keys.

Path	When	Helper
`databricks/files/{sha256}/{slug}.html`	Detail page stored to DB successfully	`Uploader.Archive(ctx, "databricks", hash, "security-bulletin-...html", html)`
`failed-feeds/databricks-fetch-processor/{YYYY-MM-DD}/store-error/{filename}`	Transaction rollback during DB upsert	`Uploader.Quarantine(...)`
`failed-feeds/databricks-fetch-processor/{YYYY-MM-DD}/parse-error/{filename}`	Detail page parsed but no CVEs mapped	`Uploader.Quarantine(...)`
`failed-feeds/databricks-fetch-processor/{YYYY-MM-DD}/parse-error/sitemap.xml`	Sitemap XML decode failed	`Uploader.Quarantine(...)`

The processor does not quarantine fetch failures by default — the sitemap fetch is fatal (notifier.Errored + os.Exit), and per-bulletin fetch failures are surfaced via RecordError only (no payload bytes available to quarantine).

Slack Notifications

internal/notify emits the standard four lifecycle events plus per-bulletin warnings via RecordError:

Event	Trigger	Stats
`Started`	Process begins	—
`NoWork`	Sitemap has no matching security-bulletin-CVE URLs	reason string
`Completed`	Clean finish, all error counters zero	`{discovered, stored, skipped, fetchErrors, parseErrors, storeErrors}`
`Errored`	Fatal error OR any per-item error counter > 0	Same stats + error context
`RecordError` (warn)	Per-bulletin fetch/parse/store failure	One line per failed item, batched into the closing summary

SetOvertimeCancel wires notifier to cancel the request context when the soft deadline fires on ECS scheduled runs.

Flags

Flag	Default	Description
`--all`	false	Reprocess every discovered bulletin (alias for `--force`)
`--limit`	0	Cap bulletins processed per run (0 = unlimited)
`--force`	false	Bypass `sourceAdvisoryRef` resume set

ECS Schedule

Property	Value
Cron	`cron(0 6 ? * SAT *)` — weekly Saturdays 06:00 UTC
CPU	256
Memory	512 MB
Expected duration	20 min (well under for current 1-bulletin volume; budget left for future growth)

Soft deadline applies only when EXPECTED_DURATION_MINUTES is set (ECS-only). Local backfills via just go-databricks-fetch-backfill run to completion per the AGENTS.md backfill rule.

Architecture

flowchart TD A[EventBridge cron 06:00 UTC weekly Saturdays] --> B[ECS Fargate task] B --> C[GET kb.databricks.com/sitemap.xml] C -->|fetch failed| Z1[notify Errored → exit] C -->|ok| D[DiscoverSecurityBulletinURLs filter] D -->|0 matches| Z2[notify NoWork → exit] D -->|N URLs| E[loadProcessedAdvisories from CVEMetadata] E --> F{For each bulletin URL} F -->|limit reached or soft deadline| Y[notify Completed or Errored] F -->|already in resume set & not --force| K[skipped++ → next] F -->|new or --force| G[GET detail page with 20s timeout] G -->|fetch failed| Q0[RecordError → fetchErrors++] G -->|ok| H[ParseDetailPage + MapBulletin] H -->|0 CVEs mapped| Q1[Quarantine parse-error → RecordError] H -->|N CVEs| I[For each CVE: db.WithTx → StoreCVESourceData] I -->|tx failed| Q2[Quarantine store-error → RecordError] I -->|ok| J[Archive HTML to s3://databricks/files/sha256/] J --> F Q0 --> F Q1 --> F Q2 --> F K --> F Y --> Z3[exit]

flowchart LR F[Fetch detail page] --> P[Parse HTML + Map CVEs] P -->|0 CVEs| QP[Quarantine parse-error] P -->|N CVEs| T{Tx commit per CVE} T -->|fail| QS[Quarantine store-error] T -->|ok| A[Archive databricks/files/sha256/] A --> M[mark URL processed]

Key Files

File	Purpose
`cmd/databricks-fetch-processor/main.go`	Orchestration, sitemap discovery, fetch loop, S3 + notifier wiring
`internal/databricks/types.go`	`Bulletin`, `Sitemap`, `SitemapLoc` structs + source constants
`internal/databricks/parser.go`	Sitemap filter + KB-page regex parser (`DiscoverSecurityBulletinURLs`, `ParseDetailPage`)
`internal/databricks/mapper.go`	`MapBulletin(b Bulletin) []osv.CVESourceData` — one row per CVE
`internal/databricks/parser_test.go`	Sitemap-filter + JDBC-bulletin golden tests

Future Work

Org-level GitHub Security Advisories — if Databricks adopts the org-level GHSA tab on github.com/databricks, the existing ghsa-git-processor will cover these advisories with richer per-CVE metadata; this processor would then become redundant for new bulletins.
CPE-vector-grounded CWE inference — if KB advisories ever ship without a CWE assignment, the planned CPE-dictionary vector embedding feeding the existing vulnetix.cwe pass would be the right place to fill the gap. Today every Databricks bulletin discovered carries a CVE in the URL and the upstream CWE is already present, so no aienrich pass adds value here.
Sub-sitemap pagination — if kb.databricks.com starts emitting a sitemap index (<sitemapindex> + child <sitemap> entries) when the KB grows, DiscoverSecurityBulletinURLs will need a recursion step.

S3 Persistence

Archive path: databricks/files/{sha256}/{filename} ✓
Quarantine path: failed-feeds/databricks-fetch-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
Failure reasons emitted: parse-error, store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[databricks-fetch-processor] PROC -->|success| ARCHIVE[("S3: databricks/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/databricks-fetch-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.