adobe-security-fetch-processor

Status: Live Source: Adobe Security Bulletin Type: fetch (HTML scraping) Source slug: adobe Schedule: Weekly, Tuesday 06:00 UTC (cron(0 6 ? * 2 *)), 256 CPU / 512 MB, EXPECTED_DURATION_MINUTES=30.

Overview

Ingests Adobe Product Security Bulletins (APSB) and Advisories (APSA) from the public Adobe security portal. Adobe publishes security updates for ~70 product lines (Acrobat, Photoshop, ColdFusion, Experience Manager, Commerce, etc.). Each bulletin detail page carries a “Vulnerability details” table with per-CVE metadata (CVSS v3.1, CWE, severity, impact) plus affected-version and solution data.

Each CVE in a multi-CVE bulletin becomes its own CVEMetadata row under source="adobe", with the APSB/APSA ID stored as a CVEAlias. Bulletins without CVE IDs (legacy pre-2012 APSA) are stored under the APSB/APSA ID itself. The Adobe PSIRT blog (blogs.adobe.com/psirt) is defunct — helpx is the sole authoritative source.

Source

Property	Value
Master list URL	`https://helpx.adobe.com/security/security-bulletin.html`
Auth	None — fully public
Format	HTML (no RSS, no JSON API)
Pagination	None — entire archive on one page
Total bulletins	~800 (APSB + APSA, back to 2005)
Cadence	Patch Tuesday (~5–15 new bulletins per month)
CVEs per bulletin	1–15
Identifier shape	`APSB\|APSA{YY}-{NN}`; nearly all carry `CVE-YYYY-NNNN+`

Page Structure

Master list — product sections (## Adobe {Product}) with rows linking to detail pages:

Column	Source
Bulletin link	Link text and `href` (e.g. `APSB26-44` → `/security/products/acrobat/apsb26-44.html`)
Title	Link text after the ID
Originally posted	`MM/DD/YYYY`
Last updated	`MM/DD/YYYY`

Detail page — three tables of interest:

Header — Date Published, Priority (1–3), Summary paragraph
Vulnerability details — per-CVE row: CVE Number, Vulnerability Category (linked CWE + text), Vulnerability Impact, Severity, CVSS base score, CVSS vector
Affected versions / Solution — Product, Affected Versions, Platform, Updated Version, Availability link

Parsing

internal/adobe/parser.go uses compiled regexp (no DOM tree — fast on 800-page batches):

Pattern	Purpose
`productHeadingRe`	Splits master list into per-product sections
`bulletinRowRe` + `bulletinLinkRe`	Extracts each `(BulletinID, Title, URL, DatePublished, DateUpdated)`
`vulnTableRe` + `vulnRowRe` + `vulnCellRe`	Tokenises the vulnerability-details table on detail pages
`cveRe`	`CVE-YYYY-NNNN+` extraction with case-insensitive match
`cvssVectorRe`	Extracts `CVSS:3.1/...` vector strings
`htmlTagRe`	Strips HTML for clean text cells

Date strings (MM/DD/YYYY and April 14, 2026) parse to Unix seconds via ParseDate. datePublished prefers the detail page’s Date Published cell, falls back to the “Last updated on {Month D, YYYY}” line, then to the master-list row’s “Originally posted” cell. If all three fail, 0 is written — the column is NOT NULL integer, and mapSingle sets the pointer unconditionally rather than leaving it nil. Legacy APSA entries without a vulnerability-details table fall through to summary-text regex extraction; if still no CVE, the bulletin is stored under its APSB/APSA ID.

Master-list parse history. bulletinLinkRe originally required the bulletin id to be wrapped in <strong>. Adobe serves three shapes (<b>, <strong>, bare), so the parse silently collapsed from 814 bulletins to 103, and the master-list table’s “Originally posted” / “Last updated” <td> cells were never read at all — which is why every stored row carried datePublished = 0. Both are fixed (internal/adobe/parser.go now reads rows before anchors and accepts all three id shapes).

Storage

processor.StoreCVESourceData handles all per-CVE writes inside one transaction per CVE.

Table	Rows inserted
`CVEMetadata`	One per CVE; `source="adobe"`, `sourceAdvisoryRef=detail URL`, `affectedVendor="Adobe"`, `affectedProduct=product slug`, `sourceFileHash=sha256(detail HTML)`
`CVEDescription`	One per CVE; `lang="en"`, synthesised from summary + vuln category + impact + severity
`CVEMetadataReferences`	Three at most: the bulletin detail URL (`type=advisory`, `referenceSource=Adobe`), the MITRE CVE link (`type=advisory`, `referenceSource=MITRE`), and the solution tech-note URL when one was found (`type=patch`)
`CVEMetric`	Per-CVE `cvssV3_1` with vector + base score + severity derived from the score — only when the detail page yielded a vector string
`CVEProblemType`	CWE + category text — only when the vulnerability-details table yielded a `CWE-nnn`
`CVEAffected` / `CVEAffectedVersion`	One `CVEAffected` per row of the “Affected Versions” table (vendor `Adobe`), with a single `status="affected"` `CVEAffectedVersion` carrying the raw version text. No `"fixed"` rows are emitted — the Solution table’s “Updated Version” column is not mapped.
`CVEAlias`	`(CVE-YYYY-NNNN, adobe) ↔ (APSB{YY}-{NN}, adobe)` via `db.InsertAliases` (bundle suppression keeps each CVE’s APSB edge independent)
`BulkDataDumpTracker`	One row per source; `sha256=hash(master list HTML)`, `totalCVEs=bulletins processed`

Identifier Policy

Case	`cveId`	`CVEAlias`
Bulletin has CVE(s)	`CVE-YYYY-NNNN`	`APSB{YY}-{NN}`
Advisory has CVE(s)	`CVE-YYYY-NNNN`	`APSA{YY}-{NN}`
Legacy entry has no CVE	`APSB{YY}-{NN}` or `APSA{YY}-{NN}`	none

Multi-CVE bulletins (e.g. APSB26-38 ColdFusion = 7 CVEs) emit one CVEMetadata row per CVE, all sharing the same sourceAdvisoryRef, title, and affectedProduct. Each row holds an independent (cveId, adobe) ↔ (APSB26-38, adobe) alias edge.

Incremental Strategy

Two-tier resume — both gated by --force:

Tracker freshness gate — at startup, db.GetTracker(adobe) is compared against tracker.frequency (currently 86400 s = 1 day in production). Because the schedule is weekly, this gate never actually fires on a scheduled run; it only protects against back-to-back manual invocations. Note the gate is on elapsed time, not on the stored master-list sha256 — the hash is written but never read.
Per-bulletin hash skip — db.LoadProcessedHashes(adobe) returns map[cveId]sha256. Each detail page is fetched first, then its SHA-256 is compared against every value in that map; a match skips parsing and storing. Two consequences worth knowing: the fetch cost is paid even for unchanged bulletins, and the comparison is a linear scan of the map per bulletin rather than a keyed lookup.

Rate limit: 1 second between detail fetches (skipped bulletins do not pay the sleep), so a full ~800-bulletin archive walk is roughly 15 minutes of pacing plus fetch latency.

CRIT Staging (Optional)

With --emit-crit=true, per-CVE CRIT (Cloud Resource Inventory Tag) candidate envelopes are staged via critutil.StageCandidate. After the bulletin loop completes, critpublisher.DrainKeys ingests staged candidates into the cloud-resource inventory — useful for Adobe SaaS products (Experience Manager Cloud Service, Commerce-as-a-Service) where the bulletin maps to a (Provider, Service, ResourceType) triple. Desktop products (Acrobat, Photoshop) produce no candidates.

CRIT publication is disabled when CRIT_PUBLISH_DISABLED=true or when AI spec dictionaries fail to load (warn-only).

S3 Persistence

Per the S3 Persistence Contract, both paths use the canonical helpers — no hand-rolled keys.

Path	When	Helper
`adobe/files/{sha256}/{APSB-id}.html`	Detail page stored to DB successfully	`Uploader.Archive(ctx, "adobe", hash, "APSB26-44.html", html)`
`failed-feeds/adobe-security-fetch-processor/{YYYY-MM-DD}/store-error/{filename}`	Transaction rollback during DB upsert	`Uploader.Quarantine(...)`

store-error is currently the only reason wired. A detail-page fetch failure and a “no CVEs mapped” parse failure both RecordError and increment failed but do not quarantine — for the fetch case there are no bytes to keep, and for the parse case the HTML is in hand but is not uploaded. Both are gaps against the contract rather than intentional.

Slack Notifications

internal/notify emits the standard four lifecycle events plus per-bulletin warnings via RecordError:

Event	Trigger	Stats
`Started`	Process begins	—
`NoWork`	Tracker fresh & not `--force`	reason string
`Completed`	Clean finish, `failed == 0`	`{fetched, stored, skipped, failed, bulletins, critStaged}`
`Errored`	Fatal error or `failed > 0`	Same as `Completed` + error context
`RecordError` (warn)	Per-bulletin fetch/map/store failure	one line per failed item, batched into the closing summary

SetOvertimeCancel wires notifier to cancel the request context when the soft deadline fires on ECS scheduled runs.

Flags

Flag	Default	Description
`--force`	false	Bypass tracker freshness check and re-process all bulletins
`--limit`	0	Cap bulletins per run (0 = unlimited)
`--emit-crit`	false (cli) / true (ECS)	Stage CRIT candidates and drain post-loop

ECS Schedule

Property	Value
Cron	`cron(0 6 ? * 2 *)` — Tuesday 06:00 UTC
CPU	256
Memory	512 MB
Expected duration	30 min (incremental); ~15 min for a full archive backfill

Soft deadline is applied only when EXPECTED_DURATION_MINUTES is set (ECS-only). Local backfills via just go-adobe-security-fetch-backfill run to completion per the AGENTS.md backfill rule.

Architecture

flowchart TD A[EventBridge cron Tue 06:00 UTC] --> B[ECS Fargate task] B --> C{Tracker fresh & not --force?} C -->|yes| Z1[notify NoWork → exit] C -->|no| D[GET master list HTML] D --> E[ParseMasterList → entries] E --> F[LoadProcessedHashes from CVEMetadata] F --> G{For each bulletin} G -->|limit reached or soft deadline| Y[Drain CRIT → UpsertTracker] G -->|next| H[GET detail page with retry] H -->|fetch failed| Q1[Quarantine fetch-error → RecordError] H -->|ok| I[sha256 detail HTML] I --> J{Hash already processed & not --force?} J -->|yes| K[skipped++ → next] J -->|no| L[ParseDetailPage + MapBulletin] L -->|no CVEs mapped| Q2[Quarantine parse-error → RecordError] L -->|CVEs| M[For each CVE: db.WithTx → StoreCVESourceData] M -->|tx failed| Q3[Quarantine store-error → RecordError] M -->|ok| N[Archive detail HTML to S3] N --> O{--emit-crit?} O -->|yes| P[CRIT NewEnvelope + StageCandidate per CVE] O -->|no| G P --> G Y --> Z2[notify Completed or Errored]

flowchart LR F[Fetch detail page] --> H[sha256] H --> R{hash in processed set?} R -->|yes & not --force| S[Skip] R -->|no| P[Parse + Map] P -->|0 CVEs| QP[Quarantine parse-error] P -->|N CVEs| T{Tx commit per CVE} T -->|fail| QS[Quarantine store-error] T -->|ok| A[Archive adobe/files/sha256/]

Key Files

File	Purpose
`cmd/adobe-security-fetch-processor/main.go`	Orchestration, freshness gate, fetch loop, tracker, notifier, CRIT drain
`cmd/adobe-security-fetch-processor/crit_mapper.go`	Maps Adobe SaaS products → CRIT candidate envelopes
`internal/adobe/types.go`	`BulletinEntry`, `Vuln`, `AffectedProduct`, `BulletinDetail`
`internal/adobe/parser.go`	Master list + detail page regex parsing
`internal/adobe/mapper.go`	`MapBulletin(detail, entry) []osv.CVESourceData`
`internal/adobe/parser_test.go`	Golden tests against recorded fixtures

Future Work

CPE-vector-grounded CWE inference for the small number of legacy pre-2012 APSA entries with no CVE assignment. The CWE pass (the surviving aienrich gap-filler) does not currently wire on fetch-class processors; the planned CPE-dictionary vector embedding is what would make low-context records like these worth running through it. Deferred until that pipeline exists.

S3 Persistence

Archive path: adobe/files/{sha256}/{filename} ✓
Quarantine path: failed-feeds/adobe-security-fetch-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
Failure reasons emitted: store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[adobe-security-fetch-processor] PROC -->|success| ARCHIVE[("S3: adobe/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/adobe-security-fetch-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.