pwno-fetch-processor — Design

Status: schedule DISABLED (schedule_enabled = false in terraform/go-schedules.tf, 2026-07-10). The upstream source is gone — https://bugs.pwno.io/ returns 402 Payment Required / DEPLOYMENT_DISABLED (a disabled Vercel deployment) from every network, not just AWS egress. The task, image and code are intact; re-enable the schedule when the deployment is restored. The design below describes the intended behaviour.

Source: bugs.pwno.io (Next.js RSC payload) Type: fetch (HTML scrape of an embedded JSON array) Source slug: pwno Schedule: daily at 05:45 UTC (cron(45 5 * * ? *)) — currently disabled

Daily ingest of Pwno’s public bug-disclosure index into CVEMetadata under the pwno source namespace. Pwno is the AI-driven security research startup founded by Ruikai Peng; they publish 50+ memory-safety bugs in major open-source projects (FFmpeg, Chromium, Firefox, WebKit, Redis, Postgres, Linux, Adobe DNG, …) under their own PWNO-NNNN namespace pending CVE assignment.

There is no JSON / RSS / Atom feed. The index is a server-rendered Next.js page; the bug list is hydrated from a JSON array embedded in the RSC payload. We parse that embedded JSON directly — never the rendered DOM.

Architecture

graph LR EB[EventBridge
daily cron] --> ECS[ECS Fargate Task
go-pwno-fetch-processor] ECS --> Tracker{BulkDataDumpTracker
fresh?} Tracker -- yes --> Skip[no-op
notify NoWork] Tracker -- no --> HTTP[GET https://bugs.pwno.io/] HTTP --> Parse[parseBugsJSON
RSC chunk → bugs array] Parse --> Loop[for each pwnoBug] Loop --> Hash{prev sourceFileHash
== row hash?} Hash -- yes --> Unchanged[unchanged++] Hash -- no --> Tx[BEGIN tx] Tx --> Meta[(CVEMetadata
upsert)] Tx --> Desc[(CVEDescription
delete + insert en)] Meta --> Commit[COMMIT] Desc --> Commit Loop --> TrackerW[(BulkDataDumpTracker
upsert)] TrackerW --> Notify[notify Completed]

Data Flow

sequenceDiagram participant Cron as EventBridge (daily) participant Proc as pwno-fetch-processor participant Site as bugs.pwno.io participant DB as PostgreSQL Cron->>Proc: ECS RunTask Proc->>DB: GetTracker(source=pwno) DB-->>Proc: lastProcessedAt, frequency alt fresh & not --force Proc-->>Cron: NoWork (exit 0) end Proc->>Site: GET / (User-Agent: vdb-manager/...) Site-->>Proc: HTML w/ self.__next_f.push([1,"…"]) chunks Proc->>Proc: pushScriptRe.FindAll → decode each chunk Proc->>Proc: locate "bugs":[ → matchClosingBracket Proc->>Proc: json.Unmarshal → []pwnoFeedEntry loop for each entry Proc->>Proc: derive flags (redacted/rejected/patched/hasWriteup) Proc->>Proc: rowHash = sha256(canonical entry JSON) Proc->>DB: SELECT sourceFileHash WHERE cveId,source alt hash matches Proc->>Proc: unchanged++ else Proc->>DB: BEGIN Proc->>DB: INSERT … ON CONFLICT UPDATE CVEMetadata Proc->>DB: DELETE+INSERT CVEDescription (lang=en) Proc->>DB: COMMIT end end Proc->>DB: UpsertTracker(source=pwno, fetchHash, rows) Proc-->>Cron: Completed{inserted,updated,unchanged,failed}

Source → DB Mapping

The Pwno feed entry shape (pwnoFeedEntry) is the JSON object embedded in the RSC bugs array:

RSC field	Type	Meaning
`id`	string	`PWNO-NNNN`
`project`	string	e.g. `FFmpeg`, `Chromium`, `Linux`
`component`	string	sub-component or repeats `id`
`title`	string	one-line description, or `[REDACTED]`
`type`	string	e.g. `OOB Write`, `UAF`
`date`	string	`YYYY-MM-DD`
`pageId`	*string	non-null ⇒ writeup exists
`status`	string	`patched` \| `in_progress` \| `rejected`
`highlighted`	bool	site flag
`pinned`	bool	site flag
`comment`	string	populated when `status = rejected`

`CVEMetadata` columns

Column	Source / derivation
`cveId`	`entry.id` (PWNO-NNNN)
`source`	const `"pwno"`
`dataVersion`	const `"1.0"`
`state`	`RESERVED` if redacted • `REJECTED` if `status='rejected'` • else `PUBLISHED`
`datePublished`	`entry.date` parsed as UTC midnight, stored as epoch seconds
`title`	`entry.title`; if redacted/empty → `"{type} in {project} (under embargo)"`
`sourceAdvisoryRef`	`https://bugs.pwno.io/{N}` when `pageId` set, else `https://bugs.pwno.io/`
`affectedVendor`	`entry.project` (Pwno doesn’t separate vendor/product)
`affectedProduct`	`entry.project` (same value bound to `$7` twice in the SQL)
`lastFetchedAt`	`time.Now().UnixMilli()` at run
`rawDataJSON`	scrape envelope: original entry fields + `sourceUrl`, `scraper`, `scrapedAt`
`sourceFileHash`	`sha256(canonical entry JSON)` — used for drift detection
`fetchCount`	`+1` on every UPDATE branch

`CVEDescription` row (always one, lang=`en`, containerType=`cna`)

Branch	`value`
Has plain description	`entry.title`
Redacted / empty	`Pwno entry {id} — {type} in {project}, disclosed {date}. Description withheld pending coordinated disclosure.`
Rejected (any of the above)	append `(Rejected upstream: {comment})`

The description is fully replaced (DELETE + INSERT inside the same tx) so duplicates can’t accumulate.

`CVEAiDiscovery` (one per entry, always)

Every Pwno disclosure is by definition an AI-driven discovery — their deductive engine finds the bug and a researcher triages it — so each entry also gets an AI-discovery row that /api/vdb/v2/ai-discoveries and the website’s AI-Discovered Vulnerabilities article aggregate by aiOrg / aiSystem / harness. This is what makes the processor worth running even though the PWNO-NNNN ids never join the CVE graph.

Column	Value
`uuid`	`md5('ai-discovery:{id}:pwno')` — deterministic, so processor and manual-SQL paths converge
`aiOrg` / `aiSystem` / `harness`	`Pwno` / `Pwno deductive engine` / `Pwno deductive engine` (stable tags — changing them invalidates the article’s filters)
`humanAssisted`	`true`
`discoveryDate`	entry date in epoch ms (BigInt); falls back to now
`vulnType`	`entry.type`
`severityAtDiscovery`	`unknown` (Pwno publishes no per-bug CVSS)
`displayId` / `idType`	`PWNO-NNNN` / `OTHER`
`sourceUrl` / `notes`	writeup URL (or the index) / `status=…; writeup=…` (+ rejection comment)
`isNotable`, `milestoneType`, `cvssAtDiscovery`	set once on insert and never overwritten — editorial fields stay curated

The row is upserted on the unchanged path too, in its own small transaction, so entries that predate the markup feature get backfilled without touching CVEMetadata.

`CVEAlias`

db.InsertAliases(ctx, tx, id, "pwno", nil, …) runs on every changed entry — no explicit aliases exist, but the call keeps the same-cveId cross-source backfill contract every other processor follows. It is a no-op today because PWNO-NNNN ids are single-source.

`BulkDataDumpTracker` (`source = "pwno"`)

Column	Value
`source`	`pwno`
`lastProcessedAt`	`now()` at end-of-run
`frequency`	`86400` (daily, seconds — hard-coded by `db.UpsertTracker`)
`sha256`	`sha256(full HTML body)` of the index page
`totalCVEs`	`len(bugs)`

Business Logic

Freshness gate

Skip the run when now - tracker.lastProcessedAt < tracker.frequency unless --force is passed. This makes the schedule self-throttling — a manual RunTask mid-day will exit fast as NoWork rather than re-hammering the source.

Entry classification

Each pwnoFeedEntry is folded into a pwnoBug with derived booleans:

HasWriteup = pageId != nil && *pageId != "". When set, build WriteupURL = bugs.pwno.io/{N} where N is the integer suffix (leading zeros stripped).
IsRedacted = title == "[REDACTED]" (case-insensitive).
IsRejected = status == "rejected".
IsPatched = status == "patched".

The state column is the only place these collapse into a single CVE-style enum: REJECTED wins over RESERVED wins over PUBLISHED.

Idempotency / change detection

Two hashes work together:

Page hash (fetchHash) — sha256 of the full HTML body. Stored on the tracker; lets a future enhancement skip parsing entirely when nothing changed page-wide.
Row hash (RowHash) — sha256 of the canonicalized per-entry JSON. Compared against the previously stored CVEMetadata.sourceFileHash. If equal, the row is reported as unchanged and no transaction is opened — lastFetchedAt and fetchCount only bump on actual change.

Transaction shape

For every changed (or new) entry we open a single tx that:

Upserts CVEMetadata (ON CONFLICT (cveId, source)).
Deletes the existing lang=en CVEDescription row for (cveId, source).
Inserts a fresh CVEDescription row.

The DELETE+INSERT enforces the “exactly one English description” invariant cheaply without needing a unique index.

Failure handling

Per-row upsert failures are logged and counted into failed; the loop keeps going so one bad row never blocks the rest of the index.
Tracker write failure exits non-zero — we’d otherwise re-do the whole batch on next tick.
Any failed > 0 total exits non-zero so EventBridge / dashboards surface the partial failure, but the successful writes remain committed.

Scrape brittleness boundary

Resilient to:

Field reordering inside pwnoFeedEntry.
New fields appearing on the entry (ignored by json.Unmarshal).
Multiple self.__next_f.push([1, "…"]) chunks — we walk all of them and pick the one containing "bugs":[.
] characters appearing inside string values within the bugs array (handled by matchClosingBracket’s in-string tracking).

Brittle to (fails loudly with a parse error rather than silent bad data):

Pwno renaming the bugs key in the RSC payload.
Pwno switching away from Next.js RSC self.__next_f.push hydration.

S3 Persistence

Not used. This processor does not currently archive payloads or quarantine failures to S3. Per the S3 Persistence Contract this is non-compliant — see the compliance matrix for the implementation roadmap.

Expected paths when implemented:

Archive: pwno/files/{sha256}/{filename}
Quarantine: failed-feeds/pwno-fetch-processor/{YYYY-MM-DD}/{reason}/{filename}
Likely reasons: fetch-error