CERT-TW RSS Processor — Design (RETIRED)

Retired 2026-08-09

This processor has been removed from the fleet. It duplicated cert-tw-fetch with no incremental value: the same upstream site, the same parser (internal/certtw.ParseAdvisoryPage) and the same source='cert-tw' slug, differing only in how advisories were discovered. This one read rss-132-1.xml, a recent subset; cert-tw-fetch paginates lp-132-1-{page}-60.html, the full listing, so it reaches strictly more. Both schedules were ENABLED, so production was paying twice to produce the same 587 rows.

The document is kept because the RSS discovery path and its parsing decisions are worth having on record if the listing page ever changes shape.

Overview

Fetches vulnerability advisories from the Taiwan CERT (TWCERT/CC) RSS 2.0 feed, scrapes each TW and EN advisory page, and stores records keyed by CVE ID (source=cert-tw). Items with no CVE IDs are skipped. One CVEMetadata record is created per CVE ID; the TWCERT TVN ID is stored as an alias. Sibling CVE IDs are passed as aliases too but are dropped by the bundle-suppression guard in db.InsertAliases when an advisory lists more than one CVE (internal/db/cvealias.go:65-88) — the TVN alias always survives, because it is not CVE-prefixed.

Overlaps with cert-tw-fetch-processor. Both processors scrape the same TWCERT advisory pages with the same parser and write source='cert-tw'; the fetch processor keys rows by TVN-ID, this one keys them by CVE ID. Both also dedupe on CVEMetadata.sourceAdvisoryRef, and both store the TW advisory URL in that column — so once the fetch processor (06:00 UTC) has stored an advisory, this processor (07:00 UTC) sees the URL in its processed set and skips the item, never minting the per-CVE rows (cmd/cert-tw-rss-processor/main.go:130-132). In production that shows up as 511 TVN- rows against 42 CVE- rows under source='cert-tw'. Anything that needs the CVE-keyed view should traverse the TVN↔CVE CVEAlias edges the fetch processor writes rather than relying on this processor’s rows.

Feeds and Pages

URL Transformation (TW → EN)

The EN advisory URL is derived from the TW URL by three substitutions:

1. /tw/ → /en/
2. cp-132- → cp-139-
3. -1.html → -2.html

Example: https://www.twcert.org.tw/tw/cp-132-3541-aa2e4-1.html → https://www.twcert.org.tw/en/cp-139-3541-aa2e4-2.html

The EN page fetch is soft-fail: if it returns an error or contains no Normal_table, processing continues with TW-only data.

Page Structure

Each advisory page contains a class="Normal_table" HTML table with <th> (label) and <td> (value) rows. Field names differ by language:

Parsing

Description Assembly

Each language description is assembled as:

{Description text}

Affected Products:
{Affected Products text}

Solution:
{Solution text}

Empty sections are omitted. This ensures remediation context is captured without a dedicated schema column.

CVSS Notes

The CVSS table cell on multi-CVE advisories contains one vector per CVE separated by <BR/> tags (uppercase, self-closing). The processor uses a case-insensitive <br> regex in stripHTML and a strict vector-extraction regex in parseCVSS to avoid capturing trailing CVE labels or scores.

When a single advisory covers multiple CVEs with different vectors, all CVEs in that advisory share the first vector extracted. Per-CVE vector mapping is not currently implemented.

Date Formats

Handled formats (in priority order):

• RFC1123Z / RFC1123 (from RSS <pubDate>)
• RFC3339
• 2006/01/02, 2006-01-02
• 2006/01/02 15:04:05, 2006-01-02 15:04:05
• CJK format 2025年01月02日 (normalised to 2025/01/02 before parsing)

Storage

No new tables or columns.

Incremental Strategy

URL-based dedup via sourceAdvisoryRef in CVEMetadata. The set of already-processed TW advisory URLs is loaded at startup. Items whose URL is already present are skipped unless --all or --force is set.

Flags

Rate Limiting

500 ms delay between advisory page fetches (TW + EN count as one advisory pair). Each fetch retries up to 3 times on transient errors with exponential backoff (2 s, 4 s).

ECS Schedule

Daily (cron(0 7 * * ? *)), 256 CPU units, 512 MB memory, expected_duration_minutes=30.

Key Files

S3 Persistence

• Archive path: cert-tw/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/cert-tw-rss-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: parse-error, store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.