CERT-TW Fetch Processor Design

Overview

Fetches Taiwan CERT (TWCERT/CC) vulnerability advisories from the paginated HTML listing at https://www.twcert.org.tw/tw/lp-132-1-{page}-60.html. For each new advisory, both the Chinese (TW) and English (EN) detail pages are fetched to obtain CVSS scores, descriptions, and metadata.

• Source: cert-tw
• Type: fetch (plain HTTP — no Cloudflare challenge)
• Primary ID: TVN-ID (cveId in CVEMetadata)
• Aliases: CVE-IDs stored in CVEAlias
• Language: Bilingual — zh-TW (description/title from TW page) and en (from EN page)

Source Characteristics

Architecture

No Browser Required

The TWCERT/CC listing and detail pages are plain HTML served without Cloudflare or JavaScript challenges. Standard Go net/http requests with browser-mimicking headers are sufficient.

Listing Page Structure

<table summary="此為TVN...">
  <tr>
    <td data-title="TVN ID"><span>TVN-202603009</span></td>
    <td data-title="標題"><span>
      <a href="/tw/cp-132-10805-a53f6-1.html">Gigabyte｜Performance Library - Insecure Deserialization</a>
    </span></td>
    <td data-title="CVE ID"><span>CVE-2026-4416</span></td>
  </tr>
</table>

Detail Page Structure

<!-- TW page: /tw/cp-132-{id}-{slug}-1.html -->
<!-- EN page: /en/cp-139-{id}-{slug}-2.html -->
<table class="Normal_table">
  <tr><th>TVN ID</th><td>TVN-202603009</td></tr>
  <tr><th>CVE ID</th><td>CVE-2026-4416</td></tr>
  <tr><th>CVSS</th><td>7.8 (High)<BR/>CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H</td></tr>
  <tr><th>影響產品</th><td>Performance Library 25.12.31.01 prior versions</td></tr>
  <tr><th>問題描述</th><td>[Chinese description]</td></tr>
  <tr><th>解決方法</th><td>[Chinese solution]</td></tr>
  <tr><th>公開日期</th><td>2026-03-30</td></tr>
</table>
<span class="update">更新日期：<em>2026-03-31</em></span>

EN URL is derived from the TW URL by substituting /tw/ → /en/, cp-132- → cp-139-, and -1.html → -2.html.

Pagination Strategy

Pages are numbered 1-indexed in the URL path: lp-132-1-{page}-60.html. Pagination stops when:

1. A page returns 0 advisory rows (last page reached), or
2. A page returns fewer rows than PageSize (60), or
3. Soft deadline is exceeded, or
4. Two consecutive pages where all items are already processed (early-stop for daily runs, skipped when --all)

Data Flow

EventBridge cron (daily, 06:00 UTC)
  → ECS Fargate task (cert-tw-fetch-processor, scratch image)
    → loadProcessedTVNIDs from CVEMetadata WHERE source='cert-tw' AND cveId LIKE 'TVN-%'
    → loop pages (page=1, 2, 3, ...):
        fetchPageWithRetry → listing HTML
        ParseListingPage → []ListingItem (TVN-ID, title, CVE-IDs, detail URL)
        for each new item:
          fetchPageWithRetry (TW detail page)
          fetchPageWithRetry (EN detail page, soft-fail)
          ParseAdvisoryPage → *Advisory (CVSS, descriptions, solution, pub date)
          MapFetchAdvisory → *CVESourceData (TVN-ID primary, CVE-IDs as aliases)
          StoreCVESourceData (per-advisory transaction)
        stop if all-known pages reached (non-backfill) or last page
    → notifier.Completed → SNS → Slack (on errors or overtime)

Package Structure

internal/certtw/
  types.go    — Advisory struct, Source/FetchBaseURL/ListingPathFmt/PageSize constants, ListingItem type
  parser.go   — ParseListingPage (listing table), ParseAdvisoryPage (detail Normal_table), ParseFeed (RSS)
  mapper.go   — MapFetchAdvisory (TVN-ID primary), MapAdvisory (CVE-ID primary, used by RSS processor)
  browser.go  — (not used by fetch processor)

cmd/cert-tw-fetch-processor/
  main.go     — flags, pagination loop, detail page fetching, DB writes, error alerting

Key Types

// ListingItem — from listing page table row
type ListingItem struct {
    TVNID     string   // e.g. "TVN-202603009"
    Title     string   // advisory title
    DetailURL string   // full TW advisory detail URL
    CVEIDs    []string // CVE IDs from listing column
}

// Advisory — fully-parsed from detail pages (reused from RSS processor)
type Advisory struct {
    TWID             string
    TWURL            string
    ENURL            string
    TWTitle          string   // Chinese title
    ENTitle          string   // English title
    TWDescription    string   // zh-TW description + affected products + solution
    ENDescription    string   // en description + affected products + solution
    CVEIDs           []string // stored as aliases
    CVSSVector       string
    CVSSScore        float64
    CVSSSeverity     string
    AffectedProducts string
    Solution         string
    ReferenceURLs    []string
    PubDate          string
    PubDateUnix      int64
    FileHash         string
}

Parsing Strategy

ParseListingPage: Parses <tr> elements from the listing table. For each row, finds <td data-title="TVN ID">, <td data-title="標題">, and <td data-title="CVE ID"> cells. Extracts TVN-ID (text), detail URL (<a href>), title (text), and CVE IDs.

ParseAdvisoryPage (reused from RSS processor): Parses <table class="Normal_table"> rows from TW and EN detail pages. Extracts th/td key-value pairs for TVN-ID, CVE-ID, CVSS, affected products, description, solution, and publish date.

Mapping Strategy

MapFetchAdvisory (new function for fetch processor):

• cveId = TVN-ID (e.g. TVN-202603009)
• source = cert-tw
• title = English title (falls back to Chinese title)
• Descriptions[0] = Chinese description with lang="zh-TW"
• Descriptions[1] = English description with lang="en" (if available)
• Aliases = CVE-IDs from the advisory
• Metrics = CVSS vector and score
• rawDataJSON = full advisory fields including tw_id, cve_ids, titles, CVSS, pub date

Note: The RSS processor (cert-tw-rss-processor) uses MapAdvisory which creates one record per CVE-ID. The fetch processor uses MapFetchAdvisory which creates one record per TVN-ID. Both processors write to source='cert-tw' but with different primary ID schemes; loadProcessedTVNIDs filters to cveId LIKE 'TVN-%' to avoid confusion.

Error Alerting

Uses the existing SNS → Lambda → Slack pipeline (internal/notify):

Fatal error conditions that exit immediately:

• 3 or more consecutive listing page fetch failures

Per-item detail page fetch failures are accumulated via RecordError but do not halt the run (the advisory is stored with listing data as fallback).

Container

Uses FROM scratch (no Chromium needed — plain HTTP):

FROM scratch AS cert-tw-fetch-processor
COPY --from=cert-builder /etc/ssl/ /etc/ssl/
COPY --from=builder --chmod=0755 /out/cert-tw-fetch-processor /app/cert-tw-fetch-processor
ENTRYPOINT ["/app/cert-tw-fetch-processor"]

ARM64: The ECS task runs on ARM64 (cpuArchitecture: ARM64). Go cross-compilation handles this without issues.

Scheduling

The pagination loop stops at that soft deadline (cmd/cert-tw-fetch-processor/main.go:108-111). When EXPECTED_DURATION_MINUTES is unset the deadline falls back to a hardcoded 20 minutes — and just go-cert-tw-fetch-backfill unsets the variable, so a --all history walk is cut off after 20 minutes (~1.5 s per advisory ⇒ a few hundred advisories) instead of running to completion. Re-run the recipe until the “last page reached” line appears, or set EXPECTED_DURATION_MINUTES high for the run.

Local Usage

# Full history backfill (all pages)
just go-cert-tw-fetch-backfill prod true 0

# Recent only, limit 60 (one page)
just go-cert-tw-fetch-backfill prod false 60

# Local DB (uses .env)
just go-cert-tw-fetch-backfill local false 20

# Force reprocess (overwrites existing records)
just go-cert-tw-fetch-backfill prod false 0 true

Verification Queries

-- Count stored records (TVN-ID primary)
SELECT COUNT(*) FROM "CVEMetadata" WHERE source = 'cert-tw' AND "cveId" LIKE 'TVN-%';

-- Recent advisories with titles
SELECT "cveId", title, "datePublished"
FROM "CVEMetadata"
WHERE source = 'cert-tw' AND "cveId" LIKE 'TVN-%'
ORDER BY "datePublished" DESC NULLS LAST
LIMIT 10;

-- CVE aliases for a TVN advisory
SELECT "primaryCveId", "aliasCveId"
FROM "CVEAlias"
WHERE "primaryCveId" LIKE 'TVN-%'
LIMIT 10;

-- CVSS scores
SELECT "cveId", "vectorString"
FROM "CVEMetadata"
WHERE source = 'cert-tw' AND "cveId" LIKE 'TVN-%' AND "vectorString" IS NOT NULL
LIMIT 5;

-- Bilingual descriptions
SELECT d.lang, LEFT(d.value, 100)
FROM "CVEDescription" d
JOIN "CVEMetadata" m ON d."cveId" = m."cveId"
WHERE m.source = 'cert-tw' AND m."cveId" LIKE 'TVN-%'
LIMIT 10;

Known Limitations

1. No date filtering: The listing URL has no date parameter. The processor relies on the processed-TVN-ID set for incremental runs and early-stop logic for daily use.
2. N+1 detail fetches: Each new advisory requires 2 additional HTTP requests (TW + EN detail pages). Full history backfill of ~2000 advisories may take 30–60 minutes.
3. Coexistence with RSS processor: Both processors write to source='cert-tw'. The RSS processor uses CVE-IDs as primary; the fetch processor uses TVN-IDs. The loadProcessedTVNIDs query filters to cveId LIKE 'TVN-%' so this processor is not confused by the RSS processor’s rows — but the reverse is not true. The RSS processor dedupes on sourceAdvisoryRef, and both processors write the TW advisory URL into that column, so every advisory this processor stores first (06:00 UTC vs the RSS processor’s 07:00 UTC) is skipped by the RSS processor and never gets per-CVE rows. Production currently holds 511 TVN- rows and 42 CVE- rows under source='cert-tw'. Treat the TVN row plus its CVEAlias CVE edges as the canonical shape for this source.
4. Detail page fallback: If a detail page fetch fails, only listing data (TVN-ID, title, CVE-IDs) is stored — no CVSS, no description, no publish date.
5. EN page optional: The English detail page is fetched as a best-effort. Advisories without an EN page store only Chinese descriptions.

S3 Persistence

• Archive path: cert-tw/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/cert-tw-fetch-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.