tencent-cloud-fetch-processor — Design (IMPLEMENTABLE)

1. Overview

• Tier: Tier-2 IaaS (Phase 4)
• Verdict: IMPLEMENTABLE — first-party CVE-bearing advisories reachable via paginated server-rendered HTML, no anti-bot
• Source: https://cloud.tencent.com/announce listing → https://cloud.tencent.com/announce/detail/{id} per advisory
• Owner: Tencent Cloud
• Licence: Public, no auth
• Schedule cadence: cron(0 15 * * ? *) — daily 15:00 UTC. Tencent posts 【安全通告】 (security advisories) at irregular cadence; daily check is sufficient given the monotonic ID enables incremental crawl.
• ECS resources: cpu_units=256, memory_mb=512, expected_duration_minutes=15
• Reads: External HTML, BulkDataDumpTracker (last-seen ID), LoadProcessedHashes (resume)
• Writes: CVEMetadata, CVEMetadataReferences, CVEAlias, BulkDataDumpTracker, S3 archive, optional CRIT staging
• Extended dictionary: internal/critutil/dictionaries/extended/tencent.json (§15.2)

2. Source contract (Phase 0 recon — 2026-05-08)

Recon-protocol grid:

Source contract table:

CVE presence verified:

$ curl -sS https://cloud.tencent.com/announce/detail/2277 | grep -oE 'CVE-[0-9]{4}-[0-9]+' | sort -u
CVE-2026-31431

Sample fixture: https://cloud.tencent.com/announce/detail/2277 — Linux Kernel “Copy Fail” CVE-2026-31431, dated 2026-04-30.

Language note: Content is Chinese-only. CVE refs (CVE-YYYY-NNNN), advisory IDs (numeric), URLs in references section, and dates (YYYY-MM-DD) are language-neutral and extractable by regex without NLP. The title prefix 【安全通告】 is a stable ASCII-bracketed sentinel for filtering security advisories from operational ones. Description text stored verbatim; downstream translation out of scope.

3. Architecture diagram

[Listing pages 1..N] ── HTTP fetch ──> regex filter `【安全通告】`
                                              │
                                              ▼
                            [advisory IDs] ── enumerate, skip seen
                                              │
                                              ▼
                            [GET /announce/detail/{id}]
                                              │
                                              ▼
                            tencent.ParseDetail ── extract:
                              - title (zh-CN), date, advisory_id
                              - CVE aliases (regex)
                              - severity (CVSS extracted from body)
                              - references (URLs in body)
                                              │
                                              ▼
                            tencent.MapToSourceData ── osv.CVESourceData
                                              │
                                              ▼
                            processor.StoreCVESourceData
                                              │
                                              ▼
                            (tx) UpsertCVEMetadata + InsertAliases +
                                 CVEDescription + CVEReference +
                                 BulkDataDumpTracker
                                              │
                                              ▼
                            optional CRIT staging per matched service
                                              │
                                              ▼
                            S3 archive (re-serialised JSON)

4. Source → DB field mapping

5. Identifier policy

• Primary cveId: First CVE alias by canonical sort. If no CVE alias exists, mint TC-{YYYY}-{NNNN} via db.LoadMaxGcveSequence (mirrors elastic ESA-YYYY-NN pattern from §15.6).
• Aliases written via db.InsertAliases — never raw INSERT (per AGENTS.md alias contract): the Tencent advisory ID prefix tencent-{integer} plus every CVE-YYYY-NNNN extracted from title + body.
• source="tencent-cloud" on every CVEMetadata + CVEAlias row.
• Bundle suppression: when title carries multiple CVEs (rare — most Tencent advisories are 1:1), db.InsertAliases handles the CVE-prefix bundle suppression automatically.

6. CRIT / VEX

Single-envelope CRIT per advisory keyed to title-inferred service (mirrors zoom-fetch-processor §15.6 pattern):

VEX semantics:

• vex_status="fixed" when title/body says “已修复” or includes a patched-version note
• shared_responsibility: provider_only for managed services (CDB, COS, TKE control-plane, SCF); customer_action_required for CVM (customer reboots)
• fix_propagation: automatic for managed services; version_update for CVM

The §15.2 extended dict carries 5 service entries; cover-rate test asserts each title keyword routes to a dict-resolvable service.

7. S3 layout

Per AGENTS.md:

• Archive: tencent-cloud/files/{sha256}/{advisory-id}.html where advisory-id = "TC-{integer}"
• Quarantine: failed-feeds/tencent-cloud-fetch-processor/{date}/{reason}/{filename}

Reuse s3client.Uploader.Archive / Quarantine.

8. Error handling

Listing pages occasionally redirect to a localised landing page when served from non-CN IPs; if <div class="announce-list"> is empty, re-fetch with Accept-Language: zh-CN,en;q=0.5 once before quarantining. Per-detail parse failures are quarantined as parse-error. Detail pages that match 【安全通告】 but contain no CVE alias get minted a TC- ID and proceed (so we don’t lose Tencent-internal-only advisories).

9. Performance

CPU 256, memory 512 MB. Expected duration <15 minutes per scheduled run. Per-run cap: 100 new advisories (incremental beyond the last-seen integer ID). Backfill (full 216-page sweep) takes ~30 minutes at 200ms per page + ~2s per detail-page fetch.

10. Backfill

go-tencent-cloud-fetch-backfill TARGET="local" FORCE="false" \
    LIMIT="0" EMIT_CRIT="false"

Recipe pattern matches go-zoom-fetch-backfill (§15.6). unset EXPECTED_DURATION_MINUTES. Pass LIMIT=0 for full history; LIMIT=N for N-most-recent.

11. Implementation order

1. internal/tencent/{types,client,parse,parse_test}.go with testdata/listing-page-1.html + testdata/detail-2277.html captured fixtures
2. cmd/tencent-cloud-fetch-processor/{main,crit_mapper,crit_mapper_test}.go
3. Containerfile final stage, terraform module, task-manager.toml, dashboard targets.go, post-push-ecr.sh hook, justfile recipe
4. Live test against .env local DB
5. --emit-crit=true on schedule (spot-check freeze ended 2026-05-09 per §15.13; new producers ship at true and rely on post-deploy CloudWatch monitoring + per-source revert on >5% stage-failure)

S3 Persistence

Not used. This processor does not currently archive payloads or quarantine failures to S3. Per the S3 Persistence Contract this is non-compliant — see the compliance matrix for the implementation roadmap.

⚠ Not in the compliance matrix — status needs verification.

Expected paths when implemented:

• Archive: tencent-cloud/files/{sha256}/{filename}
• Quarantine: failed-feeds/tencent-cloud-fetch-processor/{YYYY-MM-DD}/{reason}/{filename}
• Likely reasons: (none documented)