tencent-blade-fetch-processor — Design

Status: Implemented — Wayback Machine ingestion path
Source: Wayback Machine snapshots of blade.tencent.com/en/advisories/<slug>/ (the live Tencent Blade portal is geo-restricted to mainland China and unreachable from ap-southeast-2) Type: fetch (HTTP fetch against web.archive.org)
Processor Name: tencent-blade-fetch-processor
AWS Schedule Name: go-tencent-blade-fetch-processor
Source Namespace: tencent-blade

1. Overview

• Purpose: Ingest Tencent Blade Team advisories via the Wayback Machine. The original portal at blade.tencent.com is geo-blocked (verified live: connections from ap-southeast-2 to the main domain hang). The Wayback Machine has indexed the English advisory pages and is publicly reachable from any region.
• Owner: Tencent Blade Team (elite security research team within Tencent)
• Licence: Publicly disclosed vulnerabilities; archival use covered by Wayback’s terms.
• Schedule cadence: Runs weekly on Sundays at 04:00 UTC (cron(0 4 ? * SUN *)).
• Rationale: Low volume (~5 historical advisories indexed; ~1–3 new per month upper bound based on the team’s publication rate); daily check picks up new Wayback-indexed snapshots within a day of crawl.
• ECS resources: cpu_units=256, memory_mb=512, expected_duration_minutes=15
• Reads: Wayback CDX API + archived HTML; BulkDataDumpTracker (freshness); LoadProcessedHashes (resume)
• Writes: CVEMetadata, CVEDescription, CVEMetadataReferences, CVEAlias (same-cveId cross-source edges only), BulkDataDumpTracker; S3 archive + quarantine of the archived HTML. Plus the shared pipeline’s derived containerType="vulnetix" cvssV4_0 CVEMetric row.
• Never written in practice: an upstream CVEMetric row (the parser populates no CVSS vector — see §5.4), CVEMetadata.affectedVendor / affectedProduct (parsed fields exist on the struct but nothing fills them), CVEAffected / CVEAffectedVersion, and any internal/aienrich output (no enricher is constructed).
• Soft deadline: softDeadlineDuration defaults to 15 minutes unconditionally (main.go:80-84), so unsetting EXPECTED_DURATION_MINUTES in the backfill recipe does not remove the cap.

2. Source Contract — Wayback architecture

Sample CDX row (live, May 2026):

["com,tencent,blade)/en/advisories/loradawn",
 "20220314035841",
 "https://blade.tencent.com/en/advisories/loradawn/",
 "text/html", "200", "ZRJA7NG6TUVLN5XXJDX5E7VD2KYUBDGS", "4137"]

→ DetailURL becomes https://web.archive.org/web/20220314035841/https://blade.tencent.com/en/advisories/loradawn/

3. Architecture

4. Data Flow

5. Source → DB Field Mapping

5.1 CVEMetadata

5.2 CVEDescription

5.3 CVEMetadataReferences

Only two kinds of URL are emitted — the parser does not scan the archived page for arbitrary links. ParseDetailPage builds the list from (a) the canonical un-archived blade.tencent.com advisory URL and (b) one cve.mitre.org/cgi-bin/cvename.cgi?name=<CVE> entry per sibling CVE found in the body. MapAdvisory then classifies each by URL shape:

5.4 CVEMetric

MapAdvisory emits a cna metric row when Advisory.CVSSVector is non-empty:

In practice this branch is dead: ParseDetailPage never assigns CVSSVector / CVSSScore / CVSSVersion / Severity, so no upstream metric row has ever been written for source='tencent-blade'. The only metric these records carry is the shared pipeline’s description-derived containerType="vulnetix" cvssV4_0 row. Archived Blade advisories do quote a CVSS vector in prose, so this is an extraction gap rather than an upstream limitation.

5.5 CVEAlias

Always call db.InsertAliases(ctx, tx, cveID, source, nil, logger) — even with nil aliases, per AGENTS.md alias contract. This ensures same-cveId cross-source edges are backfilled. No bundle suppression needed (1 CVE per advisory).

6. Identifier Policy

7. CRIT / VEX

No CRIT staging needed. Tencent Blade Team is a security research team, not a cloud provider. Their advisories do not attribute vulnerabilities to (Provider, Service, ResourceType) triples. No extended dictionaries needed.

8. S3 / Source-File Archive Layout

• Bucket: S3_BUCKET_NAME (from env)
• Archive key prefix: tencent-blade/files/{sha256}/{filename}
• Quarantine key prefix: failed-feeds/tencent-blade-fetch-processor/{YYYY-MM-DD}/{reason}/{filename}
• Payload schema: Raw detail page HTML (preserved for round-trip debugging)
• Quarantine reasons:
  • parse-error — detail page HTML could not be parsed
  • store-error — processor.StoreCVESourceData returned an error

9. Error Handling & Slack

• Per-record failures: notifier.RecordError(msg); batch continues
• Fatal failures (list page fetch, DB connection, tracker update): notifier.Errored(processor, stats, err) + os.Exit(1)
• NoWork: notifier.NoWork(processor, "data is fresh") when tracker is within frequency
• Retry profile: 3 attempts with exponential backoff (2s, 4s) for list page; per-detail-page fetch has 1 attempt (low volume, re-run next day is fine)
• Slack stats dict: {"fetched": N, "stored": N, "skipped": N, "failed": N}

10. Performance

• Concurrency: Sequential (low volume, no need for workers)
• Rate limit: 1 req/sec between detail page requests
• Per-request timeout: 15 seconds
• Soft deadline: EXPECTED_DURATION_MINUTES - 10 minutes (default 5 min if env unset)
• Resume strategy: db.LoadProcessedHashes (hash-based skip set)
• Pool sizing: db.NewPool defaults (read=5, write=5)

11. Backfill

Not warranted. The source is small enough (~50–150 total advisories) that --force --limit=0 on the same binary covers full historical reprocessing in a single run. No separate cmd/tencent-blade-fetch-backfill/ needed.

The justfile recipe go-tencent-blade-fetch-backfill runs the same binary with EXPECTED_DURATION_MINUTES unset so backfill runs to completion.

12. Implementation Tasks

Phase 1: Reconnaissance & Discovery (BLOCKED)

Phase 2: Go Processor Implementation (DONE — stubs in place)

Phase 3: Infrastructure & Configuration (DONE)

Phase 4: Testing & Validation (after recon)

13. File Checklist

New Files

• scripts/go-processors/cmd/tencent-blade-fetch-processor/main.go
• scripts/go-processors/internal/tencentblade/types.go
• scripts/go-processors/internal/tencentblade/parser.go
• scripts/go-processors/internal/tencentblade/mapper.go
• scripts/go-processors/tencent-blade-fetch-processor.design.md (this file)

Modified Files

• Containerfile.go-processors — add tencent-blade-fetch-processor target
• scripts/task-manager.toml — add [tasks.tencent-blade-fetch-processor]
• terraform/go-schedules.tf — add local + module
• justfile — add go-tencent-blade-fetch-backfill recipe
• .claude/hooks/post-push-ecr.sh — add to TARGETS
• scripts/task-dashboard/cmd/ecr-build/targets.go — add to simpleTargets

14. Naming Convention Compliance

Per AGENTS.md Processor Naming Convention:

15. Risk Register

16. Appendix: Reference Processors

Similar fetch processors to study during implementation:

17. Open Questions

1. What is the exact detail page URL pattern?
   Hypothesis: https://blade.tencent.com/en/advisories/detail/{cve-id}/ or similar. Needs confirmation.

2. Does the list page show all advisories or is it paginated?
   If paginated, what is the pagination mechanism (?page=N, ?offset=N, cursor-based)?

3. Is there embedded structured data (JSON-LD, application/ld+json)?
   Many modern advisory sites include Schema.org Vulnerability markup.

4. Does the site publish CVSS v4.0 vectors or only v3.1?
   Affects CVEMetric row construction.

5. Are there non-CVE advisories (e.g., Blade-specific IDs)?
   If yes, primary cveId may need to be a Blade ID with CVE as alias.

6. Is there a Chinese version with more complete data?
   URL: https://blade.tencent.com/zh/advisories/cve-list/ — may have more advisories or richer descriptions.

S3 Persistence

• Archive path: tencent-blade/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/tencent-blade-fetch-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: parse-error, store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.