cert-pt-rss-processor Design

Overview

Fetches security advisories from the Portuguese national cybersecurity centre (CNCS — Centro Nacional de Cibersegurança / CERT-PT) RSS feed and creates CVEMetadata records for each CVE ID found.

• Source: cert-pt
• Feed: https://www.cncs.gov.pt/docs/alertas-seguranca/feed-rss/index.xml
• Language: Portuguese (pt)
• Schedule: Daily at 06:00 UTC (cron(0 6 * * ? *))
• CPU / Memory: 256 / 512 MiB

Data Source

CNCS publishes security alerts at https://www.cncs.gov.pt/docs/alertas-seguranca/ as an RSS 2.0 feed. Each <item> contains:

The RSS feed is an index. The full advisory content (complete Descrição, Resolução, Referências with all links, affected systems) lives on the article page at <link>. The processor fetches each article page for authoritative data and falls back to the RSS <description> CDATA if the fetch fails.

Article Page Structure

Each article page (dyn.cncs.gov.pt/pt/alerta-detalhe/art/…) contains:

<article class="post post-large blog-single-post …">
  <div class="post-date">…</div>           <!-- date: not parsed -->
  <h2 class="alert-title">…</h2>          <!-- title: not parsed (use RSS title) -->
  <div class="alert-text container">       <!-- metadata: TIPO, SISTEMAS AFETADOS, ECOSSISTEMA -->
    …
  </div>
  <div>
    <div class="alert-detail-title">Descrição</div><p>…</p>
    <div class="alert-detail-title">Impacto</div><p>…</p>
    <div class="alert-detail-title">Resolução</div><p>…</p>
    <div class="alert-detail-title">Referências</div><p>…</p>
  </div>
</article>

WAF / Cookie Challenge

dyn.cncs.gov.pt is protected by an OpenResty WAF that issues a __diamwall cookie challenge:

1. First request (no cookie) → HTTP 307 back to same URL + Set-Cookie: __diamwall=…
2. Second request (cookie replayed by jar) → HTTP 200 with content

The processor uses a persistent cookiejar shared across all article fetches in a run. This means the challenge is solved at most once per process, and subsequent fetches go straight through. Up to 15 redirects are followed per request; failures (including persistent 517 Bot Management blocks) fall back to RSS description silently at DEBUG level.

Processing Logic

RSS feed (XML)
  └─ ParseFeed → []Item
       └─ ParseItem → *Advisory   (initial parse from RSS <description>)
            └─ fetchPage(item.Link)  ← article page fetch (cookie-jar client)
                 ├─ success → ParseArticleHTML(adv, pageHTML)  [overwrites RSS data]
                 └─ failure → keep RSS-parsed data (silent DEBUG log)
                      ParseArticleHTML:
                        ├─ extract article.post block
                        ├─ splitBySections (div.alert-detail-title or h2–h4)
                        │    ├─ Descrição   → Advisory.Description
                        │    ├─ Impacto     → Advisory.ImpactText
                        │    ├─ Resolução   → Advisory.Resolution
                        │    └─ Referências → Advisory.ReferenceURLs
                        ├─ extractCVEIDs  (regex CVE-\d{4}-\d{4,})
                        └─ extractCVSS    (vector or score-only)
       └─ MapAdvisory → []*CVESourceData (one per CVE ID)
            └─ StoreCVESourceData (per-CVE transaction)

Advisory Filtering

• Advisories without CVE IDs are skipped (skippedNoCVE counter).
• One CVEMetadata record is created per CVE ID found.
• CVE IDs from the same advisory are stored as cross-aliases in CVEAlias.

Resume / Dedup

• Key: advisory <link> URL, stored as sourceAdvisoryRef.
• loadProcessedAdvisories() queries CVEMetadata WHERE source='cert-pt' AND sourceAdvisoryRef IS NOT NULL.
• Reprocess with --all or --force flags.

HTML Parsing

The RSS description uses two formats depending on advisory age:

ParseArticleHTML handles both:

1. Unescape HTML entities up to twice (RSS CDATA is often double-encoded).
2. Try <article class="...post..."> block (full-page fetch — unused currently).
3. Apply splitBySections: prefers div.alert-detail-title pattern on tie or greater count; falls back to h2–h4 headings.
4. Map heading text to fields via normaliseHeading (strips Portuguese diacritics).

Section Normalisation Map

CVSS Extraction (priority order)

1. Full vector: CVSS:(4\.0|3\.1|3\.0)/[A-Z0-9:/.]+ → VectorString + type detection
2. Bare v2 vector: AV:[NALP]/AC:[LMH]/Au:[MSN]/C:[NPC]/I:[NPC]/A:[NPC] → cvssV2_0
3. Score-only: (?i)(?:CVSS\s+)?(?:Base\s+)?[Ss]core[:\s]+([\d.]+) → BaseScore + BaseSeverity

Database Records

CVEMetadata

CVEDescription

CVEAlias

Cross-links CVE IDs that appear in the same advisory — but only when the advisory lists exactly two CVEs. db.InsertAliases treats a CVE-prefixed primary id carrying two or more CVE-prefixed aliases as bundling and drops the CVE entries (internal/db/cvealias.go:65), so a three-or-more-CVE alert writes no CVE↔CVE edges. Every row still gets same-cveId cross-source edges linking cert-pt to every other source holding that CVE.

CVEMetadataReferences

• Advisory URL (type advisory)
• Extracted reference URLs from Referências section (classified by domain)

CVEMetric

Populated when CVSS data is found:

• Full vector: metricType detected from the prefix, vectorString set
• Score-only: metricType = cvssV3_1, baseScore + baseSeverity set (cvssV2_0 is only ever produced from a bare v2 vector, never score-only)

The vector is written to CVEMetric.vectorString only. The denormalised CVEMetadata.vectorString column is never set (the mapper does not populate CVESourceData.VectorString), so it is NULL for every cert-pt row even when a vector was extracted — 231 vector-bearing CVEMetric rows exist against 296 metadata rows with an empty vectorString.

Error Handling & Alerting

Uses the shared notify package (SNS → Slack in ECS; no-op locally):

Infrastructure

Local Development

# Run against prod (positional args: TARGET ALL LIMIT FORCE)
just go-cert-pt-rss-backfill prod false 10 true

# Build and run in ARM64 container
just cert-pt-rss-processor prod

# Verify results
psql "$DATABASE_URL" -c "
  SELECT \"cveId\", title, \"sourceAdvisoryRef\"
  FROM \"CVEMetadata\" WHERE source='cert-pt' LIMIT 10;"

Limitations

• CVE-less advisories (vendor bulletins, general alerts) are skipped entirely.
• CVSS severity is not embedded in CVSS vectors (no score lookup table); severity is only set when a score is explicitly stated in the advisory text.
• TIPO / SISTEMAS AFETADOS / ECOSSISTEMA sidebar from article pages is not captured — only Descrição, Impacto, Resolução, Referências sections are mapped.
• Portuguese-only descriptions — no English translation or cross-language enrichment.
• If Cloudflare Bot Management escalates beyond cookie challenge (HTTP 517), the processor degrades gracefully to RSS description data for that item.

S3 Persistence

• Archive path: cert-pt/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/cert-pt-rss-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: parse-error, enrich-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.