cxsecurity-rss-processor

Status: Live Source: CXSecurity World Laboratory Bug (WLB) Type: rss (three RSS 2.0 feeds + per-item ASCII page fetch) Source slug: cxsecurity Schedule: Daily at 09:40 UTC (cron(40 9 * * ? *)), expected_duration_minutes = 60, 256 CPU / 512 MB.

Overview

Fetches CXSecurity exploit advisories, Google dork entries, and general vulnerability advisories from three public RSS 2.0 feeds. For each item, downloads the ASCII page (/ascii/WLB-*), parses ExploitDB-style headers, uploads the raw body to S3, and stores the data in either the Exploit table (exploits/dorks) or CVEMetadata table (advisory-only items).

Feeds

Feed	URL	Target Table
Exploits	`https://cxsecurity.com/wlb/rss/exploit/`	Exploit
Dorks	`https://cxsecurity.com/wlb/rss/dorks/`	Exploit (category=“dork”)
All	`https://cxsecurity.com/wlb/rss/all/`	CVEMetadata (non-exploit items only)

All feeds are public, no authentication required. Items use WLB IDs (WLB-YYYYMMNNN) as unique identifiers across all feeds.

RSS Item Structure

Field	Source
WLB ID	Last path segment of `<link>` (e.g. `WLB-2026030004`)
Issue URL	`<link>` (e.g. `https://cxsecurity.com/issue/WLB-2026030004`)
ASCII URL	URL rewrite: `/issue/` → `/ascii/`
Title	`<title>`
Author	`<dc:creator>` (Dublin Core namespace)
Published	`<pubDate>` parsed as RFC 1123Z → Unix seconds
Description	`<description>` — truncated text in RSS, full content in ASCII page

ASCII Page Format

The ASCII page is an HTML-wrapped <PRE> block containing ExploitDB-style content:

<HTML><HEAD>...</HEAD><BODY><PRE>
#!/usr/bin/env python3
# Exploit Title: ...
# CVE: CVE-YYYY-NNNNN
# Date: YYYY-MM-DD
# Exploit Author: ...
# Vendor Homepage: https://...
# Software Link: https://...
# Version: ...
# Tested on: ...
# Platform: ...
# Category: ...
# CVSS: 9.9 (CRITICAL) CVSS:3.1/AV:N/...
# CWE: CWE-184 (...)

...exploit body / description / PoC code...
</PRE></BODY></HTML>

Processing Logic

Fetch all three RSS feeds
Merge by WLB ID with priority: exploit > dork > advisory
For each item:
- Download ASCII page, strip HTML wrapper
- Parse # Key: Value headers
- Extract CVE IDs from headers + body text
- Upload raw body to S3 (cxsecurity/{WLB-ID}.txt)
- Create Artifact + Link records
Route by feed type:
- exploit/dork → Exploit + ExploitCVE + ExploitAffectedProduct
- advisory (with CVE IDs, not in exploit/dork feeds) → CVEMetadata + CVEDescription + CVEMetadataReferences + CVEAffected + CVEAlias

Storage

No new tables or columns needed. All tables already exist.

Exploit Path (exploit/dork feeds)

Table	Rows
`Artifact`	One per item; R2 metadata for ASCII body
`Link`	One per item; issue URL → Artifact FK
`Exploit`	One per WLB ID; `source="cxsecurity"`, `exploitId=WLB-*`
`ExploitCVE`	One per CVE per known source; junction to CVEMetadata
`ExploitAffectedProduct`	Zero or one per exploit; from parsed headers

Advisory Path (all feed, non-exploit items)

Table	Rows
`Artifact`	One per item; R2 metadata for ASCII body
`Link`	One per item; issue URL → Artifact FK
`CVEMetadata`	One per CVE; `source="cxsecurity"`
`CVEDescription`	One per CVE; `containerType="cna"`, `lang="en"`
`CVEMetadataReferences`	One per CVE; `type="advisory"`, `referenceSource="cxsecurity"`
`CVEAffected`	When vendor/product extractable from headers (plus dependency/registry enrichment via `db.EnrichAffectedWithDependency`)
`CVEAlias`	Multi-CVE advisories: first CVE → siblings, written through `db.InsertAliases`

Not persisted

The ASCII header parser extracts a CVSS vector and a CWE id (internal/cxsecurity/mapper.go:110-127 builds Metrics and ProblemTypes), but the advisory writer processOneCVE (cmd/cxsecurity-rss-processor/main.go:528-600) only stores metadata, descriptions, references and affected rows. No CVEMetric or CVEProblemType row is written, and CVEMetadata.vectorString / affectedVendor / affectedProduct stay null. Production confirms this: 0 CVEMetric and 0 CVEProblemType rows for source='cxsecurity'.

Incremental Strategy

Exploit path: Load all exploitId values from Exploit where source='cxsecurity'
Advisory path: Load all sourceAdvisoryRef values from CVEMetadata where source='cxsecurity'
Skip items present in either set (unless --all)

Flags

Flag	Default	Description
`--all`	false	Reprocess all items, not just new ones
`--limit`	0	Maximum items to process per run (0 = unlimited)

ECS Schedule

Runs daily at 09:40 UTC (cron(40 9 * * ? *)).

Failure modes

Symptom	Cause
`feed fetch failed` warn, run continues	One of the three feeds 5xx’d after 3 retries (`main.go:275`); the other feeds still process.
`ASCII fetch failed` warn + record error	Per-item ASCII page fetch failed after 3 retries (`main.go:179`).
`could not load resume hashes, processing all`	The `Exploit`/`CVEMetadata` resume queries exceeded their 30s budget (`main.go:87-102`); the run re-processes every feed item instead of only new ones.
`approaching soft deadline, stopping`	The soft deadline is a hardcoded margin, not `internal/rundeadline.Soft`: `main.go:71` computes `EXPECTED_DURATION_MINUTES − 10` and `main.go:158` stops a further 5 minutes early. When `EXPECTED_DURATION_MINUTES` is unset (local `go-cxsecurity-rss-backfill`) it still applies a 30-minute default, so a local backfill can truncate — see the “Backfill Must Not Have a Deadline” rule in `scripts/go-processors/AGENTS.md`.

Key Files

File	Purpose
`cmd/cxsecurity-rss-processor/main.go`	Main processor
`cmd/cxsecurity-rss-processor/s3.go`	S3 upload helper
`internal/cxsecurity/types.go`	RSS feed and advisory Go structs
`internal/cxsecurity/parser.go`	Feed parsing, ASCII parsing, CVE extraction
`internal/cxsecurity/mapper.go`	Advisory → ExploitRow / CVESourceData mapping
`schemas/cxsecurity_rss_advisory.schema.json`	JSON Schema Draft 7

S3 Persistence

Archive path: cxsecurity/files/{sha256}/{filename} ✓ (ArchiveRecord, main.go:216 / main.go:236)
Quarantine path: failed-feeds/cxsecurity-rss-processor/{YYYY-MM-DD}/{reason}/{filename} ✓ (QuarantineRecord, main.go:212 / main.go:232)
Failure reasons emitted: store-error, parse-error
Additional raw-body path (outside the contract): the ASCII page body is also uploaded verbatim to cxsecurity/{WLB-ID}.txt (main.go:193) and referenced by the Artifact row.

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[cxsecurity-rss-processor] PROC -->|success| ARCHIVE[("S3: cxsecurity/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/cxsecurity-rss-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.