hashicorp-discuss-fetch-processor — Design

1. Overview

  • Purpose: Ingest HashiCorp security advisories (HCSEC-YYYY-NN) from the HashiCorp Discuss forum security category.
  • Source URL: https://discuss.hashicorp.com/c/security/52
  • Owner: HashiCorp Security Team
  • Licence: Publicly readable; no auth required
  • Schedule cadence: Runs weekly on Fridays at 05:00 UTC (cron(0 5 ? * FRI *)).
  • ECS resources: 256 cpu_units, 512 memory_mb, expected_duration_minutes = 15
  • Reads: Discourse JSON API (/c/security/52.json, /t/{slug}/{id}.json?include_raw=1)
  • Writes: CVEMetadata, CVEDescription, CVEReference, CVEAlias, BulkDataDumpTracker, S3 archive

2. Source contract (from Phase 0 recon)

  • Cadence: ~1-2 new advisories per month; daily poll is conservative
  • Volume: ~194 total topics across 7 pages (30 per page); ~1-2 new per month
  • Identifier shape: HCSEC-YYYY-NN (source-native); CVE-YYYY-NNNN embedded in post text
  • Backfill feasibility: Yes — all historical topics accessible via pagination
  • Anti-bot / auth: None observed; public Discourse instance
  • Pagination: ?page=N on category JSON endpoint; more_topics_url signals next page
  • Freshness: No Last-Modified/ETag; use per-topic content hash for change detection

3. Architecture diagram

graph LR Source[discuss.hashicorp.com/c/security/52.json] --> Client[hashicorpdiscuss.Client] Client --> List[FetchTopicList pages] Client --> Detail[FetchTopicDetail per topic] Detail --> Parse[Extract HCSEC ID + CVEs + tags] Parse --> Map[MapToSourceData] Map --> Tx[(pgx Tx)] --> Pipeline[processor.StoreCVESourceData] Pipeline --> CVEMetadata Pipeline --> CVEAlias[(db.InsertAliases)] Pipeline --> CVEDescription & CVEReference Tx -.commit.-> S3[s3client.ArchiveRecord]

4. Source → DB field mapping

Source fieldTargetNotes
topic.titleCVEMetadata.titleFull title including HCSEC prefix
topic.created_atCVEMetadata.datePublishedISO 8601 → Unix seconds
post.rawCVEDescription.valueRaw markdown text
topic.slug + topic.idCVEMetadata.sourceAdvisoryRefPermalink
Extracted HCSEC-YYYY-NNCVEMetadata.cveIdPrimary identifier
Extracted CVE-YYYY-NNNNCVEAlias.aliasCveIdCross-cveId edges
PermalinkCVEAlias.aliasCveIdSelf-reference edge
Tag security-*CVEMetadata.affectedProductMapped to product name
"HashiCorp"CVEMetadata.affectedVendorStatic
SHA-256 of post.rawCVEMetadata.sourceFileHashChange detection

5. Identifier policy

  • cveId = HCSEC-YYYY-NN (source-native, kept as primary)
  • Aliases = all extracted CVE-YYYY-NNNN + topic permalink
  • Rationale: HashiCorp advisories frequently bundle multiple CVEs under one HCSEC (e.g., meta-bulletin HCSEC-2025-22 links to 8 sub-bulletins). Using HCSEC as primary avoids bundle-suppression complexity and keeps one row per advisory. CVEs are discoverable via the alias graph.
  • SuppressDerivedMetrics = true because descriptions cover the whole advisory, not a single CVE.

⚠ The topic permalink must stop being passed as an alias (audit 2026-08-06). map.go:44 prepends the https://discuss.hashicorp.com/t/{slug}/{id} URL to Aliases. Since db.InsertAliases gained non-CVE alias backfill (commit 08f7f28, 2026-05-25), an alias with no CVEMetadata row of its own gets a placeholder row minted under the writing source — so the next successful run would create ~193 CVEMetadata rows whose cveId is a URL. The permalink already lives in sourceAdvisoryRef and in a CVEMetadataReferences row; it is not an identifier and does not belong in the alias graph. Production is clean only because nothing has been stored since 2026-05-07 (see §9).

6. CRIT / VEX

No CRIT staging. HashiCorp advisories are for on-prem/enterprise software products (Vault, Consul, Terraform, Nomad, Boundary, Packer, Vagrant, Waypoint, Sentinel), not cloud-provider resource types. No (Provider, Service, ResourceType) triples to map.

7. S3 / source-file archive layout

  • Bucket: $S3_BUCKET_NAME
  • Key prefix: hashicorp-discuss/files/{sha256}/{hcsec-id}.json
  • Payload schema: Normalised JSON of osv.CVESourceData
  • Quarantine reasons:
    • store-error — transaction commit failed after retries
    • fetch-error — topic detail HTTP request failed

8. Error handling & Slack

  • Per-record vs batch: Per-record failures are logged and counted; batch continues
  • Retry profile: 3 attempts, 500 ms × attempt backoff, transient network errors retried
  • Slack events:
    • Started — at processor startup
    • Completed — with {"stored": N} stats
    • Errored — with {"stored": N, "errors": M} and error detail
    • NoWork — when tracker freshness check skips

9. Performance

  • Concurrency: 4 fetch workers (configurable via --workers)
  • Per-request timeout: 30 seconds
  • Rate-limit pacing: 250 ms gap between dispatches
  • Soft deadline: EXPECTED_DURATION_MINUTES - 10 minutes; the fan-out goroutine stops dispatching a further 5 minutes before that (main.go:161)
  • Resume strategy: per-topic SHA-256 of post.raw compared against the stored CVEMetadata.sourceFileHash; unchanged topics are skipped

⚠ Known defects (audit 2026-08-06) — the two bullets above do not hold in ECS:

  1. The processor stores nothing on a scheduled run. softDeadline is now + (EXPECTED_DURATION_MINUTES − 10) (main.go:65) and the fan-out loop breaks at softDeadline − 5m (main.go:161). With the configured expected_duration_minutes = 15 that budget is zero, so the dispatcher exits before emitting a single topic and the run reports Completed{stored:0} — which also bumps the tracker, so nothing looks wrong. Production evidence: the newest lastFetchedAt for source='hashicorp-discuss' is 2026-05-07, the day the processor landed, with nothing written since. Local backfills are unaffected because they unset EXPECTED_DURATION_MINUTES.
  2. The resume set can never hit. db.LoadProcessedHashes returns a map keyed by cveId (internal/db/resume.go:58) but the lookup key is HashString(topic.Slug) (main.go:193-194), so once (1) is fixed every topic is re-fetched and re-stored on every run. The key must be the HCSEC-YYYY-NN id.

Both are code bugs, not design changes — the intended behaviour is what the two bullets above describe.

10. Backfill

  • Backfill warranted? Yes — all historical topics are accessible
  • Justfile recipe: go-hashicorp-discuss-fetch-backfill
  • Expected runtime: ~5 minutes for full ~194 topics
  • Differs from scheduled binary: Unsets EXPECTED_DURATION_MINUTES so backfill runs to completion

S3 Persistence

  • Archive path: hashicorp-discuss/files/{sha256}/{filename} ✓
  • Quarantine path: failed-feeds/hashicorp-discuss-fetch-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
  • Failure reasons emitted: store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[hashicorp-discuss-fetch-processor] PROC -->|success| ARCHIVE[("S3: hashicorp-discuss/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/hashicorp-discuss-fetch-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.