Bugcrowd Crowdstream Processor — Design Document

1. Overview

Purpose: Fetch Bugcrowd crowdstream submissions and disclosed vulnerability reports via the unauthenticated JSON API, and store them across CVEMetadata (+ relations), Exploit, and BugBountySubmission tables.

Data source: Bugcrowd Crowdstream JSON API — GET https://bugcrowd.com/crowdstream.json

API notes:

• Unauthenticated — no API key or session required
• Three tiers of data:
  1. Global feed (/crowdstream.json?page=N) — 7-day rolling window, ~925 entries, 20/page
  2. Per-program feed (/engagements/{code}/crowdstream.json?page=N) — configurable window (7 days to 6+ months), includes disclosed entries
  3. Disclosure reports (/disclosures/{uuid}/{slug}) — full HTML writeups with CVE IDs, CWEs, technical descriptions
• Offset pagination (page number, not cursor)
• No documented rate limits, but browser UA headers required to avoid bot detection

Schedule: Daily at 06:30 UTC (cron cron(30 6 * * ? *) — Terraform EventBridge)

Timeout: 60 minutes

Resources: 256 CPU units, 512 MB memory

What it reads:

• Bugcrowd Crowdstream JSON API (public, no auth)
• Bugcrowd disclosure report HTML pages (public, no auth)

What it writes:

• BugBountySubmission (source=bugcrowd — every submission)
• Exploit (source=bugcrowd — every submission, category=bug-bounty)
• ExploitCVE (junction to the BUGCROWD-* record, plus one per real CVE found on a disclosure page)
• ExploitAffectedProduct (vendor/product from program info)
• CVEMetadata (source=bugcrowd) — one row per submission, keyed BUGCROWD-{YYYY}-{submission uuid}, plus a minimal placeholder row per real CVE id found on a disclosure page (db.EnsureMinimalCVEMetadata)
• CVEDescription, CVEMetadataReferences, CVEMetric, CVEProblemType (CVE relation tables, all keyed on the BUGCROWD-* id)
• CVEAlias (db.InsertAliases — links the BUGCROWD-* id to every CVE id parsed off the disclosure page)
• BulkDataDumpTracker (source=bugcrowd_crowdstream — freshness)

Identifier scheme

Bugcrowd submissions have no public advisory identifier, so the processor mints one: BUGCROWD-{year}-{submission uuid}, where the year comes from created_at (main.go:702-708). This id is the CVEMetadata.cveId for every crowdstream entry — disclosed or not, CVE-bearing or not — which is why the row count tracks total submissions (~8.6k) rather than the disclosed subset. Real CVE ids from a disclosure page become CVEAlias edges on that record, never the primary id. No GcveIssuance row is minted for these ids.

Environment variables:

• DATABASE_URL — required
• DATABASE_URL_READ — optional

2. API Details

Global Crowdstream

Endpoint: GET https://bugcrowd.com/crowdstream.json?page={N}

Response structure:

{
  "results": [ ... ],
  "pagination_meta": {
    "total_pages": 47,
    "totalCount": 925,
    "limit": 20,
    "currentPage": 1
  },
  "cutoff_date_label": "7 days"
}

Per-Program Crowdstream

Endpoint: GET https://bugcrowd.com/engagements/{code}/crowdstream.json?page={N}

Same response structure. Programs configure their own retention window (Atlassian: 6 months, Tesla: 6 months, most others: 7 days). Returns richer data including disclosed entries.

Submission Entry (Non-Disclosed)

{
  "id": "f8a831c3-601f-45ab-a926-c3eba637784e",
  "engagement_name": "OpenAI",
  "engagement_code": "openai",
  "engagement_path": "/engagements/openai",
  "engagement_in_progress": true,
  "visibility_public": false,
  "target": "Sora",
  "priority": 3,
  "substate": "unresolved",
  "amount": "$500",
  "points": 10,
  "crowdstream_amount_visible": true,
  "disclosed": null,
  "created_at": "2026-03-16T22:24:00.200Z",
  "accepted_at": "9 Apr 2026",
  "claimed_at": null,
  "closed_at": null,
  "is_pinned": false,
  "submission_state_text": "Submission accepted on target: Sora",
  "submission_state_date_text": "Accepted on 9 Apr 2026",
  "researcher_username": null,
  "logo_url": "https://logos.bugcrowdusercontent.com/...",
  "logo_color": "fff"
}

Submission Entry (Disclosed)

Disclosed entries have additional fields:

{
  "id": "2c29d932-f428-4d5c-ac74-9d195398944e",
  "engagement_name": "Atlassian",
  "engagement_code": "atlassian",
  "target": "Bitbucket Data Center",
  "priority": 1,
  "substate": "resolved",
  "disclosed": true,
  "title": "RCE in Bitbucket DataCenter via HazelCastPort",
  "disclosure_report_url": "/disclosures/ce3d9a93-2168-4785-90b7-47f66a5b8162/rce-in-bitbucket-datacenter-via-hazelcastport",
  "disclosed_at": "12 Apr 2022",
  "researcher_username": "SnowyOwl",
  "created_at": "2022-01-26T22:12:19.045Z",
  "accepted_at": "17 Feb 2022",
  "closed_at": "2022-03-18T19:56:36.076Z"
}

Disclosure Report Pages

HTML pages at https://bugcrowd.com/disclosures/{uuid}/{slug} contain:

• Vulnerability title and description
• CVE IDs (e.g., CVE-2022-26133, CVE-2021-26084)
• CWE classifications (e.g., CWE-917, CWE-116)
• Technical details, attack vectors, root cause analysis
• External references (vendor advisories, GitHub issues, Snyk entries)
• Researcher info and timeline

Example: The disclosure for “RCE on Confluence Data Center via OGNL Injection” contains CVE-2021-26084, CWE-917, CWE-116, and references to the Atlassian security advisory.

Priority Mapping

3. Business Logic

Two-Phase Processing Strategy

Phase 1 — Discover programs and collect submissions:

1. Fetch global crowdstream (/crowdstream.json), paginating all pages
2. Extract unique engagement_code values (active programs)
3. For each program, fetch per-program crowdstream (all pages) — this captures disclosed entries not in the 7-day global window
4. Batch upsert all submissions into BugBountySubmission + Exploit (500 per transaction with SAVEPOINTs)
5. Collect entries where disclosed: true and disclosure_report_url is present

Phase 2 — Enrich disclosed submissions: 6. Skip disclosures already stored as Exploit (check LoadProcessedExploitIDs) 7. For each new disclosure, fetch the HTML report page 8. Parse CVE IDs, CWE IDs, description, and reference URLs via regex 9. Update BugBountySubmission with CVE/CWE data 10. For each CVE ID: EnsureMinimalCVEMetadata, UpsertCVEMetadata, InsertReferences, InsertProblemTypes, InsertDescriptions, InsertExploitCVE

Per-Submission Processing (Phase 1)

Every submission (disclosed or not) — processSubmission, main.go:373-515:

1. UpsertBugBountySubmission — full metadata
2. UpsertExploit — category=bug-bounty
3. DeleteExploitAffectedProducts + InsertExploitAffectedProduct — from program info
4. UpsertCVEMetadata for the minted BUGCROWD-{year}-{uuid} id
5. InsertReferences — the disclosure report URL, when present
6. InsertDescriptions — the submission title
7. InsertMetrics — a cvssV3_1 row carrying only baseSeverity, mapped from the Bugcrowd priority (no vector, no score)
8. InsertExploitCVE — junction between the Exploit and the BUGCROWD-* record

Per-Disclosure Processing (Phase 2)

Disclosed submissions whose report page yielded a CVE id or a description — enrichDisclosure, main.go:520-698. All writes still key on the BUGCROWD-* id: 9. UpsertBugBountySubmission / UpsertExploit again, now with cveIds + weaknessCweId 10. UpsertCVEMetadata again, now with the aliases JSON column populated 11. InsertReferences — report URL (advisory) + extracted external references (technical) 12. InsertProblemTypes — CWEs from the disclosure page 13. InsertDescriptions — description from the disclosure page (or title as fallback) 14. db.InsertAliases — BUGCROWD-* ⇄ every CVE id on the page 15. EnsureMinimalCVEMetadata + InsertExploitCVE per real CVE id

Bounty Amount Cleaning

Bugcrowd returns amounts as formatted strings (e.g., "$10,000"). The processor strips $ and , before storing as bountyAmount (string, preserving decimal precision).

Timestamp Handling

• created_at, closed_at — RFC3339 format, parsed to Unix ms
• disclosed_at — human-readable format ("19 Jul 2023"), parsed via "2 Jan 2006" layout
• accepted_at — display-only format ("9 Apr 2026"), not stored

Rate Limiting

• 500ms between per-program crowdstream JSON fetches
• 1s + 0-200ms random jitter between disclosure HTML page fetches
• Soft deadline: EXPECTED_DURATION_MINUTES - 10 minutes — but the fallback is a hardcoded 50 minutes, applied even when the variable is unset (main.go:61-65), so a local backfill is also truncated at 50 minutes

Resume / Freshness

• BulkDataDumpTracker (source=bugcrowd_crowdstream) — skip if last run was less than frequency seconds ago
• Disclosed reports: loadEnrichedExploitIDs skips a disclosure only when its Exploit.cveIds column is already populated (main.go:760-762). A disclosure page that yielded no CVE id is therefore re-fetched on every run.
• On incremental runs: only programs appearing in the global 7-day feed are re-fetched

Idempotency

BugBountySubmission and Exploit are keyed (reportId, source) and (exploitId, source) and use ON CONFLICT DO UPDATE, so those are idempotent. CVEMetadata is keyed (cveId, source) and likewise upserts. CVEDescription and CVEMetric replace their rows.

CVEMetadataReferences and CVEProblemType are not idempotent: their shared helpers issue a bare ON CONFLICT DO NOTHING while the tables carry no unique index beyond the uuid primary key, so every re-run appends a fresh duplicate row. Production currently holds ~42k reference rows for ~830 distinct (cveId, url) pairs under this source.

4. Architecture Diagram

5. Deployment Diagram

6. Processing Flow

7. Data Mapping

8. Field Mapping Reference

9. Infrastructure

10. Verification

# Local test
just go-bugcrowd-json-backfill

# Check BugBountySubmission
psql "$DATABASE_URL" -c 'SELECT COUNT(*) FROM "BugBountySubmission" WHERE source = '\''bugcrowd'\'''

# Check CVE population (disclosed reports only)
psql "$DATABASE_URL" -c 'SELECT COUNT(*) FROM "CVEMetadata" WHERE source = '\''bugcrowd'\'''

# Check Exploit population
psql "$DATABASE_URL" -c 'SELECT COUNT(*) FROM "Exploit" WHERE source = '\''bugcrowd'\'''

# Check junctions
psql "$DATABASE_URL" -c 'SELECT COUNT(*) FROM "ExploitCVE" WHERE source = '\''bugcrowd'\'''

# Check disclosed with CVEs
psql "$DATABASE_URL" -c 'SELECT "reportId", "title", "cveIds", "weaknessCweId" FROM "BugBountySubmission" WHERE source = '\''bugcrowd'\'' AND "cveIds" IS NOT NULL LIMIT 10'

# Idempotency — run again, counts should not increase
just go-bugcrowd-json-backfill

# Docker build
podman build --target bugcrowd-json-processor -f Containerfile.go-processors scripts/go-processors/

S3 filenames

The unit of work is a crowdstream submission, not a file, so the payload is the submission re-serialised to canonical JSON and {filename} is {submission uuid}.json (main.go:313-323). Archive fires only for submissions whose savepoint committed; quarantine fires for per-submission failures and for every submission in a batch whose transaction rolled back (main.go:358-367). The scraped disclosure-page HTML is not archived.

S3 Persistence

• Archive path: bugcrowd/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/bugcrowd-json-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.