HackerOne Hacktivity Processor — Design Document

1. Overview

Purpose: Fetch publicly disclosed bug bounty reports from HackerOne’s unauthenticated GraphQL API and store them across CVEMetadata (+ relations), Exploit, and BugBountySubmission tables.

Data source: HackerOne GraphQL API — POST https://hackerone.com/graphql

API notes:

• Unauthenticated — returns only publicly disclosed reports
• Relay-style cursor pagination (first/after/pageInfo)
• Supports incremental filtering: where: {disclosed_at: {_gt: "ISO8601"}}
• No order_by support — default order is most-recently-disclosed first
• Introspection disabled — schema discovered empirically (April 2026)
• No documented rate limits, but aggressive bot detection — browser UA headers required

Schedule: Daily at 06:15 UTC (cron cron(15 6 * * ? *) — Terraform EventBridge)

Timeout: 60 minutes

Resources: 256 CPU units, 512 MB memory

What it reads:

• HackerOne GraphQL API (public, no auth)

What it writes:

• BugBountySubmission (source=hackerone — every report)
• Exploit (source=hackerone — every report, category=bug-bounty)
• ExploitCVE (linking reports to CVE records)
• ExploitAffectedProduct (vendor/product from team info)
• CVEMetadata (source=hackerone) — every report gets a primary record keyed H1-<reportId> (main.go:478), not only reports that carry a CVE ID. Reports that do carry cve_ids additionally get a minimal FK-target row per aliased CVE (EnsureMinimalCVEMetadata, main.go:559) whose datePublished is 0 and which holds no content of its own — those rows exist to satisfy the CVEAlias foreign key and must be excluded from any coverage metric.
• CVEDescription, CVEMetadataReferences, CVEMetric, CVEProblemType (CVE relation tables)
• CVEAlias (via db.InsertAliases — H1-<reportId> ↔ each cve_ids entry)
• BulkDataDumpTracker (source=hackerone_hacktivity — cursor/freshness)

Environment variables:

• DATABASE_URL — required
• DATABASE_URL_READ — optional

2. GraphQL API Details

Query Structure

{
  reports(first: 100, after: "cursor", where: {disclosed_at: {_gt: "2026-04-01T00:00:00Z"}}) {
    pageInfo { hasNextPage endCursor }
    edges {
      cursor
      node {
        id title url state substate
        created_at disclosed_at closed_at cve_ids
        severity {
          rating score attack_vector attack_complexity
          privileges_required user_interaction scope
          confidentiality integrity availability
        }
        weakness { id external_id name description }
        team { handle name url }
        reporter { username name }
        structured_scope { asset_identifier asset_type }
        bounties { edges { node { amount bonus_amount } } }
      }
    }
  }
}

Available Top-Level Queries

Field Availability

Filtering

The where argument accepts FiltersReportFilterInput with at least:

• disclosed_at: { _gt: "ISO8601", _is_null: Boolean }
• reporter: ... (untested)

No order_by — results are ordered by disclosed_at descending by default.

Sample Responses

Report with CVE and bounty:

{
  "title": "Add labels to arbitrary issues/prs & compromise github actions label checks",
  "url": "https://hackerone.com/reports/3527771",
  "cve_ids": ["CVE-2026-3306"],
  "severity": { "rating": "medium" },
  "weakness": { "external_id": "cwe-639", "name": "Improper Access Control - Generic" },
  "team": { "handle": "github" },
  "bounties": { "edges": [{ "node": { "amount": "0.00", "bonus_amount": "0.00" } }] }
}

Report without CVE:

{
  "title": "libcurl: Integer truncation in curl_easy_ssls_import()",
  "url": "https://hackerone.com/reports/3658049",
  "cve_ids": [],
  "severity": { "rating": "medium" },
  "weakness": null,
  "team": { "handle": "curl", "name": "curl" },
  "bounties": { "edges": [] }
}

3. Business Logic

Processing Strategy

Single phase — paginate all new reports:

1. Check BulkDataDumpTracker for last run timestamp
2. If fresh (< frequency), skip with NoWork
3. Build incremental filter: disclosed_at._gt = last run time (ISO 8601)
4. First run (no tracker): full backfill with no date filter
5. Paginate through all matching reports (100 per page)
6. Batch upsert every 500 reports in a single transaction with SAVEPOINTs

Per-Report Processing

Every report (regardless of CVE presence):

1. UpsertBugBountySubmission — full report metadata
2. UpsertExploit — disclosed reports are exploit/PoC evidence
3. DeleteExploitAffectedProducts + InsertExploitAffectedProduct — from team info
4. UpsertCVEMetadata for H1-<reportId> — title, datePublished from disclosed_at, vendor/product from the team
5. InsertReferences — link to HackerOne report URL
6. InsertProblemTypes — CWE from weakness (if present)
7. InsertMetrics — CVSS 3.1 from severity (if present)
8. InsertDescriptions — report title as description
9. InsertExploitCVE — junction between the Exploit and the H1- record

Reports with cve_ids (additionally, for each CVE ID): 10. InsertAliases — H1-<reportId> ↔ CVE edge (the canonical alias write path) 11. EnsureMinimalCVEMetadata — guarantee the alias FK target exists 12. InsertExploitCVE — junction between the Exploit and the real CVE

A report whose disclosed_at is null lands with CVEMetadata.datePublished = 0 (main.go:479) — in practice HackerOne always sets disclosed_at on the reports the unauthenticated API returns, so every H1- row in production carries a real date.

CVSS Vector Construction

HackerOne severity components are mapped to CVSS 3.1 abbreviations:

Vector is only constructed when all 8 components are present. If any is nil, only baseSeverity and baseScore are stored.

Rate Limiting & Retry

• 1-second delay between GraphQL page fetches
• Single retry with 5-second backoff on fetch failure
• Soft deadline: 50 minutes (configurable via EXPECTED_DURATION_MINUTES)

Idempotency

All writes use ON CONFLICT DO UPDATE or ON CONFLICT DO NOTHING. Safe to re-run. The BugBountySubmission and Exploit tables use (reportId, source) and (exploitId, source) unique constraints respectively.

4. Architecture Diagram

5. Deployment Diagram

6. Processing Flow

7. Data Mapping

8. Field Mapping Reference

9. New Table: BugBountySubmission

Migration SQL (idempotent)

CREATE TABLE IF NOT EXISTS "BugBountySubmission" (
    "uuid" TEXT NOT NULL,
    "reportId" TEXT NOT NULL,
    "source" TEXT NOT NULL,
    "title" TEXT NOT NULL,
    "reportUrl" TEXT,
    "substate" TEXT,
    "teamHandle" TEXT,
    "teamName" TEXT,
    "reporterUsername" TEXT,
    "reporterName" TEXT,
    "bountyAmount" TEXT,
    "bonusAmount" TEXT,
    "assetIdentifier" TEXT,
    "assetType" TEXT,
    "cveIds" TEXT,
    "weaknessCweId" TEXT,
    "weaknessName" TEXT,
    "severityRating" TEXT,
    "severityScore" DOUBLE PRECISION,
    "createdAtSource" BIGINT,
    "disclosedAt" BIGINT,
    "closedAt" BIGINT,
    "createdAt" BIGINT NOT NULL,
    "updatedAt" BIGINT,
    CONSTRAINT "BugBountySubmission_pkey" PRIMARY KEY ("uuid")
);
CREATE UNIQUE INDEX IF NOT EXISTS "BugBountySubmission_reportId_source_key"
    ON "BugBountySubmission"("reportId", "source");
CREATE INDEX IF NOT EXISTS "BugBountySubmission_source_idx"
    ON "BugBountySubmission"("source");
CREATE INDEX IF NOT EXISTS "BugBountySubmission_teamHandle_idx"
    ON "BugBountySubmission"("teamHandle");
CREATE INDEX IF NOT EXISTS "BugBountySubmission_substate_idx"
    ON "BugBountySubmission"("substate");
CREATE INDEX IF NOT EXISTS "BugBountySubmission_severityRating_idx"
    ON "BugBountySubmission"("severityRating");
CREATE INDEX IF NOT EXISTS "BugBountySubmission_disclosedAt_idx"
    ON "BugBountySubmission"("disclosedAt");
CREATE INDEX IF NOT EXISTS "BugBountySubmission_reporterUsername_idx"
    ON "BugBountySubmission"("reporterUsername");

Design Rationale

• Source-agnostic: source column supports future Bugcrowd/Intigriti processors
• Bounty as string: Preserves decimal precision without float rounding
• CWE denormalized: weaknessCweId enables fast filtering without JOIN
• Unique constraint: (reportId, source) prevents duplicates per platform
• No FK to CVEMetadata: Reports may have zero CVE IDs; CVE linkage is handled via Exploit/ExploitCVE

10. Infrastructure

11. Verification

# Local test
just go-hackerone-json-backfill

# Check BugBountySubmission
psql "$DATABASE_URL" -c 'SELECT COUNT(*) FROM "BugBountySubmission" WHERE source = '\''hackerone'\'''

# Check CVE population
psql "$DATABASE_URL" -c 'SELECT COUNT(*) FROM "CVEMetadata" WHERE source = '\''hackerone'\'''

# Check Exploit population
psql "$DATABASE_URL" -c 'SELECT COUNT(*) FROM "Exploit" WHERE source = '\''hackerone'\'''

# Check junctions
psql "$DATABASE_URL" -c 'SELECT COUNT(*) FROM "ExploitCVE" WHERE source = '\''hackerone'\'''

# Idempotency — run again, counts should not increase
just go-hackerone-json-backfill

# Docker build
podman build --target hackerone-json-processor -f Containerfile.go-processors scripts/go-processors/

S3 Persistence

• Archive path: hackerone/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/hackerone-json-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: store-error

The unit of work is a record, not a file, so each report is re-serialised to canonical JSON and keyed by {reportId}.json (main.go:310). Payload bytes are computed before the transaction opens so no DB connection is held during the S3 PUT; committed reports are archived and failed ones quarantined after the transaction resolves (main.go:292-305). Skipped when S3_BUCKET_NAME is unset.