MISP Galaxy JSON Processor — design

Why

The MISP Project publishes a public threat-actor cluster at https://raw.githubusercontent.com/MISP/misp-galaxy/main/clusters/threat-actor.json. Roughly 1k actors are described; ~60–70 of them carry CVE references in their meta fields (notably meta.refs[]) and free-text description/synonyms. These actor↔CVE links are high-signal exploit-intel: a CVE that a named APT is known to use is materially more interesting than one with no in-the-wild mention, even when no PoC code exists.

This processor mirrors the existing github-poc-processor pattern: each threat-actor that references a CVE becomes a row in the Exploit table with source = "misp-galaxy", joined to CVEMetadata via ExploitCVE. Unlike a PoC repo, the artifact is the actor profile itself; the originalUrl points back to the cluster JSON so consumers can navigate to the upstream documentation.

Source

Single JSON file fetched on every run:

https://raw.githubusercontent.com/MISP/misp-galaxy/main/clusters/threat-actor.json

Top-level shape:

{
  "name": "Threat Actor",
  "uuid": "7cdff317-…",
  "values": [
    {
      "uuid": "1cb7e1cc-d695-…",
      "value": "APT1",
      "description": "PLA Unit 61398 …",
      "synonyms": [...],
      "meta": {
        "refs": ["https://…", "…CVE-2014-1761…"],
        "country": "CN",
        "synonyms": [...],
        ...
      }
    }
  ]
}

CVE extraction

For each values[] entry:

1. Walk every string in meta (scalar + arrays) and apply regex CVE-\d{4}-\d{4,7} (case-insensitive).
2. Apply the same regex to description and synonyms[].
3. Uppercase + dedupe → cveIDs.
4. If empty, skip the actor (no exploit row written).

DB writes

Exploit columns:

ExploitCVE rows are produced by resolveCVESources (same helper used by github-poc-processor / nse / yara ingestion) — so for each CVE we link against every (cveId, source) row that exists in CVEMetadata, plus alias-equivalents via CVEAlias. CVEs that aren’t yet in CVEMetadata are silently skipped; a future reprocess will pick them up once a primary source ingests them.

Tracker / freshness

BulkDataDumpTracker row: source = "misp_galaxy_threat_actor". A run is skipped if tracker.LastProcessedAt + Frequency × 1s is in the future. Frequency: 86 400 s (daily). The tracker stores the SHA256 of the fetched JSON so that an unchanged feed exits as noWork after fetch+hash.

Source enum value

misp-galaxy — distinct from any other exploit source. The dataset is small (≤ 1k actors, ≤ 100 with CVEs), so there’s no need for sharding or per-actor resume; a full re-walk costs <1 s of CPU.

Required environment

• DATABASE_URL (write)
• DATABASE_URL_READ
• standard AWS credentials (no S3 needed — JSON is small enough to skip archival; the upstream URL is permanent and the artifact is text)
• optional EXPECTED_DURATION_MINUTES (default 10)

Schedule

Daily at 03:30 UTC via EventBridge (cron(30 3 * * ? *)). Picked to avoid contention with github-poc-processor (04:00) and nse (03:00).

Sizing

256 CPU / 512 MB. The whole feed is ~3 MB JSON; in-memory parse + ~75 DB upserts. p90 runtime is well under one minute; we set expected_duration_minutes = 10 so the soft-deadline path is exercised but never the bottleneck.

Verification

just go-misp-galaxy-fetch prod

A successful run prints actor / CVE-link counts and exits 0. Confirm rows:

SELECT COUNT(*) FROM "Exploit" WHERE "source" = 'misp-galaxy';
SELECT COUNT(*) FROM "ExploitCVE" e
  JOIN "Exploit" x ON x."uuid" = e."exploitUuid"
  WHERE x."source" = 'misp-galaxy';

S3 Persistence

Not used. This processor does not currently archive payloads or quarantine failures to S3. Per the S3 Persistence Contract this is non-compliant — see the compliance matrix for the implementation roadmap.

Expected paths when implemented:

• Archive: misp-galaxy/files/{sha256}/{filename}
• Quarantine: failed-feeds/misp-galaxy-json-processor/{YYYY-MM-DD}/{reason}/{filename}
• Likely reasons: (none documented)