0day-today-fetch-backfill

Status: Live (local-only, on demand) Source: a local clone of the 0day.today archive (--repo, default /data/0day.today.archive) Type: fetch (flat-file archive walk) Source slug: 0day-today (in Exploit.source — this processor writes no CVEMetadata) Schedule: None. There is no ECS task definition, no ECR target and no EventBridge schedule. Run it by hand with just go-0day-today-fetch-backfill.

Overview

0day.today is one of the largest public exploit archives and the only one of its size with no machine-readable API. The archive is mirrored as a git repo of flat .txt files — one exploit per file, grouped into five category directories — plus an index.json carrying the metadata the file headers omit (platform, canonical link, raw URL).

This backfill turns that into Exploit rows so the exploit-intelligence surface (/exploits, exploitation-maturity signals, SSVC inputs, the CWSS priority calculation) can cite 0day.today alongside ExploitDB, Metasploit, Nuclei and Nmap NSE. It is a backfill rather than a scheduled processor because the upstream is a git mirror that a human refreshes; there is nothing to poll.

Without it, roughly 39k exploits — the single largest non-ExploitDB corpus we hold — are invisible to exploit-maturity scoring, and the exploit bodies are not in S3 for offline static analysis.

Archive layout

0day.today.archive/
  index.json                 # [{exploit_id, cve[], platform, original_link, github_raw_url, title, author}]
  web-applications/{id}.txt
  dos-poc/{id}.txt
  remote-exploits/{id}.txt
  local-exploits/{id}.txt
  shellcode/{id}.txt

Only those five category directories are walked (categories in main.go:47); files are sorted by category/filename so ordering — and therefore statefile resume — is deterministic. A missing category directory is skipped, not an error.

Each .txt file has a fixed five-field header followed by a blank line and the exploit body:

id: 41235
date: 03/14/2026
title: Example CMS 2.1 - Remote Code Execution
author: someone
cves: CVE-2026-1234, CVE-2026-1235

# Exploit Title: Example CMS 2.1 - RCE
...

The title: field may span multiple lines (everything up to author: is folded into it). A trailing # 0day.today [...] promo footer line and any trailing blank lines are stripped from the body before hashing. NUL bytes are removed and invalid UTF-8 sequences dropped (main.go:471-474) — without this, PostgreSQL rejects the insert with SQLSTATE 22021.

Records produced

datePublished comes from the date: header parsed as MM/DD/YYYY and is stored in milliseconds; an unparseable date yields nil, never 0 (main.go:592-599).

CVE extraction

CVE IDs are merged from three sources in priority order (mergeCVEs, main.go:612-634), deduplicated case-insensitively:

1. index.json cve[] for this exploit id
2. the file’s cves: header line (comma-separated)
3. every CVE-\d{4}-\d{4,} match anywhere in the exploit body

Source 3 is deliberately broad — exploit authors routinely cite the CVE only in a code comment — but it also picks up CVEs merely mentioned in prose (“similar to CVE-2019-0708”). Treat ExploitCVE rows from this source as exploit-adjacent evidence rather than a proof of exploitability for that exact CVE.

Affected-product extraction

Four header conventions are tried in order (extractAffectedProducts, main.go:648-699); the first that yields anything wins:

1. ExploitDB style — # Exploit Title:, # Version:, # Software Link:, # Vendor Homepage:, # Software Web Page:
2. Advisory style (LiquidWorm / zeroscience) — Vendor:, Product Web Page:, Affected version:
3. Metasploit module — detected via class … < Msf:: / MetasploitModule, product taken from the 'Name' => field
4. Title fallback — {product} {version} split out of the exploit title

Resume

Two independent layers, both bypassed by --force:

db.LoadKnownCVEIDs is also loaded up front so ExploitCVE junctions can be written against every (cveId, source) pair that exists. That query scans CVEMetadata for all CVE-% ids and takes minutes on production — it is the dominant startup cost of the run (see AGENTS.md “Known Issues”).

Failure modes

Flags

The justfile recipe adds NO_PULL (default false), which git -C {repo} pulls the archive before the walk, and resolves REPO to an absolute path because the binary runs from scripts/go-processors.

Local development

# Refresh the archive clone and backfill against production
just go-0day-today-fetch-backfill prod

# Re-run without touching the clone, no S3 uploads
just go-0day-today-fetch-backfill prod NO_PULL=true SKIP_S3=true

psql "$DATABASE_URL" -c "
  SELECT count(*) AS exploits,
         count(\"datePublished\") AS with_date,
         count(\"r2Key\") AS with_s3
  FROM \"Exploit\" WHERE source = '0day-today';"

S3 Persistence

• Archive path: 0day-today/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/0day-today-fetch-backfill/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: parse-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.