VARIoT JSON Processor — Design
Overview
Fetches IoT vulnerability and exploit records from the VARIoT JSON API
(variotdbs.pl). Vulnerabilities are stored in CVEMetadata + relations.
Exploits are stored in Exploit + ExploitCVE + ExploitAffectedProduct.
Two binaries share the same pagination logic:
- ECS processor (
variot-json-processor): daily, fetches last 3 days - Local backfill (
variot-json-backfill): iterates year-by-year to 1999
API
| Endpoint | URL | Auth | Description |
|---|---|---|---|
| Vulns | /api/vulns/?jsonld=false&since=DATE&limit=100 | Authorization: Token ... | Paginated CVE records |
| Exploits | /api/exploits/?jsonld=false&since=DATE&limit=100 | Same | Paginated exploit records |
Pagination: next/previous URLs in response. Date filter: since/before.
Data Mapping
Vulnerabilities → CVEMetadata
| API Field | DB Target |
|---|---|
cve | CVEMetadata.cveId |
title.data | CVEMetadata.title |
description.data | CVEDescription (lang=en) |
cvss.data[].cvssV2/V3[] | CVEMetric (vectorString, baseScore) |
problemtype_data.data[].problemtype | CVEProblemType (CWE IDs) |
affected_products.data[] | CVEAffected (vendor, model, version) |
references.data[].url | CVEMetadataReferences |
patch.data[].url | CVEMetadataReferences (type=fix) |
external_ids.data[] | CVEAlias (JVNDB, CNVD, etc.) |
Exploits → Exploit
| API Field | DB Target |
|---|---|
id | Exploit.exploitId |
title.data | Exploit.title |
type.data | Exploit.category |
exploit.data | SHA256 → Exploit.bodyContentHash |
external_ids (CVE IDs) | ExploitCVE junctions |
affected_products.data[] | ExploitAffectedProduct |
Rate Limiting & Retry
- 100ms sleep between API page requests
- 3 retries with exponential backoff (2s, 4s, 8s)
- HTTP 429: sleep 30s then retry
- HTTP 5xx: sleep 5s then retry
Daily ECS Task
Fetches since=3-days-ago, UPSERTs all records. Schedule: cron(0 6 * * ? *).
Backfill
Iterates year-by-year from --start-year (default: current) to --end-year
(default: 1999). Stops after 2 consecutive years with zero results.
Secrets
VARIOT_API_TOKEN stored in AWS Secrets Manager (prod/vdb-manager/api-keys)
and .env.production for local testing.
Key Files
| File | Purpose |
|---|---|
cmd/variot-json-processor/main.go | ECS daily processor |
cmd/variot-json-backfill/main.go | Local historical backfill |
internal/variot/types.go | API response and advisory types |
internal/variot/parser.go | JSON parsing, CVSS/CWE/affected extraction |
internal/variot/mapper.go | Advisory → CVESourceData, exploit helpers |
internal/variot/client.go | Shared pagination with retry logic |
schemas/variot_json_advisory.schema.json | JSON Schema Draft 7 |
S3 Persistence
Not used. This processor does not currently archive payloads or quarantine failures to S3. Per the S3 Persistence Contract this is non-compliant — see the compliance matrix for the implementation roadmap.
Expected paths when implemented:
- Archive:
variot/files/{sha256}/{filename} - Quarantine:
failed-feeds/variot-json-processor/{YYYY-MM-DD}/{reason}/{filename} - Likely reasons:
parse-error