variot-json-backfill
Status: Live (local-only) Source: VARIoT JSON API (
variotdbs.pl) Type:jsonSource slug:variot— shared withvariot-json-processorSchedule: none. No ECS task definition, no EventBridge schedule. Requires:VARIOT_API_TOKEN
Overview
variot-json-processor is the daily ECS task and reads only the last 3 days
of updates. That is correct for keeping current but leaves VARIoT’s entire
history — back to roughly 2002 — unreachable. This tool is that path.
It walks backwards year by year and stops when two consecutive years return zero results. The two-year rule rather than one exists because a sparse year in the middle of the archive should not be mistaken for the end of it.
It shares internal/variot with the daily processor, including the pagination
logic, so the two cannot drift in how they page the API.
Run it with:
just go-variot-json-backfill prod
Being a backfill, it must run to completion — it carries no ECS soft deadline
(see the “Backfill must not have a deadline” rule in
scripts/go-processors/AGENTS.md).
Records produced
Identical to the daily processor, because the mapping is shared:
| Condition | Records |
|---|---|
| Vulnerability record | CVEMetadata (source="variot"), CVEDescription, CVEMetric, CVEAffected, CVEMetadataReferences |
| Exploit record | Exploit, ExploitCVE, ExploitAffectedProduct |
Both kinds are written under source="variot".
Known issue — run the date fix first
datePublished is a NOT NULL int4 seconds column, so it cannot represent
dates before roughly 1902. VARIoT returns 1900-01-01 for some old records.
A previous fix in internal/variot/mapper.go stopped the resulting int4
overflow crash by leaving DatePublished unset when the value is not positive
— which converts an unrepresentable date into a silent zero rather than
preserving it.
That matters more here than for the daily processor: 8,858 of the 31,777
existing variot rows (28%) already have datePublished = 0, and a walk back
to 2002 is precisely where pre-1902 upstream dates live, so a full historical
run will generate proportionally more of them.
Fix the date handling before running the full backfill, or it bakes the
defect in at scale. Either clamp such records to the earliest representable
date while recording the true date in rawDataJSON, or widen the column. A
datePublished of 0 sorts the record to 1970 in every date-ordered view.
Why VARIoT matters
VARIoT is EU-funded and specialises in IoT devices — a class mainstream feeds cover poorly, because the vendors are numerous, small, and frequently do not participate in the CVE programme at all. It is also unusual in publishing both vulnerabilities and matched exploits, so an IoT record can arrive with its known exploit already linked rather than needing separate correlation.