malwarehost-backfill

Status: Live, local-only, and redundant in normal operation Source: none — this reads the database, not an upstream feed Type: projection / one-shot (no -processor suffix, no data-acquisition type) Source slug: n/a — writes no CVEMetadata Schedule: none, and none is needed. There is no task-manager.toml entry, no Containerfile.go-processors target, no ECR tag, no ECS task definition and no EventBridge schedule. The same function runs automatically inside malscan-stix-processor, which is scheduled every 15 minutes (terraform/go-schedules.tf:3165).

Overview

MalwareHost is the deduplicated, STIX-ready view of every network host we have ever seen implicated in a malicious package or malware campaign. The raw observations live in two places with different shapes — OsmThreatIoc (from the OpenSourceMalware threat feed) and MalwareIoc (from the 16 registry malware processors) — and neither is queryable as “give me every known-bad IP”, because the same host appears many times under different advisories, spellings and IOC type names (ips vs ipv4, domains vs domain).

This command rebuilds that view. It reads both tables, normalises every value through internal/hostnorm, aggregates duplicates into one host row per (kind, value), and records each original observation as a MalwareHostLink row so provenance survives the deduplication.

Its whole reason to exist is being a manual entry point: the projection is otherwise a side effect of a feed-publishing task, and after a bulk import — a campaign backfill, an OSM ingest, a registry malware sweep — you want to materialise the hosts immediately rather than wait for the next scheduled run. It is os.Exit(1) on any error and prints the two counts, so it also doubles as the way to verify the projection by hand.

What it reads

Table	Filter	Code
`OsmThreatIoc`	`iocType IN ('domains','ipv4','ipv6','ips')`	`internal/db/malwarehost.go:279`
`MalwareIoc`	`iocType IN ('domain','ipv4','ipv6','onion')`	`internal/db/malwarehost.go:308`

Note the two vocabularies differ (ips/domains plural on the OSM side, singular plus onion on the registry side); the union is normalised into three hostType values.

Records produced

Table	One row per	Notes
`MalwareHost`	unique (hostType, value)	`stixIndicatorId` = `indicator--<deterministic v5-shaped UUID>` over `"host:<stixType>:<value>"` (`hostnorm.IndicatorID`), `stixPattern` = `[ipv4-addr:value = '…']` (`hostnorm.Pattern`). `validFrom` is the earliest and `lastSeen` the latest `firstSeen` across observations; `maxSeverity` is the highest-ranked severity; `truncated` is sticky — true if any observation was truncated
`MalwareHostLink`	unique provenance tuple	Dedup key is `SHA256(source, kind, value, osmThreatUuid, cveId, cveSource, iocType)` (`hostnorm.LinkKey`), so re-ingesting the same observation is a no-op

Production, 2026-08-06: 8,095 hosts (7,518 IPV4, 565 DOMAIN, 12 IPV6) and 23,860 links (23,077 from REGISTRY, 754 from OSM, 29 from OSS_MALICIOUS_PACKAGES), last updated the same day — i.e. the scheduled path is keeping it current.

Both writes happen in one transaction (internal/db/malwarehost.go:226-238) via bulk upsert, so a failed run leaves the store exactly as it was. Deterministic ids mean re-running produces byte-identical rows, which is what makes the published STIX bundle diff minimally.

Invocation

# Defaults to TARGET=prod — see the warning below
just malwarehost-backfill
just malwarehost-backfill local

Two footguns in the recipe (justfile:351). First, TARGET defaults to "prod", unlike every other recipe in the file, so a bare just malwarehost-backfill writes to production. Second, the recipe’s own comment advertises “Pass flags e.g. --source registry --limit 100” and takes a *FLAGS catch-all — but cmd/malwarehost-backfill/main.go does not import flag and parses no arguments at all, so any flag passed is silently ignored and the full projection runs regardless.

Failure modes

No flags, no scoping. Every run rebuilds the whole projection from both source tables. There is no --source, no --limit and no incremental mode despite the recipe implying otherwise. Cheap today at ~24k links; it scales with the IOC corpus.
No notifier, no EMF, no S3. Unlike the scheduled processors this command emits no task.started / task.completed events and no CloudWatch metric, so a manual run is invisible to the observability dashboard. Appropriate for a local tool, worth knowing when debugging.
Silent normalisation drops. hostnorm.Normalize returning !ok skips the value with no counter and no log line (internal/db/malwarehost.go:186-188), so an IOC vocabulary change upstream would shrink the projection without any signal.

S3 Persistence

Not applicable. This command consumes no upstream payload — it is a database-to-database projection, so there is nothing to archive or quarantine. The S3 Persistence Contract applies to processors that ingest an external feed.