Documentation

Architecture

System design, agent conventions, and shared-library reference for the VDB Manager pipeline.

Runtime Infrastructure

All scheduled processors run as ECS Fargate tasks inside the vdb-scheduler cluster (aws_ecs_cluster.vdb in terraform/ecs.tf). EventBridge Scheduler triggers each task on its configured cron; failed invocations are routed to the vdb-scheduler-dlq SQS dead-letter queue (aws_sqs_queue.scheduler_dlq in terraform/logs.tf) with a 14-day retention. The scanner processor (go-scanner-processor) is also hosted in this cluster but is triggered on-demand via ecs:RunTask rather than by a schedule.

Processor Naming Convention

How processor names map across cmd/ directories, Containerfile targets, ECR tags, ECS task definitions, EventBridge schedules, CloudWatch log groups, task-manager.toml, terraform/go-schedules.tf, and justfile recipes.

S3 Persistence Contract

Every processor archives successful payloads to {source}/files/{sha256}/{filename} and quarantines failures to failed-feeds/{processor}/{date}/{reason}/{filename} in a single bucket.

S3 Persistence Compliance Matrix

Per-processor audit of S3 archive and quarantine implementation status across the ~168 Go processor commands.

Threat-Actor Attribution

A shared engine resolves the AUTHOR of a malicious package from package/repo identity plus upstream registry, GitHub, and Docker Hub lookups, harvests their public keys, and exposes the result through the Pro-gated v2 API.

Malware Detection Engine

The shared malscan-engine that every registry processor runs over a package's source — factual evidence, weak triggers and a combination gate, gated per-ecosystem by a capability config.

Known-Bad IOC Feed (STIX)

Malicious domains, IPs and URLs from the malware pipeline are republished as public per-ecosystem STIX 2.1 feeds, which the detection engine then matches future packages against — a closed feedback loop.

Shared Business Logic Library

The Go packages under scripts/go-processors/internal/ that the cmd/ processors share: database access, AI enrichment, malware/threat-actor intelligence, S3 persistence, and the run-lifecycle helpers. Per-source feed parsers live in their own internal/<source>/ package and are documented on each processor's page.

OSM Manager

The console surface for OpenSourceMalware: reviewing what we hold, checking a resource against their live database, and submitting our own findings with the quality gate that keeps them publishable.