Repo Scan Git Processor — Design Document

1. Overview

Purpose: Discover git repositories referenced anywhere in the VDB (package upstreams, dependencies, malware samples/attributions, CVE references, exploits, YARA rules) and run the Vulnetix CLI security scans against them, once per tenant org that references each repo.

Schedule: Twice daily at 02:00 and 14:00 UTC (cron(0 2,14 * * ? *)). A 12h cadence suits the workload — the queue only refills when a repo’s pushedAt advances, so a shorter period mostly re-runs discovery for nothing, while each run can clone and scan for a long time.

Disabled by default (schedule_enabled = false): the scan-state migration must be applied to the target database first — without it every run exits 1 at the startup schema check and pages via task.errored. Enable once the migration is applied and a -dry-run has been reviewed.

Timeout: 120 minutes (per-repo timeout PER_REPO_TIMEOUT_MINUTES, default 30).

Resources: 1024 CPU units, 4096 MB memory, 100 GiB ephemeral storage (throwaway clones). Image is alpine-based (not scratch) because it needs a real userland: git plus the Vulnetix CLI.

What it reads:

• Source columns swept for github.com repo references: PackageVersion.sourceRepoUrl, CVEAffected.repo/collectionURL, CVEMetadataReferences.url, MalwareThreatActor.repoUrl, MalwareAttribution.claimedRepoUrl, OsmThreat.claimedRepoUrl/resourceIdentifier, OsmThreatActor.repoUrl, Exploit.originalUrl, Dependency.purl (pkg:github), YaraRule.sourceUrl.
• Org linkage: GitHubRepository.orgId, GitHubRepoDependency.orgId, InsightsRun.orgId (scope=REPO) — all saasOrgId values — joined to VdbOrganization to get the principal to authenticate as.
• Authentik REST API: GET /api/v3/core/users/?attributes={"vdb_principal_uuid":…}, reading the user’s hmac_secret attribute.

What it writes:

• GitHubRepository scan-state columns (see §3): needsScan, scannedAt, scanSha, scanSkipReason, scanError, scannedOrgIds.
• Per-org scan evidence (BOMs, analysis) is uploaded to Vulnetix by the CLI, not stored in this DB.

Environment variables:

• DATABASE_URL / DATABASE_URL_READ — RDS connection strings.
• AUTHENTIK_API_URL — Authentik base (e.g. https://auth.vulnetix.com).
• AUTHENTIK_API_TOKEN — admin API token (from the vdb-site-api secret).
• BYOK_KMS_KEY_ARN — KMS key that decrypts each org’s hmac_secret.
• VULNETIX_CLI_PATH — CLI binary (default vulnetix on PATH).
• SCAN_SUBCOMMANDS — comma list (default analyze,cdx,cbom,aibom).
• SCAN_BATCH (10), MAX_REPO_MB (2048), PER_REPO_TIMEOUT_MINUTES (30), DISCOVERY_MAX_PER_SOURCE (50000).

Identity model (do not conflate)

Two uuids are in play — see vdb-api auth-and-tenancy:

Discovery yields saasOrgIds; scanning needs principals. They are joined via VdbOrganization.saasOrgId. A repo is scanned once per SaaS tenant, authenticating as one active principal of that tenant (preferring the tenant’s own row) — scanning per principal would rescan for every team member.

Subcommand upload behaviour

analyze, cbom and aibom submit their results automatically when authenticated (/v2/cli.insights, /v2/cli.cbom, /v2/cli.ai-bom). cdx does not — it is documented as “Generate a standalone CycloneDX SBOM without VDB lookup or upload”, so it writes a local artifact that this processor then discards along with the rest of .vulnetix. Set SCAN_SUBCOMMANDS to use sca instead if the SBOM should be persisted per org.

The CLI spells it analyze; analyse is not a registered alias.

2. Business Logic

1. Discover referenced github.com owner/repo full names across the source columns (each source capped and tolerant of missing tables), then flag matching GitHubRepository rows needsScan=true — never scanned, or changed since the last scan (pushedAt advanced past scannedAt).

2. Claim a batch of queued repos (oldest first), bumping scannedAt.

3. Per repo (panic-recovered, per-repo timeout): resolve owning orgs; if none, skip (no_orgs). Guard disk: pre-clone free-space check plus a size watchdog that aborts the clone if it exceeds MAX_REPO_MB (too_large). Clone once (shallow, single-branch, no-tags) via go-git into a temp dir.

4. Per owning org: resolve VULNETIX_API_KEY for the tenant’s principal from Authentik + KMS (cached), wipe .vulnetix in the repo and an isolated HOME, export the org credentials, run vulnetix auth verify and confirm the authenticated principal uuid, then run each scan subcommand (continue-on-error). Wipe .vulnetix again. No re-clone between orgs.

   Credential isolation: the CLI resolves credentials in a fixed order (VULNETIX_API_TOKEN → VULNETIX_API_KEY+VULNETIX_ORG_ID → VVD_* → .vulnetix files). All of those are stripped from the ambient environment before each org’s own values are layered on, so an inherited token cannot silently hijack every scan.

5. Record scannedAt/scanSha/scannedOrgIds, remove the clone, move on.

Every terminal outcome (success, too_large, no_orgs, clone_failed) sets needsScan=false so repos and skips are not reprocessed unless the repo changes or -force is passed.

3. Data Model note

The scan queue + dedup ledger reuse GitHubRepository. The scan-state columns (needsScan, scannedAt, scanSha, scanSkipReason, scanError, scannedOrgIds) plus the (needsScan, scannedAt) index are owned by the schema — Prisma migration 20260813000001_github_repository_scan_state in Vulnetix/saas. The processor does not create them; it verifies they exist at startup and fails fast with an actionable message if the migration has not been applied. (Creating them from here would leave them invisible to Prisma, so a migrate reset or db push would silently drop the ledger.)

Because the ledger is GitHubRepository, v1 covers github.com repos; non-github references are discovered but not scanned — see Vulnetix/saas#19 for the longer-term per-(repo, org) coverage decision.

4. Running it

# Preview only — read-only. Reports, per discovered repo, whether it resolves to
# a GitHubRepository row and which orgs it would be scanned for.
just go-repo-scan-git-backfill                 # DRY_RUN defaults to true
just go-repo-scan-git-backfill prod            # same, against .env.production

# One real repo, end to end.
just go-repo-scan-git-backfill prod false octocat/Hello-World

# Drain the existing queue without re-running discovery.
just go-repo-scan-git-backfill prod false "" 0 false true

Flags: -dry-run, -repo owner/name, -limit N, -force (re-flag repos already scanned or skipped), -no-discover.

5. Verifying

-- Queue depth and outcomes.
SELECT "needsScan", "scanSkipReason", count(*)
FROM "GitHubRepository" GROUP BY 1, 2 ORDER BY 3 DESC;

-- What a run actually scanned, and for whom.
SELECT "fullName", "scanSha", "scannedOrgIds",
       to_timestamp("scannedAt" / 1000) AS scanned
FROM "GitHubRepository"
WHERE "scanSha" IS NOT NULL ORDER BY "scannedAt" DESC LIMIT 20;

-- Repos permanently skipped, and why.
SELECT "scanSkipReason", count(*), min("fullName") AS example
FROM "GitHubRepository" WHERE "scanSkipReason" IS NOT NULL GROUP BY 1;

Expected behaviour on a second run with no upstream changes: repos flagged for scan = 0, no repos queued for scan, and the CLI is never invoked. A repo is re-queued only when GitHub’s pushedAt advances past scannedAt, or with -force.

Local end-to-end testing without AWS

The processor can be exercised against a throwaway Postgres with no AWS credentials: a secret stored without the kms:v1: prefix is treated as legacy plaintext and returned as-is, so a stub Authentik returning a plain hmac_secret is enough to drive the whole loop. Point VULNETIX_CLI_PATH at a stub binary to avoid real scans/uploads.

The properties worth asserting in such a run: