ai-models-processor

Status: Live Source: each AiProvider.baseUrl + /v1/models (OpenAI-compatible model listing) Type: json (per-provider HTTP GET, driven by rows in AiProvider) Source slug: none — this processor writes no CVEMetadata. Its target tables are AiModel and AiModelChangeLog. Schedule: Hourly at minute 15 (cron(15 * * * ? *)), expected_duration_minutes = 30.

Overview

The AI Firewall (guardrails.vulnetix.com) is a multi-provider OpenAI-compatible gateway. Org policy is expressed at model granularity: OrgAiModel rows carry an allow/deny action and foreign-key a concrete AiModel. The gateway reads those tables read-only on every proxied request. A model that is not in the catalog therefore cannot be allowed, cannot be denied, and cannot be resolved — so the catalog has to stay current with what the providers actually serve.

This processor is the only writer of that catalog. Each run:

Loads every AiProvider where globalEnabled = true, ordered by slug (main.go:150).
Fetches {baseUrl}/v1/models per provider and normalises the listing (main.go:317). Both the wrapped shape ({"data":[…]}) and the bare top-level array that Together returns are accepted.
Reconciles the listing against the rows already stored for that provider, inside one transaction per provider (main.go:193).

Providers are processed independently: one provider’s failure is recorded and the loop continues. The run only exits non-zero when every provider failed (main.go:143).

Records produced

Condition	Records
Model id seen for the first time	`AiModel` insert — `releasedAt` from the provider’s `created` field (seconds → ms) when present, else the run timestamp; `isActive = true`; `metadata` = the provider’s raw model object
`name` or `metadata` differs from the stored row	`AiModel` update + one `AiModelChangeLog` row per changed field carrying `fromValue`/`toValue`
Stored row is inactive and the model is listed again	`AiModel` update (`isActive = true`) + `AiModelChangeLog` row `isActive: false → true`
Stored active row is absent from the listing	`AiModel` update (`isActive = false`) + `AiModelChangeLog` row `isActive: true → false`

releasedAt is written once on insert and never updated afterwards — it is the catalog’s “first observed” anchor, not a provider-reported release date.

Reconciliation rules

Duplicate ids within one listing — first occurrence wins (main.go:205). Mistral ships roughly ten duplicate model ids; processing both would flip-flop metadata between the variants on every run.
Empty listing is treated as a fetch failure, never as a mass delisting (main.go:181). Without this guard a single provider hiccup would flip an entire provider’s catalog to isActive = false.
created is excluded from metadata (main.go:387). Its first-seen value is already persisted as releasedAt, and Mistral stamps created with the current request time on every response — including it would mark all of its models changed hourly.
Metadata comparison is canonicalised (main.go:404): both sides are re-encoded through encoding/json, which sorts object keys, so key ordering alone never registers as a change.

Authentication

Provider API keys are not required to proxy traffic — the gateway forwards the org’s own BYOK key. They matter only here, because some providers gate /v1/models behind auth. Probed 2026-07-13: OpenRouter serves it publicly; OpenAI, Groq, Mistral, DeepSeek, xAI, Together and Fireworks all return 401 unauthenticated.

An optional AI_MODELS_API_KEY_<SLUG> environment variable (slug upper-cased, dashes → underscores) is sent as Authorization: Bearer <key> (main.go:328, main.go:420). A provider that answers 401/403 is skipped with a warning naming the env var it needs.

Known gap: the Bearer header is hard-coded. AiProvider.credential describes the provider’s real auth shape, and Anthropic’s is {"method":"api_key_header","headers":[{"name":"x-api-key",…},{"name":"anthropic-version",…}]} — not a Bearer token. The anthropic provider therefore never authenticates and its catalog is empty, even though it is the provider whose wireApis includes messages (the surface Claude Code speaks).

Failure modes

Situation	Behaviour
`DATABASE_URL` unset	log + `os.Exit(1)` before any work
`AiProvider` query fails	`notifier.Errored` + `os.Exit(1)`
No enabled providers	`notifier.NoWork` + exit 0
One provider fetch/parse/write fails	`notifier.RecordError`, `failed++`, continue with the next provider
Every provider failed	`notifier.Errored` + `os.Exit(1)`
Soft deadline reached	stop starting new providers, report what completed

The soft deadline defaults to 15 minutes and is derived from EXPECTED_DURATION_MINUTES when that is greater than 10 (main.go:80). Because a clean run ends in notifier.Completed, and Completed does not flush messages accumulated through RecordError, a provider that fails on every run (see the Anthropic gap above) does not raise a Slack alert — it is visible only in the task logs and the EMF record-error metric.

Flags

None. Behaviour is driven entirely by the AiProvider rows and the optional AI_MODELS_API_KEY_<SLUG> environment variables.

Key files

File	Purpose
`scripts/go-processors/cmd/ai-models-processor/main.go`	whole processor — provider load, fetch, reconcile, changelog
`../saas/prisma/models/ai-firewall.prisma`	`AiProvider` / `AiModel` / `AiModelChangeLog` / `OrgAiModel` schema and ownership notes
`terraform/go-schedules.tf` (`module "ai_models_processor"`)	schedule, CPU/memory, expected duration
`scripts/task-manager.toml` (`[tasks.ai-models-processor]`)	dashboard entry and the `expected_duration_minutes` the ECR hook injects

S3 Persistence

Not used. This processor does not currently archive payloads or quarantine failures to S3. Per the S3 Persistence Contract this is non-compliant — see the compliance matrix for the implementation roadmap.

Expected paths when implemented:

Archive: ai-models/files/{sha256}/{filename}
Quarantine: failed-feeds/ai-models-processor/{YYYY-MM-DD}/{reason}/{filename}
Likely reasons: (none documented)