Tree-sitter Query Generation Activity — Design Document

Activity key: vulnetix.treesitter Code root: scripts/go-processors/internal/aienrich/treesitter.go Prompt: scripts/go-processors/internal/aienrich/prompts/treesitter.md Seed: saas/prisma/migrations/20260516000001_add_tree_sitter_queries/migration.sql Schema spec: .repo/treesitter-schema.md

1. Overview

Purpose: For every primary CVEMetadata write — NVD, GHSA, OSV, MSRC, RHSA, Red Hat CSAF — emit zero-or-more tree-sitter queries that match the vulnerable code pattern in parsed source code. Downstream Vulnetix services load these queries verbatim via the tree-sitter C API and run them against user code to detect at-risk call-sites.

Type: aienrich activity (not a standalone processor) — fires from (*Enricher).RunWithAliases per target, after the affected / ATT&CK / CWE passes. No ECS task, no EventBridge schedule, no Containerfile target.

Trigger surface: every record committed by every processor that calls enricher.RunBatch (NVD recent / modified / year, GHSA git, GHSA RSS, OSV, MSRC CSAF, RHSA RSS, Red Hat CSAF). The structured-JSON passes share the default 90 s per-CVE budget from RunWithAliases.

Tree-sitter primer: a query is an S-expression whose nodes match grammar symbols, @captures tag interesting sub-nodes, and #predicates filter matches (#eq?, #match?, #any-of? plus not- variants). #directives (#set!) carry annotations. The full grammar is at the tree-sitter docs.

What it reads:

Read replica: Pix row for vulnetix.treesitter (provider, model, responseMaxTokens, temperature, topP, systemPromptText)
Read replica: same context as vulnetix.affected via gatherCVEContext — CVEMetadata snapshots (per source), deduped CVEDescriptions, canonicalised CVEMetadataReferences
Cloudflare AI Gateway: one chat-completion call to the configured model

What it writes:

PixLog — always, regardless of outcome
CVETreeSitterQuery — one row per emitted query, dedup’d on (cveId, source, language, sha1(queryText))
CVETreeSitterCapture — one row per @capture in the query
CVETreeSitterPredicate — one row per predicate AND per directive (distinguished by the kind column)

Environment variables: identical to every other aienrich activity — PIX_INFERENCE_ENABLED=true, CF_AIG_TOKEN (or AI_GATEWAY_TOKEN), AI_GATEWAY_URL.

Pix row (seeded by saas migration 20260516000001_add_tree_sitter_queries):

Field	Value
`activityKey`	`vulnetix.treesitter`
`provider`	`cloudflare-ai-gateway`
`inferenceModel`	`@cf/zai-org/glm-5.2`
`responseMaxTokens`	`8192` — raised from the seeded `4096`, which truncated ~19% of glm-5.2 answers
`systemPromptText`	`NULL` on seed — populate via `just pix-prompts-sync prod`

Until pix-prompts-sync runs the runtime falls back to the markdown embedded in the Go binary and emits one warn log per call (Pix.systemPromptText empty, using embedded fallback).

2. Business Rules

2.1 Activity gate (`(*Enricher).runTreeSitter`, `treesitter.go`)

#	Condition	Returns	Reason
1	`e == nil`	n/a	aienrich disabled (no-op everywhere)
2	`Pix` row for `vulnetix.treesitter` missing or `enabled=false`	error	“configured-off” — same behaviour as every other activity
3	`Pix.inferenceModel` empty	error	fail-loud — misconfigured Pix row

2.2 Output contract (`prompts/treesitter.md`)

The model MUST return either:

A single fenced JSON block:

{ "queries": [ { "language", "name", "description", "queryText",
                 "captures": [...], "predicates": [...], "directives": [...] } ] }

with every query carrying non-empty language, name, and queryText.

The bare sentinel string FAILURE_TO_DERIVE_TREE_SITTER_QUERY.

2.3 Parse + persist rules (`treesitter.go`)

#	Condition	Effect
4	`isSentinel(response, "FAILURE_TO_DERIVE_TREE_SITTER_QUERY")`	PixLog row written; no CVETreeSitter* rows; activity returns `nil`
5	Fenced-JSON parse fails	PixLog `rejected=true`; activity returns parse error
6	`treeSitterHasContent` returns `false` (every query is missing language, name, or queryText)	PixLog `rejected=true`; activity returns “model output empty after parse”
7	At least one valid query → `persistTreeSitter` transaction	one INSERT into `CVETreeSitterQuery` per valid query (`ON CONFLICT (cveId, source, language, queryHash) DO UPDATE` — idempotent); captures + predicates DELETE’d then re-INSERTed so they stay in sync with the latest model output
8	Capture name with leading `@` (model didn’t strip it)	leading `@` trimmed before insert (we store the bare name)
9	Predicate name starts with `not-` (e.g. `not-eq`)	canonical positive form stored (`eq`); `negated=true`
10	Predicate `args` is nil / empty	persisted as `[]` (JSONB column is always a valid array)
11	Mixed predicates + directives	both flow into `CVETreeSitterPredicate`; `kind` column = `"predicate"` or `"directive"`

2.4 Caching + cache invalidation

Same as every other activity: pixConfig cached 5 min; invalidatePixCache("vulnetix.treesitter") fires on gateway error so a misconfigured Pix row is re-read on the next attempt.

2.5 PixLog contract

A PixLog row is always written before runTreeSitter returns. The row carries vulnId = cveID, vulnSource = source, full prompts, response text (or sentinel literal / "ERROR: …"), token usage, and rejected = true for parse failures / empty output / gateway errors; false for success and sentinel.

3. Data Model

Three tables (Prisma models in prisma/models/treesitter.prisma). Postgres DDL is in the migration referenced at the top of this page; the unique index (cveId, source, language, queryHash) is the idempotency key.

erDiagram CVEMetadata ||--o{ CVETreeSitterQuery : has CVETreeSitterQuery ||--o{ CVETreeSitterCapture : has CVETreeSitterQuery ||--o{ CVETreeSitterPredicate : has CVETreeSitterQuery { string uuid PK string cveId string source string derivedBy string language string name string description string queryText string queryHash bigint createdAt } CVETreeSitterCapture { string uuid PK string queryId FK string name string kind } CVETreeSitterPredicate { string uuid PK string queryId FK string kind string name bool negated json args }

4. Architecture Diagram

graph TD subgraph "cmd/* primary processors" NVD[nist-nvd-* json] GHSAG[ghsa-git] GHSAR[ghsa-rss] OSV[osv-json] MSRC[msrc-csaf] RHSA[rhsa-rss] REDHAT[redhat-csaf] end subgraph "internal/processor/" PIPE[pipeline.go
Run, processBatch] end subgraph "internal/aienrich/" ENR[enricher.go
RunBatch / RunWithAliases] TS[treesitter.go
runTreeSitter, persistTreeSitter] CTX[context.go
gatherCVEContext] CLIENT[client.go
callGateway] PIX[pix.go
loadPixConfig] LOG[pixlog.go
writePixLog] PROMPT[prompts/treesitter.md
embedded] end subgraph "internal/db (Postgres)" META[(CVEMetadata)] DESC[(CVEDescription)] REFS[(CVEMetadataReferences)] PIXTBL[(Pix)] PIXLOGTBL[(PixLog)] TSQ[(CVETreeSitterQuery)] TSC[(CVETreeSitterCapture)] TSP[(CVETreeSitterPredicate)] end subgraph "External" CFG[Cloudflare AI Gateway
glm-5.2] end NVD --> PIPE GHSAG --> PIPE GHSAR --> ENR OSV --> ENR MSRC --> ENR RHSA --> ENR REDHAT --> ENR PIPE --> ENR ENR --> TS TS --> PIX PIX --> PIXTBL TS --> CTX CTX --> META CTX --> DESC CTX --> REFS TS --> CLIENT CLIENT --> CFG CLIENT --> PROMPT TS --> LOG LOG --> PIXLOGTBL TS --> TSQ TSQ --> TSC TSQ --> TSP

5. Decision Tree (per CVE)

flowchart TD A[runTreeSitter cveID, source] --> B[load Pix vulnetix.treesitter] B --> C{row exists + enabled?} C -->|no| Z1[return error] C -->|yes| D[gatherCVEContext] D --> E[buildPrompts system+user] E --> F[callGateway] F --> G{HTTP error?} G -->|yes| Z2[PixLog ERROR rejected=true] G -->|no| H{response == sentinel?} H -->|yes| Z3[PixLog success return nil] H -->|no| I[unmarshalFencedJSON] I --> J{parse ok?} J -->|no| Z4[PixLog rejected=true] J -->|yes| K{at least one valid query?} K -->|no| Z4 K -->|yes| L[persistTreeSitter tx] L --> M[per query: UPSERT CVETreeSitterQuery] M --> N[DELETE+INSERT CVETreeSitterCapture] N --> O[DELETE+INSERT CVETreeSitterPredicate] O --> P[COMMIT] P --> Q[PixLog success return nil]

6. Data Mapping

6.1 Inputs assembled into the user-prompt JSON (`cveContext`)

Identical to the vulnetix.affected pass — same dedup’d snapshot built by gatherCVEContext:

JSON path	Source table / column
`cveID`	input parameter
`sources[]`	`CVEMetadata` rows collapsed on `(title, vendor, product)`
`descriptions[]`	`CVEDescription.value` deduped on normalised text
`references[]`	`CVEMetadataReferences.url` canonicalised + deduped

6.2 Outputs persisted

`CVETreeSitterQuery` (UPSERT on `(cveId, source, language, queryHash)`)

Column	Value
`uuid`	`uuid.NewString()` (new row) or existing on conflict
`cveId`	input parameter
`source`	input parameter (e.g. `nvd`, `github`, `osv`, `rhsa`, …)
`derivedBy`	`"vulnetix"`
`language`	tree-sitter grammar id, lower-case (`c`, `javascript`, …)
`name`	model-provided short snake_case identifier
`description`	model-provided one-line description (nullable)
`queryText`	full S-expression, trimmed
`queryHash`	`sha1(queryText)` lowercase hex — idempotency key
`createdAt`	`time.Now().UnixMilli()` (BigInt ms)

`CVETreeSitterCapture`

Column	Value
`queryId`	FK to `CVETreeSitterQuery.uuid`
`name`	capture name with leading `@` stripped
`kind`	model-provided hint (`identifier`, `node`, …), nullable

`CVETreeSitterPredicate`

Column	Value
`queryId`	FK to `CVETreeSitterQuery.uuid`
`kind`	`"predicate"` or `"directive"`
`name`	canonical positive form (`eq`, `match`, `any-of`, `set`, …)
`negated`	`true` when the model emitted `not-*` or set `negated:true`
`args`	JSONB array of arg strings (captures keep their `@`)

`PixLog` row (always)

Same shape as every other aienrich activity — see the ghsapoc design doc for the column-by-column contract. The only differences: vulnId = cveID, vulnSource = source, and responseText is either the JSON block, the FAILURE_TO_DERIVE_TREE_SITTER_QUERY sentinel, or "ERROR: …".

7. Operational Notes

Idempotency: (cveId, source, language, queryHash) is unique. Re-running the same query for the same CVE+language is a free no-op. Captures and predicates are replaced on every successful run so they stay in sync with the most recent model output for that exact query body.
Multiple languages per CVE: a vulnerability that affects both C and C++ produces two rows (one per language). The model is instructed to emit one query per affected language.
Cost: similar token profile to vulnetix.affected (1.5–3 K prompt, 500–2 K completion). The model spends most of its budget reasoning about whether the inputs warrant a query at all — non-derivable cases return the sentinel cheaply.
Latency: 30–90 s per CVE (reasoning-model bound). Sequential per batch in RunBatch — same throughput envelope as affected + attack.
Model swap: just pix-model-set vulnetix.treesitter '<model>' (idempotent; no binary redeploy needed).
Prompt edit: edit prompts/treesitter.md, then just pix-prompts-sync prod. Cache TTL is 5 min so changes propagate within ≤ 5 min.
Disable globally: omit PIX_INFERENCE_ENABLED from a single task’s environment block in terraform/go-schedules.tf. Disable just this activity: UPDATE "Pix" SET enabled = false WHERE "activityKey" = 'vulnetix.treesitter'.
Verification: just go-ghsapoc-test prod already exists for the ghsapoc activity; there is no dedicated treesitter-test cmd, but running any primary processor with PIX_INFERENCE_ENABLED=true against a single CVE produces a PixLog row and (on success) CVETreeSitter* rows for inspection.

S3 Persistence

Not used. This processor does not currently archive payloads or quarantine failures to S3. Per the S3 Persistence Contract this is non-compliant — see the compliance matrix for the implementation roadmap.

⚠ Not in the compliance matrix — status needs verification.

Expected paths when implemented:

Archive: treesitter-activity/files/{sha256}/{filename}
Quarantine: failed-feeds/treesitter-activity/{YYYY-MM-DD}/{reason}/{filename}
Likely reasons: (none documented)