conan-git-processor

Status: Live Source: conan-center-index (git) Type: git (runtime clone/pull via go-git — no system git binary, no baked data stage) Source slug: conan Schedule: Runs daily at 02:15 UTC (cron(15 2 * * ? *)).

Overview

Continuously watches the ConanCenter index — the upstream git repository of C/C++ recipes — for new and updated recipes and flags malicious recipe changes.

The index is cloned (shallow, depth 1) or, on a persistent --repo-dir, pulled at runtime using pure-Go git (go-git), so the container’s final stage can be scratch (CA certs only, no git binary and no baked advisory data). Each run captures the HEAD commit SHA, then:

Incremental (watermark present) — walks commits from HEAD back to the saved watermark SHA and collects the set of recipe names whose files changed (recipes/<name>/.../conanfile.py, config.yml, conandata.yml).
First run / --force — walks the most recent --max-commits commits (default 300) and collects the touched recipe names, newest first, bounded by --limit.
Shallow clone with no diff history — when neither walk yields anything (conan.RecentRecipes returns empty on a depth-1 clone), the processor falls back to conan.TreeRecipes and enumerates every recipe in the working tree.

For each candidate recipe it reads recipes/<name>/config.yml to resolve the newest version and its folder, then reads the recipe bundle:

conanfile.py — the recipe Python that runs at conan install / build time (the supply-chain execution vector). This is fed to the install-script detectors.
conandata.yml + config.yml — declared source URLs / sha256 and the version→folder map. Concatenated with conanfile.py and fed to the general / shell / source-url detectors and IOC extraction.
Upstream GitHub repo (only when the recipe’s homepage = / url = is a github.com project) — repository, contributors, and license, persisted GitHub-first in its own committed transaction.

Records produced

Condition	Records
Every scanned recipe	`PackageVersion` (ecosystem `conan`); `updatedAt` = recipe-changing commit timestamp; newest version/folder/license/homepage + commit author + non-minting context signals in `metadata` JSON; `discoveredFrom = conan-center-index`
Upstream is github.com	`GitHubRepository` + `GitHubRepoContributor` + license fields
Malicious (≥1 evidence detection)	`CVEMetadata` (`source="conan"`, `GCVE-110-CONAN-YYYY-NNNNNN`, `isMaliciousPackage=true`), one `CVEDescription` per detection, `CVEProblemType` (CWE-506 + specifics), `CVEAffected` (vendor `conan`, all versions), `CVEMetadataReferences` (conan center page + repo + homepage), `PackageVersionCVE`, `GcveIssuance`
Malicious + actor resolved	`ThreatActor` (the git author of the recipe-changing commit + embedded contact emails) + `MalwareThreatActor` edges + `MalwareAttribution` (`attributed`, claimed/victim upstream GitHub)
Malicious	`MalwareIoc` rows (exfil endpoints, IPs, domains, URLs, emails, install-commands, file hashes)

Detection

Reuses the shared malscan-engine detect engine. The conanfile.py goes into the install-script detectors (it is the execution vector); the full recipe bundle goes into the general / shell / source-url detectors. Conan/Python-specific evidence targets self.run(...) shell execution, os.system / subprocess invocation, tools.download + unzip of off-recipe URLs, and exec/eval of base64-decoded payloads. Known-bad artifact hashes are checked against the shared MalwareIoc hash set. Because conanfile.py lives inside the trusted, reviewed conan-center-index monorepo, a malicious recipe change is attributed to the git author of the commit that introduced it. Findings carry one of three classes: evidence (a factual malicious behaviour — mints the advisory on its own), trigger (a weak corroborating signal such as a high-entropy embedded payload or a supply-chain ownership/identity change — never mints alone), and context (reputation/risk, recorded as PackageVersion.metadata only). detect.CombinedVerdict mints on any evidence, a known-bad package owner, a high-entropy payload combined with an ownership/identity change, or two independent identity-change families changing together. The full finding-class model and the per-ecosystem capability config (22 capabilities) live in Malware Detection.

Known-bad IOC matching — the package’s declared source is also matched against the public per-ecosystem known-bad STIX feed (domains, IPs, URLs), loaded once per run and matched in memory. A hit is evidence (CWE-506) folded into the finding set before the combination gate, and is recorded as a MalwareIoc row whose references carry the file/line and STIX provenance.

Suppression gates

Three independent gates can drop a verdict after the engine has fired, so a finding set is not by itself a mint:

Gate	Effect
`internal/legitmaintainer`	actor candidates matching a known-legitimate maintainer identity are never linked as threat actors
`internal/curationgate`	a `(ecosystem, package, version)` explicitly cleared by human curation flips `isMal` back to false inside the store transaction — no advisory is minted
`internal/aimalgate` (`GO_MALWARE_GATE_MODEL`)	before minting, the LLM gate is asked whether the evidence is genuinely malicious. Only a confident benign answer drops the verdict; any gateway or parse error fails open, so the gate can only remove false positives. Cleared records are counted in the `aiCleared` stat

Resume

The HEAD commit SHA is stored in BulkDataDumpTracker under source conan-center-index (the sha256 column holds the last processed commit SHA). The first run (no watermark) walks recent commits so recent activity is swept; subsequent runs diff watermark..HEAD.

The tracker is written from the current HEAD unconditionally at the end of the run, including when --limit or the soft deadline cut the walk short. Recipes that were in the candidate set but never reached are therefore not revisited on the next run — they are only picked up again if one of their files changes. After a truncated run, re-run with --force to sweep recent commits again.

Flags

Flag	Default	Meaning
`--force`	`false`	Reprocess the most-recent recipes, ignoring the saved commit watermark
`--limit`	`0`	Max recipes to process this run (0 = unlimited)
`--repo-dir`	temp dir	Persistent index repo directory (pulls instead of re-cloning)
`--max-commits`	`300`	On first run / `--force`, how many recent commits to scan for touched recipes

S3 Persistence

Archive path: conan/files/{sha256}/{filename} ✓
Quarantine path: failed-feeds/conan-git-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
Failure reasons emitted: store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[conan-git-processor] PROC -->|success| ARCHIVE[("S3: conan/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/conan-git-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.