alpine-apk-fetch-processor

Status: Live Source: Alpine package index + aports recipes Type: fetch (APKINDEX gzip-tar download + per-package APKBUILD raw fetch) Source slug: alpine-apk Ecosystem: alpine Schedule: Runs hourly (at minute 15) (cron(15 * * * ? *)).

Overview

This is a package-tracker for the Alpine Linux apk ecosystem — distinct from alpine-json-processor, which imports Alpine CVE advisories. Each run fetches the APKINDEX.tar.gz for edge/main and edge/community (x86_64), parses every package record, dedups by package name (keeping the newest build), and processes the records whose build timestamp (t:) is newer than a stored watermark.

The APKINDEX is a gzip tar containing a text file (APKINDEX) of records separated by blank lines; each record is a set of key:value lines — P: name, V: version, T: description, U: upstream URL, L: license, m: maintainer (Name <email>), o: origin (aports subdirectory), t: build timestamp (unix seconds), c: aports commit hash.

For every candidate package it gathers:

Package metadata — name, version, description, upstream URL, license, maintainer, origin, repo (main/community), build timestamp, commit hash.
APKBUILD build recipe — the package’s APKBUILD (a shell script) is the build/install execution vector, analogous to AUR’s PKGBUILD. It is fetched best-effort from the aports GitLab mirror raw endpoint: https://gitlab.alpinelinux.org/alpine/aports/-/raw/master/<repo>/<origin>/APKBUILD where <repo> is main or community (from the index source) and <origin> is the o: field. The .pre-install/.post-install scripts live inside the binary .apk (not the index) and are intentionally out of scope.
Upstream GitHub repo (only when the package’s U: URL is a github.com project) — repository, contributors, and license, persisted GitHub-first in its own committed transaction.

Records produced

Condition	Records
Every scanned package	`PackageVersion` (ecosystem `alpine`); `updatedAt` = APKINDEX `t:` build timestamp; version/maintainer/origin/repo/commit + non-minting context signals in `metadata` JSON
Upstream is github.com	`GitHubRepository` + `GitHubRepoContributor` + license fields
Malicious (≥1 evidence detection)	`CVEMetadata` (`source="alpine-apk"`, `GCVE-110-APK-YYYY-NNNNNN`, `isMaliciousPackage=true`), one `CVEDescription` per detection, `CVEProblemType` (CWE-506 + specifics), `CVEAffected` (vendor `alpine-apk`, all versions), `CVEMetadataReferences` (pkgs.alpinelinux.org page + APKBUILD + upstream), `PackageVersionCVE`, `GcveIssuance`
Malicious + actor resolved	`ThreatActor` (aports maintainer name + email + APKBUILD-embedded contact emails) + `MalwareThreatActor` edges + `MalwareAttribution` (`attributed`, claimed/victim upstream GitHub)
Malicious	`MalwareIoc` rows (exfil endpoints, IPs, domains, URLs, emails, install-commands, file hashes from the APKBUILD)

Detection

Reuses the shared malscan-engine detect engine. Because the APKBUILD is a shell script, the existing shell / pkgbuild / source-url detectors apply directly — no new language support is required. The APKBUILD text goes into PkgbuildContent; InstallScriptContent is empty (no separate install artifact is fetched). Evidence rules that fire include curl | sh / pipe-to-shell, base64-decode then exec, suspicious source= download URLs, and obfuscated variable concatenation exec. Findings carry one of three classes: evidence (a factual malicious behaviour — mints the advisory on its own), trigger (a weak corroborating signal such as a high-entropy embedded payload or a supply-chain ownership/identity change — never mints alone), and context (reputation/risk, recorded as PackageVersion.metadata only). detect.CombinedVerdict mints on any evidence, a known-bad package owner, a high-entropy payload combined with an ownership/identity change, or two independent identity-change families changing together. The full finding-class model and the per-ecosystem capability config (22 capabilities) live in Malware Detection.

Known-bad IOC matching — the package’s declared source is also matched against the public per-ecosystem known-bad STIX feed (domains, IPs, URLs), loaded once per run and matched in memory. A hit is evidence (CWE-506) folded into the finding set before the combination gate, and is recorded as a MalwareIoc row whose references carry the file/line and STIX provenance.

Gates before a mint

Three independent gates sit between a malicious verdict and an advisory, and all three can only ever remove a detection (audit 2026-08-06):

Gate	Where	Effect
Cross-registry ownership (`hijack.Verdict`)	`main.go:295`	Folds ownership/identity change and owner-known-bad (P4) into the verdict alongside content evidence. Alpine exposes no prior-revision maintainer, so the live ownership signal is owner-known-bad; content evidence (P0) still mints on its own.
LLM false-positive gate (`aimalgate`)	`main.go:306-319`	When `PIX_INFERENCE_ENABLED` + AI-gateway credentials are set, the model is asked whether the package is genuinely malicious; only a confident “benign” drops the verdict (counted as `aiCleared`). Any gateway or parse error fails open, keeping the verdict. Local runs are inference-free by default.
Human curation (`curationgate.PackageCleared`)	`main.go:342`	A package/version a reviewer has already cleared is never re-minted, even if detection fires again.

Every actor candidate additionally passes legitmaintainer.SkipActor (main.go:410) so a legitimate distro maintainer is never linked as a threat actor.

After the walk, malwareactor.PostPass (main.go:148) runs the shared GitHub/registry actor-resolution engine over alpine-apk records that still have no attribution, writing ThreatActor / MalwareThreatActor / MalwareAttribution rows for the ones it can resolve.

S3 persistence

The fetched APKBUILD is archived content-addressed on success (alpine-apk/files/{commit-sha}/{origin}/APKBUILD) and quarantined under failed-feeds/alpine-apk-fetch-processor/{date}/store-error/… when the transaction fails. Both calls are nil-safe, so an unset S3_BUCKET_NAME (local dev) changes nothing else.

Resume

A watermark is stored in BulkDataDumpTracker under source alpine-apkindex (the sha256 column holds the max processed build timestamp t: in unix seconds). The first run (no tracker) starts a window before now (--backfill-hours, default 72) so recent builds are swept; subsequent runs resume from the saved timestamp.

Flags

Flag	Default	Meaning
`--force`	`false`	Reprocess from the backfill window, ignoring the saved watermark
`--limit`	`0`	Max packages to process this run (0 = unlimited)
`--backfill-hours`	`72`	On first run / `--force`, hours before now to start (watermark = now − hours)
`--repos`	`main,community`	Comma-separated APKINDEX repos to walk

S3 Persistence

Archive path: alpine-apk/files/{sha256}/{filename} ✓
Quarantine path: failed-feeds/alpine-apk-fetch-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
Failure reasons emitted: store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[alpine-apk-fetch-processor] PROC -->|success| ARCHIVE[("S3: alpine-apk/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/alpine-apk-fetch-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.