julia-git-processor

Status: Live Source: Julia General registry (git) + GitHub API for upstream deps/build.jl Type: git (runtime go-git clone/fetch of the General registry; no baked data) Source slug: julia Schedule: Runs daily at 02:45 UTC (cron(45 2 * * ? *)).

Overview

Continuously watches the Julia General registry for newly registered package versions and flags malicious publishes. The General registry is itself a git repository whose layout is:

<FirstLetter>/<PackageName>/Package.toml   {name, uuid, repo}
<FirstLetter>/<PackageName>/Versions.toml  ["1.2.3"] -> git-tree-sha1
<FirstLetter>/<PackageName>/Deps.toml
<FirstLetter>/<PackageName>/Compat.toml

Each run clones the registry (first run) or fetch+reset --hardes it to the remote default branch (subsequent runs) via go-git — no registry data is baked into the container image. It then walks the package directories whose Package.toml or Versions.toml changed in (watermark, HEAD] — i.e. new version registrations since the previous run. The first run / --force sweeps the most recent commits (--max-commits, default 300).

The package SOURCE lives in the external repo named by Package.toml’s repo field (usually github.com/owner/Pkg.jl), not in the registry. To stay bounded the processor does not clone every external repo. For each changed package it:

• Records the PackageVersion from Package.toml (name, uuid, repo) + Versions.toml (newest version).
• Fetches Project.toml + deps/build.jl from the upstream github.com repo via the GitHub API (FetchFileAtRef). deps/build.jl runs at Pkg.build — the Julia install/build execution vector — and is the key detection signal.
• Resolves the upstream GitHub repo (repository, contributors, license), persisted GitHub-first in its own committed transaction.

Records produced

Detection

Reuses the shared malscan-engine detect engine. deps/build.jl goes into InstallScriptContent so the install-script detectors fire; Project.toml + build.jl go into PkgbuildContent for the general / shell / source-url detectors. The registry commit that registered the version drives the temporal git-log signals.

Julia-specific malware patterns include run(\…`)/ backtick shell execution inbuild.jl(CWE-78),Base.download(…)followed byinclude/evalof the downloaded artefact andccallinto system libraries (CWE-94/CWE-78), andbase64decode(…)piped intoinclude_string/eval(CWE-94/CWE-506).process/environment exfiltration patterns map to CWE-200. Findings carry one of three classes: evidence(a factual malicious behaviour — mints the advisory on its own),trigger(a weak corroborating signal such as a high-entropy embedded payload or a supply-chain ownership/identity change — never mints alone), andcontext(reputation/risk, recorded asPackageVersion.metadataonly).detect.CombinedVerdictmints on anyevidence`, a known-bad package owner, a high-entropy payload combined with an ownership/identity change, or two independent identity-change families changing together. The full finding-class model and the per-ecosystem capability config (22 capabilities) live in Malware Detection.

Known-bad IOC matching — the package’s declared source is also matched against the public per-ecosystem known-bad STIX feed (domains, IPs, URLs), loaded once per run and matched in memory. A hit is evidence (CWE-506) folded into the finding set before the combination gate, and is recorded as a MalwareIoc row whose references carry the file/line and STIX provenance.

Threat-actor model

Registry commits are frequently automated by the JuliaRegistrator bot, so the registry commit author is only a medium-confidence commit-author signal, and recognised bot identities (JuliaRegistrator, github-actions, [bot]) are skipped entirely — they are infrastructure, not actors. The declared upstream GitHub package repo is recorded as the claimed/victim repo (Source="julia-upstream"), never an actor. Emails embedded in build.jl become payload-contact actors.

Suppression gates

Three independent gates can drop a verdict after the engine has fired, so a finding set is not by itself a mint:

Resume

A commit-SHA watermark is stored in BulkDataDumpTracker under source julia-general-registry (the sha256 column holds the last processed registry HEAD commit SHA). On a resumable run the watermark is diffed against the current HEAD to enumerate only changed packages; the first run (no tracker) and --force sweep recent commits instead. When neither walk yields anything (a depth-limited clone with no usable diff history) the processor enumerates the whole registry tree via julia.TreePackages.

The watermark is written from the current HEAD on a detached context so it survives an overtime cancel — losing it would force the next run to re-sweep the whole registry. The trade-off is that packages left unscanned by --limit, the soft deadline, or a GitHub rate-limit stop are not revisited on the next run; they are only picked up again when their registry files next change. After a truncated run, --force re-sweeps recent commits.

Fetching each package’s deps/build.jl goes through the GitHub API. If a rate limit is hit whose reset falls after this run’s budget, the shared internal/githubapi client returns ErrRateLimitBudget instead of sleeping through it; the processor stops the registry walk cleanly (the watermark is still written from the current HEAD SHA) and the next scheduled run resumes. This prevents a single rate-limited package from sleeping the run into an overtime. Partial per-package failures use notify.Finalize, so a handful of transient errors keep the run green — only a systemic failure pages.

Flags

S3 Persistence

• Archive path: julia/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/julia-git-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.