drupal-git-processor

Status: Live Source: DrupalSecurityTeam/drupal-advisory-database (git) Type: git (repo baked into the image by the drupal-git-data stage, then pulled at runtime with pure-Go go-git) Source slug: drupal Schedule: Runs weekly on Tuesdays at 04:00 UTC (cron(0 4 ? * TUE *)). CPU 256 / memory 512 MB / expected_duration_minutes = 45.

Overview

Drupal’s security team publishes DRUPAL-SA-* advisories covering core and the contrib module ecosystem. Many of those advisories never receive a CVE, and the ones that do are usually filed weeks later — so this is the only feed that lets a Drupal site’s composer.lock be matched against a Drupal-numbered advisory on the day it lands.

Each run:

  1. Refreshes the repo with processor.PullOrClone against /data/drupal-advisory-database (baked at build time, .baked-sha recorded). The final image stage is scratch, so the pull goes through go-git rather than a git binary.
  2. Short-circuits on an unchanged HEAD by comparing HEAD to BulkDataDumpTracker.sha256 for source drupal_advisory; on a match without --force the run reports task.nowork and exits 0.
  3. Narrows to changed files — with a tracker present, processor.DetectChangedFiles limits the walk to paths touched in the last 3 days. First run / --force / un-diffable history → full scan.
  4. Walks advisories/ for files matching DRUPAL-*.json.
  5. Maps and stores through the shared pipeline in transactions of --batch-size (default 200), one SAVEPOINT per file.

osv.MapAdvisory keys each row on the first CVE-YYYY-NNNN+ alias when the advisory has one and on the DRUPAL-SA-* ID otherwise, so a Drupal-only advisory keeps its native identifier and a CVE-cross-referenced one merges with the CVE row from other sources. OSV malicious-package records are skipped.

Records produced

ConditionRecords
Every stored advisoryCVEMetadata (source="drupal", sourceFileHash = file SHA1), CVEDescription, CVEMetadataReferences (referenceSource="Drupal"), CVEProblemType (upstream cwe_ids), CVEAffected + CVEAffectedVersion (Drupal advisories carry Packagist package names)
Advisory carries a CVSS vectorCVEMetric per upstream severity entry
Advisory carries descriptionsone derived CVEMetric (containerType="vulnetix", cvssV4_0)
AlwaysCVEAlias edges via db.InsertAliases (OSV ID, aliases[], related[], upstream[], plus same-cveId cross-source edges)
Every affected packageDependency, DependencyRegistry, PackageVersion, PackageVersionCVE (+ GitHubRepoDependency when a GitHub repo is named) via db.EnrichAffectedWithDependency
Clean runBulkDataDumpTracker row drupal_advisory

Upstream Drupal advisories carry no CVSS vectors, so in practice the only metric on these rows is the Vulnetix-derived v4 vector.

Resume

  • Repo-level — drupal_advisory tracker SHA.
  • Per-file — db.LoadProcessedHashes("drupal") skip set on CVEMetadata.sourceFileHash.

The tracker is skipped when any file errored or the context was cancelled, so partial runs are retried rather than silently dropped. --force bypasses both layers.

AI enrichment

The pipeline is wired to aienrich, so ECS runs add the shared per-record passes (affected routines, ATT&CK, CWE inference, TreeSitter) after each batch commits. Local just runs are inference-free unless AIENRICH=true.

Failure modes

SymptomCause
repository unchanged, skipping processingtracker SHA equals HEAD — normal for a quiet week
failed to load resume hashesread-replica timeout; run continues without resume and re-upserts
soft deadline reached, stopping earlyEXPECTED_DURATION_MINUTES − 10 elapsed
files errored > 0tracker untouched, next run retries; pages only if notify.Finalize judges it systemic

Flags

FlagDefaultMeaning
--forcefalseIgnore the tracker SHA and the per-file hash skip set
--batch-size200Files per transaction
--repo/data/drupal-advisory-databaseClone location
--data-dir(repo + /advisories)Override the advisory directory

Local run: just go-drupal-git-backfill (append prod to target production).

S3 Persistence

  • Archive path: drupal/files/{sha256}/{filename} ✓
  • Quarantine path: failed-feeds/drupal-git-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
  • Failure reasons emitted: parse-error, store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[drupal-git-processor] PROC -->|success| ARCHIVE[("S3: drupal/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/drupal-git-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.