github-enrichment Design

Populates the GitHub* tables for GitHubRepository rows queued with needsEnrichment=true, via the GitHub REST API (go-github v68, GITHUB_PAT).

Overview

  • Batch size: 10 repos per run (GITHUB_ENRICH_BATCH, max 100)
  • Schedule: Every 15 minutes (cron(*/15 * * * ? *)), schedule/task-def name go-github-enrichment-processor
  • Concurrency: None needed — the claim (bump lastSyncedAt) plus oldest-first ordering keeps a straggling sibling off the same rows; zombieguard kills stale tasks
  • State: The queue IS the table — GitHubRepository.needsEnrichment = true ordered by lastSyncedAt ASC. No BulkDataDumpTracker rows.

Queue Sources

GitHubRepository rows are queued (needsEnrichment=true) by:

  • vdb-api /v2/cli.sca — upserts a placeholder row from a scan’s GitHub git remote (keyed by fullName, autoincrement id), re-armed when enrichedAt is >24h stale. vdb-api also runs a shallow inline goroutine (repo + languages + branches); this processor is the deep/catch-up pass.
  • enrich-googleosi — CreateOrUpdateGitHubRepository inserts rows with GitHub’s numeric id when a deps.dev SOURCE_REPO link points at github.com.

Processing Flow

  1. Select oldest N queued repos (needsEnrichment=true ORDER BY "lastSyncedAt" ASC), claim them by bumping lastSyncedAt
  2. Per repo, EnrichGitHubRepoFull (internal/enrichment/github_repo_enrichment.go):
    • GET /repos/{owner}/{repo} → UPDATE "GitHubRepository" ... WHERE "fullName"=$1 (nodeId, counts, license, topics, URLs, timestamps; sets enrichedAt, clears needsEnrichment/enrichmentError). Not CreateOrUpdateGitHubRepository — its ON CONFLICT ("id") insert violates the fullName unique for rows created with autoincrement ids.
    • Owner → upsert GitHubUser or GitHubOrganization (by login), link userId/organizationId
    • Branches → GitHubBranch + GitHubRepositoryBranch junction; contributors → GitHubRepoContributor; languages → GitHubRepoLanguage; UpsertOpenSSFScorecard via api.scorecard.dev
    • Releases (fully paginated, page cap 20) → GitHubRelease + embedded assets → GitHubReleaseAsset
    • Issues (1 page of 100, newest, PRs filtered out) → GitHubIssue + embedded labels → GitHubIssueLabel; repo-level comments (1 page) → GitHubIssueComment
    • PRs (1 page of 100) → GitHubPullRequest; reviews for the latest 20 PRs → GitHubPullRequestReview
    • Workflows + runs (1 page each) → GitHubWorkflow, GitHubWorkflowRun
    • Check suites on the default-branch head → GitHubCheckSuite; check runs for up to 5 suites → GitHubCheckRun
    • Deployments (1 page) → GitHubDeployment; statuses for the latest 20 → GitHubDeploymentStatus
  3. Sub-resources are best-effort (logged, never abort the repo); only a failed core repo fetch fails the repo
  4. Rate-limit guard: batch stops early when the PAT’s remaining requests drop below 500

Failure Policy

  • Permanent (HTTP 404 / 410 / 451 — private, deleted, DMCA-blocked): needsEnrichment=false + enrichmentError recorded — dropped from the queue
  • Transient (403 rate/policy, 5xx, network): enrichmentError recorded, flag stays true; the claim already bumped lastSyncedAt, so the row rotates to the back of the queue

Skipped (admin/push scope required)

GitHubCollaborator, GitHubWebhook, secret-scanning / Dependabot / CodeQL alerts — a public-repo PAT receives 403 for these.

Request Budget

≈35–60 GitHub API calls per repo → ~600/cycle at N=10, ~2,400/hour at 4 cycles — under the 5,000/hour PAT limit, with the rate floor as backstop. The GITHUB_PAT is shared with other GitHub-calling processors via Secrets Manager (go_task_secrets_github).

Tables Written

  • GitHubRepository — core metadata, enrichedAt, userId/organizationId links
  • GitHubUser / GitHubOrganization — repo owner
  • GitHubBranch, GitHubRepositoryBranch — branches + junction
  • GitHubRepoContributor, GitHubRepoLanguage — contributors, language bytes
  • OpenSSFScorecard — scorecard via api.scorecard.dev
  • GitHubRelease, GitHubReleaseAsset — releases (full history) + assets
  • GitHubIssue, GitHubIssueLabel, GitHubIssueComment — issues
  • GitHubPullRequest, GitHubPullRequestReview — PRs + reviews
  • GitHubWorkflow, GitHubWorkflowRun — Actions workflows + runs
  • GitHubCheckSuite, GitHubCheckRun — checks on the default-branch head
  • GitHubDeployment, GitHubDeploymentStatus — deployments

Schema Note

GitHub’s modern entity ids exceed int4 (issue ids >4B, workflow-run ids >27B, check-suite ids >66B). The saas migration 20260610000002_github_activity_ids_bigint widened the activity-id columns (and ScannerRun.workflowRunId/checkRunId) to bigint. GitHubRepository.id remains int4 — revisit when GitHub repo ids approach 2³¹.

S3 Persistence

Not used. This processor does not currently archive payloads or quarantine failures to S3. Per the S3 Persistence Contract this is non-compliant — see the compliance matrix for the implementation roadmap.

⚠ Not in the compliance matrix — status needs verification.

Expected paths when implemented:

  • Archive: github-enrichment/files/{sha256}/{filename}
  • Quarantine: failed-feeds/github-enrichment/{YYYY-MM-DD}/{reason}/{filename}
  • Likely reasons: (none documented)