EOL Vendor Fetch Processor — Design

Status: Live Source: vendor product-lifecycle HTML pages — Virtuozzo, Scientific Linux Type: fetch (HTML scraping, one scraper per vendor) Source slug: eol (archive prefix); tracker row eol_vendor_fetch Schedule: Runs weekly on Mondays at 05:00 UTC (cron(0 5 ? * MON *)).

Overview

Scrapes vendor product-lifecycle / end-of-life pages that endoflife.date does not carry and upserts them through the same eol.UpsertProduct write path the endoflife.date processor uses, so a consumer cannot tell where a lifecycle row came from. This is the sibling of eol-json-processor for vendors absent from that catalogue; without it those products have no EOL record at all, and the eolStatus fields on /api/vdb/v2/packages/* and the CLI’s EOL gate silently treat them as supported.

Each vendor is a eolvendor.Scraper in eolvendor.Registry (internal/eolvendor/registry.go:29) — adding a vendor is one new file. Rows written by this processor are tagged EolProduct.schemaVersion = "vendor-scrape-1" so they stay distinguishable from endoflife.date rows.

Product identity (slug / label / category / aliases / tags) is pinned in code, not read from the page, so a vendor relabelling a table row cannot fork the product into a second slug. Only the volatile lifecycle dates come from the HTML. Virtuozzo rows whose EOL cell carries a * are marked custom.eolTentative with the vendor’s own caveat text; unmapped product rows are skipped with a warning rather than guessed at.

Records produced

Table	Rows
`EolProduct`	One per scraped product, keyed on `name` (the pinned slug); `schemaVersion='vendor-scrape-1'`, `linkHtml` = the vendor page, `fetchedAt` = run start (ms)
`EolRelease`	One per lifecycle row — `releaseDate` (GA), `eoasFrom` (EOM), `eolFrom` (EOL), derived `isEol`/`isEoas` flags. Deleted and re-inserted per product on every run (upstream is the source of truth)
`EolIdentifier`	Supported by the shared write path, but neither registered scraper emits identifiers, so this table gets zero rows from this processor (prod: 15 vendor-scraped products, 0 identifiers)

Freshness gate

The processor gates itself on its own weekly cadence: it reads BulkDataDumpTracker row eol_vendor_fetch and exits NoWork when now - lastProcessedAt < 604800s. It deliberately does not read the tracker’s own frequency column, because db.UpsertTracker hard-codes that to daily. --force bypasses the gate.

Failure modes

A vendor scraper that returns an error is logged, counted, and skipped — the other vendors still run; the process exits 1 at the end if any vendor or any product upsert failed.
A page whose structure changed yields zero parsed rows, which the scraper turns into an explicit error ("no product rows parsed (page structure changed?)") rather than silently writing nothing.
The tracker is written before the error-exit check, so one bad vendor cannot force a full re-scrape next week.
⚠ A 15-minute deadline applies to local backfills too. softDeadlineDuration defaults to 15 minutes unconditionally (main.go:56) and is only overridden by EXPECTED_DURATION_MINUTES; the justfile’s unset EXPECTED_DURATION_MINUTES therefore does not remove the cap, contrary to the backfill rule in AGENTS.md. Harmless at today’s two-vendor scale, wrong in principle.

Flags

Flag	Default	Description
`--force`	false	Bypass the weekly freshness gate and scrape now

ECS Schedule

Runs weekly on Mondays at 05:00 UTC (cron(0 5 ? * MON *)); 256 CPU / 512 MB, expected_duration_minutes = 15.

CloudWatch logs retain 3 days, then archive to S3 (Glacier Deep Archive).

S3 Persistence

Archive path: eol/files/{sha256}/{filename} ✓
Quarantine path: failed-feeds/eol-fetch-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
Failure reasons emitted: store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[eol-fetch-processor] PROC -->|success| ARCHIVE[("S3: eol/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/eol-fetch-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.