eol-json-processor

Status: Live Source: endoflife.date API v1 Type: json (single full-dataset download, no pagination) Source slug: n/a — writes no CVEMetadata. Tracker source is endoflife_date. Schedule: Runs daily at 03:00 UTC (cron(0 3 * * ? *)).

Overview

Provides the canonical end-of-life dataset behind EOL-risk answers: is this runtime still supported, and how long until this release stops receiving security fixes. The sibling eol-fetch-processor scrapes vendor lifecycle pages for products endoflife.date does not cover; both write the same three tables through the shared internal/eol writer, so consumers never need to know which one supplied a row.

One HTTP GET returns every product with its full release list. Each product is then written in its own transaction, so a single bad product cannot abort the run.

Records produced

Condition	Records
Every product	`EolProduct` (upsert on `name`) — label, category, tags/aliases JSON, version command, EOAS/EOL/EOES/discontinued labels, icon/html/release-policy links, `schemaVersion`, `lastModified`, `fetchedAt`
Every product	`EolRelease` — delete-then-reinsert for the product: one row per release cycle with `isLts`/`isEoas`/`isEol`/`isEoes`/`isDiscontinued`/`isMaintained` plus their `*From` dates, latest version/date/link, and a `customJSON` blob
Every product	`EolIdentifier` — delete-then-reinsert: one row per `(type, identifier)` mapping (purl, cpe, repology, …)
Every run	`BulkDataDumpTracker` row `endoflife_date` (`sha256` written empty — freshness here is time-based, not content-based)
Every stored product	S3 archive (see § S3 Persistence)

EolRelease and EolIdentifier use delete-then-reinsert because the upstream feed is the source of truth for the whole set — a release removed upstream must disappear locally. The side effect is that EolRelease.createdAt is reset on every run and cannot be used as a “first seen” signal.

Freshness gate

Before fetching, the processor reads BulkDataDumpTracker.endoflife_date and skips the run with task.no_work when now - lastProcessedAt < frequency × 1000.

⚠ Known defect — the daily run skips every second day. db.UpsertTracker hardcodes frequency = 86400 (24 h) and lastProcessedAt is written at the end of a successful run, i.e. at cron-time + runtime. The next day’s 03:00 fire therefore sees elapsed ≈ 86400000 − runtime < 86400000 and exits “data is fresh” without fetching. Observed in production: the live tracker row’s lastProcessedAt is 03:01:26 and EolProduct.fetchedAt advances only on alternate days. The freshness window must be made strictly shorter than the schedule interval (or dropped in favour of the feed’s own lastModified).

Failure modes

Symptom	Cause
`data is fresh, skipping` on a day you expected work	The freshness defect above, or a genuine re-run within 24 h
`parse failed` + exit 1	The full-product JSON did not unmarshal; the raw body is quarantined as `parse-error`
`upsert product failed` per product	Per-product transaction failed; the product is quarantined as `store-error`, the run continues, and the run exits 1 at the end
`soft deadline reached, stopping early`	See the deadline note below

Deadline

The soft deadline is derived from EXPECTED_DURATION_MINUTES − 10, but it falls back to a hardcoded 30 minutes when the env var is unset — which is exactly the state a local just go-eol-json-backfill creates (the recipe unsets it). A local backfill is therefore silently bounded at 30 minutes, contrary to the backfill-has-no-deadline rule. At ~480 products the bound is not currently reached, but the pattern is wrong; sibling processors such as chainguard-json-processor and cran-json-processor leave the deadline at its zero value when the env var is absent.

Local execution

just go-eol-json-backfill          # local DB
just go-eol-json-backfill prod     # .env.production

S3 Persistence

Archive path: eol/files/{sha256}/{filename} ✓
Quarantine path: failed-feeds/eol-json-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
Failure reasons emitted: parse-error, store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

flowchart LR SRC[Source feed] --> PROC[eol-json-processor] PROC -->|success| ARCHIVE[("S3: eol/files/{sha256}/{filename}")] PROC -->|failure| Q[("S3: failed-feeds/eol-json-processor/{date}/{reason}/{filename}")] PROC --> DB[(PostgreSQL)]

See the S3 Persistence Contract for the full reason taxonomy.