CERT-IL Fetch Processor Design

Route changed 2026-08-10: advisories now come from the CVE List

The gov.il scrape cannot work from anywhere this fleet runs, and that was established from both sides before changing anything:

• Locally, www.gov.il answers 403 with a Cloudflare interstitial to every client tried — curl’s own User-Agent, Go-http-client, a Chrome User-Agent, and a direct POST to the /api/DynamicCollector endpoint behind the page.
• The deployed ECS task, on Fargate with a public IP, failed identically: three warm-up attempts each ending no advisory items visible after 120s (CF challenge not resolved?). So there is no IP to move to.
• Wayback has snapshots of the listing, but they contain zero ILVN ids — the page is client-rendered, so the archive holds only the shell.

The recovery: the Israel National Cyber Directorate is a CNA (assignerShortName: "INCD"), and every record it assigns carries its advisory id at containers.cna.source.advisory. So the ILVN corpus is available from CVEProject/cvelistV5, which this repo already mirrors for cvelistv5-json-backfill.

Coverage measured against the 175 rows the scrape had produced:

INCD became a CNA around 2025, which is why coverage starts there. The database stopped at ILVN-2025-0258 and cvelistV5 reaches ILVN-2026-0260, so this route resumes exactly where the scrape died — including two advisories we never had.

The CNA record is also better data than the listing ever gave: a real datePublished, a title, an English description, vendor/product/version rows, a CWE and a CVSS vector. The scrape had produced 141 rows with an empty publish_date and an empty description, because it captured listing rows without their detail pages.

--source=browser is kept, not deleted, for the day gov.il stops challenging.

What this route does not recover: the 2023–2024 advisories will never gain real dates or descriptions (they take 1 January of the year in their own identifier, declared as publish_date_derived_from), and any future ILVN advisory INCD does not assign a CVE for will not appear at all.

Overview

Ingests Israeli CERT (CERT-IL) ILVN advisories. Two routes, sharing one store path: --source=cvelist (default) reads INCD’s CNA records from a CVEProject/cvelistV5 checkout; --source=browser drives headless Chromium against https://www.gov.il/en/departments/dynamiccollectors/cve_advisories_listing, which Cloudflare currently blocks. The listing page is English-only (the /he/ URL returns a 404 from the gov.il Angular app); advisory titles describe Israeli software products and CVE types in English.

• Source: cert-il
• Type: fetch — cvelist mode reads a local checkout; browser mode needs headless Chromium (site is behind a Cloudflare managed challenge)
• Primary ID: ILVN-ID (cveId in CVEMetadata)
• Aliases: CVE-IDs stored in CVEAlias (cross-linked when the same CVE exists in NVD or other sources)
• Language: English (lang="en") — the /he/ URL does not exist for this content

Source Characteristics

Architecture

Cloudflare Bypass

The site uses Cloudflare’s managed challenge which blocks all direct HTTP requests (Go net/http, curl, etc.). The processor uses go-rod (a headless Chromium automation library) to:

1. Launch a headless Chromium browser (system binary on Alpine, auto-downloaded locally)
2. Navigate to the listing page — Cloudflare validates the JS challenge automatically
3. Wait up to 30 seconds for the Angular app to render list items
4. Read the rendered DOM HTML for each page

No JSON API is used. The gov.il Angular app’s internal API (/api/content/types/cve_advisories_listing) returns HTTP 403 to programmatic requests even with valid CF cookies.

Pagination Strategy

For skip=0 (the first page), the warm-up navigation has already loaded the page. The processor reads the HTML directly without re-navigating.

For skip=8, skip=16, etc., the processor opens a fresh browser tab (same CF clearance cookies, clean Angular bootstrap), navigates to the paginated URL, waits up to 30 seconds for advisory items to appear, extracts via JS eval, then closes the tab. Reusing the warm-up tab and navigating in-place does not work because Angular SPA intercepts the navigation and data never re-loads.

Pagination stops when a page returns fewer than PageSize (8) items.

Data Flow

EventBridge cron (daily, 06:00 UTC)
  → ECS Fargate task (cert-il-fetch-processor, Alpine + Chromium)
    → launch headless Chromium (go-rod)
    → warmUpPage: navigate to listing, resolve CF challenge (≤30s)
    → loop pages (skip=0, 8, 16, ...):
        FetchListingPage → page.Eval(extractAdvisoriesJS)
        ParseEvalResult → []Advisory (JS DOM extraction)
        MapAdvisory → *CVESourceData (one record per ILVN-ID, CVE-IDs as aliases)
        StoreCVESourceData (per-CVE transaction)
    → notifier.Completed → SNS → Slack (on errors or overtime)

Package Structure

internal/certil/
  types.go    — Advisory struct, Source/BaseURL/ListingPath/PageSize constants
  parser.go   — ParseEvalResult (JS eval JSON parsing), ParsePublishDate
  mapper.go   — MapAdvisory → *osv.CVESourceData (ILVN-ID primary, CVE-IDs as aliases)
  browser.go  — NewBrowser, FetchListingPage (headless Chrome DOM extraction)

cmd/cert-il-fetch-processor/
  main.go     — flags, warm-up, pagination loop, DB writes, error alerting

Key Types

// Advisory — parsed from listing page HTML
type Advisory struct {
    ILVNID                string    // e.g. "ILVN-2025-0258"
    CVEIDs                []string  // e.g. ["CVE-2025-55065"]
    Title                 string    // Hebrew
    Description           string    // Hebrew
    AffectedProducts      []string
    Solution              string
    PublishDate           string    // "DD.MM.YYYY"
    PublishDateUnix       int64
    CreditAcknowledgments string
    SourceURL             string
}

Parsing Strategy

FetchListingPage evaluates extractAdvisoriesJS in the live Angular DOM. The JS:

1. Queries all <li> elements and filters to those whose innerText contains ILVN-\d{4}-\d+
2. Deduplicates by ILVN-ID: picks the innermost <li> (ancestor <li> elements also match)
3. Extracts ILVN-ID, CVE-IDs (normalising Unicode hyphens), and publish date via regex on ASCII text
4. Extracts title from the first heading/link element that is not itself an ID
5. Extracts description from [class*="Descriptional"] element (Angular component class)

Items without an ILVN-ID are skipped.

Mapping Strategy

MapAdvisory returns one CVESourceData record per advisory (keyed by ILVN-ID):

• cveId = ILVN-ID (e.g. ILVN-2025-0258)
• source = cert-il
• title = English advisory title
• Descriptions[0] = English description with lang="en"
• Aliases = CVE-IDs from the same advisory (stored in CVEAlias, cross-linked when CVE exists in other sources)
• rawDataJSON = full advisory fields including ilvn_id, cve_ids, title, description, publish date

Note: CVE-IDs are stored as aliases in CVEAlias (FK to CVEMetadata). The ILVN-ID is the primary record key. rawDataJSON preserves the full advisory for traceability.

Error Alerting

Uses the existing SNS → Lambda → Slack pipeline (internal/notify):

Fatal error conditions that exit immediately:

• Chromium launch failed (binary not found or download failed)
• Warm-up navigation failed (CF challenge not resolved within 30s)
• 3 or more consecutive page fetch failures

Per-item store failures are accumulated via RecordError and reported at completion, but do not halt the run.

Container

The cert-il-fetch-processor Containerfile target uses Alpine Linux (not scratch) because go-rod requires a system Chromium binary for container environments:

FROM alpine:3.21 AS cert-il-fetch-processor
RUN apk add --no-cache chromium
COPY --from=cert-builder /etc/ssl/ /etc/ssl/
COPY --from=builder --chmod=0755 /out/cert-il-fetch-processor /app/cert-il-fetch-processor
ENTRYPOINT ["/app/cert-il-fetch-processor"]

go-rod discovers the binary at /usr/bin/chromium-browser (Alpine’s package name). Locally, if no system Chromium is found, go-rod auto-downloads a compatible Chromium build to ~/.cache/rod/browser/.

ARM64: The ECS task runs on ARM64 (cpuArchitecture: ARM64). Alpine’s chromium package supports ARM64.

Scheduling

Local Usage

# Full history backfill (positional args: TARGET DATE_FROM LIMIT)
just go-cert-il-fetch-backfill prod 1999-01-01 0

# Recent only (last 30 days), limit 20
just go-cert-il-fetch-backfill prod 2025-03-01 20

# Local DB (uses .env)
just go-cert-il-fetch-backfill local 2025-01-01 10

Note: Pass arguments positionally (not TARGET=prod). Just 1.x treats KEY=VALUE strings after the recipe name as positional argument values, not named overrides.

Verification Queries

-- Count stored records
SELECT COUNT(*) FROM "CVEMetadata" WHERE source = 'cert-il';

-- Recent advisories (titles are English)
SELECT "cveId", title, "datePublished"
FROM "CVEMetadata"
WHERE source = 'cert-il'
ORDER BY "datePublished" DESC NULLS LAST
LIMIT 10;

-- Cross-CVE aliases (when advisory contains multiple CVE-IDs)
SELECT "primaryCveId", "aliasCveId"
FROM "CVEAlias"
WHERE "primarySource" = 'cert-il'
LIMIT 10;

-- ILVN-ID from rawDataJSON
SELECT "cveId", "rawDataJSON"->>'ilvn_id' AS ilvn_id
FROM "CVEMetadata"
WHERE source = 'cert-il'
LIMIT 5;

Known Limitations

1. No total count: The DOM approach does not return a total item count. Pagination terminates when a page has fewer than 8 items.
2. Slow pagination: Each page navigation requires Angular to re-render (~10–40 seconds). A full history backfill of hundreds of pages will take significant time.
3. CF challenge may re-trigger: Cloudflare clearance cookies expire. If the challenge re-triggers during a long run, page navigation will fail and the processor will exit after 3 consecutive failures.
4. English-only listing: The advisory listing is English-only (/en/ URL). The Israeli government /he/ URL for this collector returns HTTP 404. Titles and descriptions are stored in English (lang="en").
5. ILVN-IDs not in CVEAlias: The CVEAlias table enforces a FK to CVEMetadata; since ILVN-IDs are not CVE records, they are stored only in rawDataJSON.
6. datePublished is 0 for most rows: ParsePublishDate accepts DD.MM.YYYY and YYYY-MM-DD and returns 0 on anything else, with no fallback (internal/certil/parser.go:53). The DOM extraction frequently yields an empty publish_date, and 141 of the 175 stored rows (81%) consequently carry datePublished = 0 — a NOT NULL integer column, so the zero is indistinguishable from 1970 to every consumer that sorts or filters by it.
7. No CVEAffected and no CVSS: MapAdvisory emits only metadata, description and a self-reference. affectedVendor / affectedProduct / vectorString are NULL for every cert-il row even though the listing exposes an affected-products field on the Advisory struct.

S3 Persistence

• Archive path: cert-il/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/cert-il-fetch-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.