Design: Akamai Fetch Processor

Overview

Fetches Akamai PSIRT advisories published as blog posts at www.akamai.com/blog/security/advisory-cve-* and www.akamai.com/blog/security-research/advisory-cve-* and stores the parsed vulnerability data as CVEMetadata.source = "akamai".

Source identifier: akamai Data type: fetch ECS task name: go-akamai-fetch-processor Schedule: Runs weekly on Sundays at 05:00 UTC (cron(0 5 ? * SUN *)).

Why not a CSV / RSS / sitemap processor? Akamai’s edge blocks every listing surface (CSV index, sitemap.xml, robots.txt, /blog/security, /blog/feed, /api/blog/posts) with HTTP 403 Access Denied at the URL-pattern level — not via the Bot Manager, but via a hard edge ACL. Only specific per-advisory blog URLs return 200, and only if you already know the slug. This processor therefore operates over a maintained seed list and grows it via related-advisory link extraction.

How the anti-bot gate is handled

Akamai Bot Manager (the same product Akamai sells to its customers) sits in front of every page on www.akamai.com. The challenge fires at the TLS layer — Go’s stdlib crypto/tls ClientHello is fingerprinted as non-browser and returns 403 regardless of User-Agent.

This processor uses httpclient.NewChromeImpersonating (see internal/httpclient/utls.go), which performs a Chrome 120 ClientHello via refraction-networking/utls. Combined with:

• a cookiejar.Jar (the edge sets a challenge cookie on first contact that subsequent requests must echo)
• sec-ch-ua* / sec-fetch-* Client-Hints headers via httpclient.SetBrowserFetchHeaders
• sec-fetch-site=none on cold landings, same-origin + Referer on subsequent in-session navigations
• 6-second pacing between fetches

…this defeats the gate reliably. The same client pattern is applicable to any WAF that does TLS-fingerprint gating (Cloudflare Bot Fight Mode, PerimeterX, DataDome).

Data Source

Per-advisory page structure

Every page exposes a JSON-LD <script type="application/ld+json"> block containing a BlogPosting:

{
  "@type": "BlogPosting",
  "@id": "https://www.akamai.com/content/akamai/en/blog/security/2025/dec/advisory-cve-2025-53841-...",
  "mainEntityOfPage": "...",
  "headline": "CVE-2025-53841: Guardicore Local Privilege Escalation Vulnerability",
  "description": "Get technical details about a security vulnerability...",
  "datePublished": "2025-12-09T10:00:00Z",
  "dateModified": "2025-12-09T20:38:57Z"
}

CVSS vectors are not consistently exposed in the HTML — Akamai posts often state only a textual severity (“Medium impact”). The parser captures a vector + numeric score when present and otherwise emits no CVSS metric row rather than synthesise one.

Architecture

Data Mapping

CVEMetadata (one row per advisory)

CVEAlias

• Every additional CVE-* ID extracted from the page beyond the primary.
• The Akamai slug itself as akamai-{slug} (e.g. akamai-advisory-cve-2025-53841-guardicore-local-privilege-escalation). Because this is a non-CVE identifier, db.InsertAliases also mints a placeholder CVEMetadata row for it under source='akamai' so the alias edge has an FK target. Those placeholder rows are content-free by design — exclude them from any field-coverage measurement of this source.

Aliases flow through osv.CVESourceData.Aliases → processor.StoreCVESourceData → db.InsertAliases, which writes canonical edges and runs same-cveId cross-source backfill.

CVE harvesting is page-wide. ParseAdvisory collects every CVE-\d{4}-\d{4,} match in the title and then every match anywhere in the document, in that order. Practically this means: the primary cveId is the title’s CVE (the slug shape advisory-cve-YYYY-NNNN-* guarantees one is present on real PSIRT posts), and any CVE merely mentioned in the body would become an alias. That fan-out is neutralised by db.InsertAliases’ bundle suppression — when the primary id is CVE-prefixed and more than one CVE alias is present, all CVE aliases are dropped — so no CVE↔CVE false edges reach CVEAlias. The akamai-{slug} alias is non-CVE and always survives.

CVEDescription

Single row: containerType=cna, lang=en, value= JSON-LD description (Akamai’s SEO meta blurb — short but accurate).

CVEMetadataReferences

One row keyed referenceSource = "akamai", type = "advisory", url = sourceAdvisoryRef. Additional vendor/patch references are not present in the source HTML in a stable location and so are not synthesised.

CVEMetric

Emitted only when a CVSS vector string is detected in the page. The score is best-effort (regex over the body); severity falls back to a textual Critical|High|Medium|Low token when found. Most Akamai posts produce no metric row.

CVEAffected

Not currently emitted. Akamai advisories describe vendor product families (Guardicore Agent, App & API Protector policy, etc.) rather than versioned packages; mapping these to PURL/CPE would be guesswork. The affectedProduct column captures the family for filtering.

Discovery: seed list + crawl growth

The processor starts each run with internal/akamai/seeds.go::SeedSlugs and pushes the queue forward by extracting every href="...advisory-cve-..." href from each successfully parsed page. New slugs (not previously seen this run) are appended to the queue and fetched in the same execution, subject to --limit and the soft deadline.

Persisting newly discovered slugs across runs is intentionally not implemented — committing dynamic state to seeds.go from a scheduled task would require a self-PR loop. Instead, the in-run growth keeps coverage current within a single execution; out-of-band PRs to seeds.go extend the persistent baseline.

Incremental Strategy

Two-tier resume:

--force bypasses the per-page resume and refetches everything. There is no feed-level SHA to compare against (no canonical index exists), so the tracker is informational only.

Flags

Decision tree

Files

Local Development

# Verify the Bot Manager gate is passable from this network — no DB writes
just go-akamai-fetch-probe

# Full backfill against local PostgreSQL (.env)
just go-akamai-fetch-backfill TARGET=local LIMIT=5

# Force refetch of all seeds even if sourceFileHash matches
just go-akamai-fetch-backfill TARGET=local FORCE=true

# Production (.env.production)
just go-akamai-fetch-backfill TARGET=prod

# Inspect upserted rows
psql "$DATABASE_URL" -c "SELECT \"cveId\", \"title\", \"dateUpdated\" FROM \"CVEMetadata\" WHERE source = 'akamai' ORDER BY \"dateUpdated\" DESC LIMIT 10;"

Deployment

• Containerfile target: akamai-fetch-processor (scratch + certs + binary).
• Terraform: module "akamai_fetch_processor" in terraform/go-schedules.tf (cpu=256, memory=512, weekly Sundays at 05:00 UTC).
• ECR tag: go-akamai-fetch-processor-${tag}.
• Hook: listed in .claude/hooks/post-push-ecr.sh::TARGETS.
• Dashboard: work-stats case in scripts/task-dashboard/db.go counts CVEMetadata WHERE source = 'akamai'.

Known Limitations

• Discovery is bounded by the seed list and per-run crawl. Advisories Akamai publishes that are not linked from any seeded page will not be picked up automatically. Extend seeds.go to cover new product families.
• Resume is approximate. sourceFileHash is computed over {slug, title, description, datePublished, dateModified, CVEs} — the only stable bits across fetches, since Akamai injects per-request nonces into the HTML and rotates the “Latest blog posts” sidebar. On a tiny feed (2–15 advisories/year) this is acceptable, but in practice the LoadProcessedHashes read-replica lookup often returns stale data and a fresh UPSERT happens anyway. The UPSERT is idempotent at the DB level, so the cost is one wasted write per advisory per run — negligible at this scale. Re-evaluate if seed coverage ever expands past ~100 advisories.
• CVSS data is rare. Akamai posts cite a severity word but seldom an AV vector. The processor emits no metric row rather than fabricate one. Downstream consumers that require CVSS should derive from the description via the standard cvss.DeriveV4FromDescription pipeline (enabled by default).
• No CVEAffected rows. Versioned product mapping is left to first-party Akamai catalogs that don’t currently exist publicly. The affectedVendor / affectedProduct columns are sufficient for filtering by family.
• Bot Manager is a moving target. If a future Akamai edge-config update fingerprints HelloChrome_120 specifically, swap utls.HelloChrome_120 for HelloChrome_Auto or a newer pinned profile in internal/httpclient/utls.go.
• Linode coverage. Akamai’s PSIRT now covers post-acquisition Linode advisories under the same blog surface; this processor tags them via productFromTitle() → affectedProduct = "Linode". See Linode (Superseded by Akamai) for the inheritance rationale and the SQL to query Linode-tagged rows. Add new Linode-specific advisory slugs to seeds.go as they are discovered.

S3 Persistence

• Archive path: akamai/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/akamai-fetch-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: fetch-error, parse-error, store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.