Design: ServiceNow KB Fetch Processor

Overview

Fetches ServiceNow PSIRT (Product Security Incident Response Team) advisories from the master “ServiceNow Common Vulnerabilities & Exposures (CVE) Security Advisories” KB landing page (KB1226057) plus per-advisory KB articles, parses CVE references and release-family fix matrices, stores per-advisory rows under source="servicenow", and (when --emit-crit) stages CRIT candidate envelopes for Now Platform / AI Platform / ITSM / CSM / HR / Security Operations / etc.

Source identifier: servicenow Data type: fetch (HTML scraping; ServiceNow does not publish RSS, JSON API, or CSAF for advisories) ECS task name: go-servicenow-kb-fetch-processor Schedule: Runs weekly on Fridays at 06:00 UTC (cron(0 6 ? * FRI *)). Pattern: A — inline per-record CRIT staging Phase: 1.8

⚠ CRIT staging is inert in production. --emit-crit defaults to false (main.go:56) and the ECS command in terraform/go-schedules.tf:4065 is ["/app/servicenow-kb-fetch-processor"] with no flags, so every scheduled run takes the emitCrit == false path. Everything below under Service Mapping, CRIT Envelope Shape and the S3QueueObject / CritRecord sections only executes for a local just go-servicenow-kb-fetch-backfill (whose recipe defaults EMIT_CRIT=true). scripts/task-manager.toml also declares emit-crit default true, which does not reach the container.

Data Source

Landing-page shape

The master KB1226057 page is server-rendered (AngularJS portal shell with the article body inlined as JSON-LD plus inline anchors). The advisory table is baked into the initial HTML payload — no JS execution required.

<a ... href="/kb?id=kb_article_view&sysparm_article=KB2693566" ...>KB2693566</a>

URL escaping — ServiceNow renders & as & and = as = in some contexts. The extractor’s regex normalises both.

Per-KB article shape

<title>[Security Advisory] CVE-2026-0542 - Remote Code Execution in ServiceNow AI Platform - Security - Now Support Portal</title>
<script type="application/ld+json">
  {"@type":"Article","datePublished":"2026-02-25","dateModified":"2026-02-25","articleBody":"..."}
</script>
<!-- body with affected release-family table:
       Australia | Patch X | Date
       Zurich    | Patch Y Hotfix Zb | Date
       Yokohama  | Patch X | Date
-->

Release-family names ServiceNow uses (release vehicles): Australia, Zurich, Yokohama, Xanadu, Washington, Vancouver, Utah, Tokyo, San Diego, Rome, Quebec, Paris, Orlando, New York, Madrid, London, Kingston, Jakarta.

Architecture / Data Flow

Source → Database Mappings

CVEMetadata (one row per advisory)

CVEAlias

CVE-* and GHSA-* extracted from body via regex. Passed through db.InsertAliases(ctx, tx, kbID, "servicenow", aliases, logger) per AGENTS.md alias-write contract:

• The helper canonicalises edge direction — CVE IDs become primaryCveId and KB IDs become aliasCveId because CVEs are the canonical primary.
• Empty alias list still calls InsertAliases so the same-cveId cross-source backfill runs on every store.
• One edge per source-pair (cve.org, nist-nvd, vulncheck-nvd, circl, etc.) per the helper’s enumeration.

CVEMetadataReferences

Every URL extracted from body text via https?://[^\s<>"')]+ regex, after the deny-host filter. Reference type classification:

• i==0 AND u==KB-URL → advisory
• contains cve.org / cve.mitre.org / /security/advisories/ghsa- / nvd.nist.gov/vuln/detail/cve- → advisory
• contains /releases/tag/ / /patch → patch
• everything else → web

CVEDescription

• Body text truncated to 1500 chars → one row with containerType="cna", lang="en".
• When the resolution paragraph (<b>Resolution:</b>/<b>Mitigation:</b>/<b>What you should do:</b>) is present → second row prefixed with "Resolution: ".

S3 archive

Per-advisory payload written to:

servicenow/files/{sha256}/{kbID}.json

The archive contains the ParsedAdvisory struct serialised to JSON: title, body text, resolution text, CVE/GHSA lists, ref URLs, release families, CVSS score, published date.

S3QueueObject

One row per staged CRIT envelope with bucket / key / source="servicenow-kb-fetch-processor" / processingStatus. Lifecycle:

• pending (after RegisterS3QueueObject)
• inserted (after publisher upserts CritRecord and moves S3 object to approved/inserted/)
• rejected (when revalidation fails)

Service Mapping

Bracketed product hints from title + body resolve to spec/extended dict tuples. Order matters — more-specific synonyms first.

Extended-dict layer (internal/critutil/dictionaries/extended/servicenow.json): 4 entries — now_platform:instance, ai_platform:instance, app_engine:application, automation_engine:flow. All use template_format=servicenow_table_url (matches the schema enum) and region_behavior=global-only.

Business Rules

Extracted from code conditionals — every conditional is documented as a business rule.

Resume / re-fetch (main.go:advisoryHash, fetch loop)

Master-KB filtering (parser.go:ParseLanding)

CVSS extraction (parser.go:ParseAdvisory)

The extracted score and severity are consumed only by the CRIT confidence ladder and fix_propagation inference. writeAdvisory (main.go:326) writes no CVEMetric row and leaves CVEMetadata.vectorString, affectedVendor, affectedProduct and rawDataJSON null. With CRIT staging off in ECS (see the banner above), the CVSS extraction currently has no consumer in production.

Service mapping (service_map.go:Resolve)

fix_propagation inference (crit_mapper.go:inferFixPropagation)

Confidence ladder (crit_mapper.go:confidenceFor)

Detection emission (crit_mapper.go)

vex_status / shared_responsibility coupling (crit_mapper.go)

Reference filtering (parser.go:isDenyHost)

Drain (main.go:)

CRIT Envelope Shape

Per (vulnID × ServiceMatch):

{
  "envelope_version": "2",
  "spec_version": "CRITv0.2.0",
  "producer": "servicenow-kb-fetch-processor/0.1.0",
  "cve_id": "CVE-2026-0542",
  "natural_key": {"provider": "servicenow", "service": "ai_platform", "resource_type": "instance"},
  "candidate": {
    "vulnerability_id": "CVE-2026-0542",
    "provider": "servicenow",
    "service": "ai_platform",
    "resource_type": "instance",
    "vector_string": "...",
    "template": "https://{instance}.service-now.com/api/now/ai/{ai-component}",
    "template_format": "servicenow_table_url",
    "vex_status": "fixed",
    "fix_propagation": "automatic",
    "shared_responsibility": "provider_only",
    "existing_deployments_remain_vulnerable": false,
    "resource_lifecycle": "stateful_managed",
    "temporal": {"vuln_published_date": "2026-02-25", "provider_fix_date": "2026-02-25", "service_available_date": "2023-09-20"},
    "remediation_actions": [{"sequence": 1, "type": "provider_update", "title": "Provider applies fix automatically", "auto_remediable": true}],
    "provider_advisory": {"advisory_id": "KB2693566", "advisory_url": "https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB2693566"},
    "detections": [{"provider": "servicenow", "service": "ai_platform", "detection_phase": "pre_fix", "pending_reason": "query_in_development"}]
  },
  "provenance": {
    "primary_source": {"kind": "servicenow", "advisory_id": "KB2693566", "advisory_url": "...", "s3_key": "servicenow/files/{sha256}/KB2693566.json"},
    "evidence": [
      {"rule": "source-scoped-provider", "detail": "ServiceNow PSIRT KB advisory implies provider=servicenow"},
      {"rule": "service-text-match", "detail": "body text matched \"ai platform\" ⇒ (servicenow,ai_platform,instance)"},
      {"rule": "dictionary-resolved", "detail": "(servicenow,ai_platform,instance) ∈ extended dict"},
      {"rule": "fix-propagation-source", "detail": "fix_propagation=automatic inferred from explicit body text"},
      {"rule": "cve-publish-date-lookup", "detail": "loaded canonical datePublished=2026-02-25 for CVE-2026-0542 from CVEMetadata"}
    ],
    "confidence": "high"
  }
}

Verification Queries

-- Advisory count
SELECT count(*) FROM "CVEMetadata" WHERE source='servicenow';

-- Every advisory has ≥1 reference
SELECT count(*) FROM "CVEMetadataReferences"
 WHERE source='servicenow'
   AND "cveId" IN (SELECT "cveId" FROM "CVEMetadata" WHERE source='servicenow')
 GROUP BY "cveId" HAVING count(*) = 0;
-- expected: 0 rows

-- CVE alias edges (canonical direction: CVE primary, KB alias)
SELECT count(*) FROM "CVEAlias"
 WHERE "discoveredFrom"='servicenow' AND "primaryCveId" LIKE 'CVE-%';

-- In-process drain — pending state should be empty
SELECT "processingStatus", count(*)
  FROM "S3QueueObject" WHERE source='servicenow-kb-fetch-processor'
 GROUP BY 1;
-- expected: only `inserted` (and possibly `rejected`); never `pending`

-- Service distribution
SELECT service, count(*) FROM "CritRecord"
 WHERE provider='servicenow' AND "critJSON"->'provider_advisory'->>'advisory_id' LIKE 'KB%'
 GROUP BY 1 ORDER BY 2 DESC;
-- expected weighting: now_platform > ai_platform > devops/perf_analytics/etc

Risk Surface

S3 Persistence

• Archive path: servicenow/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/servicenow-kb-fetch-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.