Design: AWS Security Bulletins JSON Processor

Overview

Fetches AWS Security Bulletins via the public aws.amazon.com directory JSON API, then per-item HTML body fetch from server-rendered bulletin pages, parses CVE/GHSA aliases and AWS service references, stores per-bulletin rows under source="aws-security-bulletins", and (when --emit-crit) stages CRIT candidate envelopes per (vulnID × AWS service).

Distinct from alas-rss-processor — covers service-level AWS bulletins (Bedrock / RDS / EKS / Lambda / Cloud Cam / FreeRTOS), not Amazon Linux package CVEs. Both are provider=aws but source and bulletin-ID shapes are disjoint, so they coexist without natural-key collision.

Source identifier: aws-security-bulletins Data type: json (JSON API for discovery + per-item HTML body) ECS task name: go-aws-security-bulletins-json-processor Schedule: Daily 13:00 UTC (cron(0 13 * * ? *)) Pattern: A — inline per-record CRIT staging Phase: 1.6

Data Source

Discovery — JSON API

GET https://aws.amazon.com/api/dirs/items/search
    ?item.directoryId=security-bulletins
    &sort_by=item.dateCreated&sort_order=desc
    &size=2000
    &item.locale=en_US

Response shape:

{
  "items": [
    {
      "item": {
        "id": "security-bulletins#aws-2025-013",
        "name": "AWS-2025-013",
        "directoryId": "security-bulletins",
        "dateCreated": "2025-06-12T17:27:17+0000",
        "dateUpdated": "2025-06-12T17:30:18+0000",
        "additionalFields": {
          "bulletinDateSort": "2025-06-12",
          "bulletinDate": "12-jun-25",
          "bulletinId": "AWS-2025-013",
          "bulletinSubjectUrl": "https://aws.amazon.com/security/security-bulletins/AWS-2025-013/",
          "bulletinSubject": "CVE-2025-6031 - Insecure device pairing in end-of-life Amazon Cloud Cam"
        }
      },
      "tags": [
        {"tagNamespaceId": "security-bulletins#flag", "name": "amazon"},
        {"tagNamespaceId": "security-bulletins#flag", "name": "informational"},
        {"tagNamespaceId": "security-bulletins#year", "name": "2025"}
      ]
    }
  ],
  "metadata": {"count": 156, "totalHits": 156}
}

Volume: 156 bulletins (full coverage; no pagination needed at size=2000). The processor warns when totalHits > size so paging can be added when AWS’s volume grows.

Per-bulletin HTML body

GET https://aws.amazon.com/security/security-bulletins/AWS-YYYY-NNN/
    User-Agent: Mozilla/5.0 (vdb-manager-aws-security-bulletins/0.1.0)

Server-rendered HTML with stable data-rg-n attributes (CSS classes are obfuscated and volatile — ignored).

Fallback: when data-rg-n="BodyText" is missing, fall back to <main> slice; if that fails, fall back to the full document. ExtractFallbackUsed flag is logged.

Architecture / Data Flow

Source → Database Mappings

CVEMetadata

CVEAlias

CVE-* and GHSA-* extracted from tag-stripped body text. Always passed through db.InsertAliases (empty list still triggers same-cveId backfill).

CVEMetadataReferences

The bulletin URL is always the first reference (advisory type). All <a href> URLs inside the BodyText div extracted, then filtered:

Whitelist (always keep — precedence):

• cve.org/CVERecord / cve.org/cverecord / cve.mitre.org
• github.com/.../security/advisories/GHSA-
• github.com/.../releases/tag/
• nvd.nist.gov/vuln/detail/CVE-
• Vendor advisory hostnames: msrc.microsoft.com, support.apple.com, chromereleases.googleblog.com, lists.debian.org, usn.ubuntu.com, access.redhat.com, bugzilla.redhat.com, kb.cert.org, oracle.com/security-alerts, cisco.com, support.f5.com, psirt.global.sonicwall.com
• docs.aws.amazon.com/

Denylist:

• *.awsstatic.com
• aws.amazon.com/{contact-us,partners,blogs,whitepapers,solutions,training,what-is}
• Locale paths: aws.amazon.com/{cn,de,es,fr,it,jp,ko,pt,tr,ar,id,zh}/
• amazonwebservicesinc.tt.omtrdc.net, *.demdex.net
• self-link to bulletin URL

CVEDescription

• Body text (first 1200 chars) → row 1.
• Resolution paragraph (when present, prefixed with "Resolution: ", 1500 chars) → row 2.

S3 archive

aws-security-bulletins/files/{sha256}/{bulletinID}.json

Service Mapping

40+ synonyms covering top AWS services. Order: longer / more specific first. Examples:

Extended-dict layer (internal/critutil/dictionaries/extended/aws.json): 10 entries — freertos:device, data_all:tenant, dynamodb_local:instance, cloud_cam:device, iot_device_defender:audit, verified_access:endpoint, q:application, lake_formation:data-lake, supply_chain:instance, clean_rooms:collaboration. All template_format=aws_arn.

Business Rules

Discovery (api.go:FetchList, main.go)

HTML extraction (html.go:ExtractHTML)

Reference filtering (html.go:filterReferences)

Service mapping (service_map.go:Resolve)

Per-bulletin keying (parser.go:CombineEntry)

CRIT mapper (crit_mapper.go:mapAWSBulletinToCRIT)

fix_propagation inference (crit_mapper.go:inferFixPropagation)

Confidence ladder (crit_mapper.go:confidenceFor)

Coupling rules

• R32 automatic ⇒ provider_only.
• R33 existing_deployments_remain_vulnerable = !(provider_only && automatic).

Drain (main.go)

• R34 Drain skipped on CRIT_DISABLE_INPROCESS_DRAIN.
• R35 Drain workers fixed at 4.

Verification Queries

SELECT count(*) FROM "CVEMetadata" WHERE source='aws-security-bulletins';
-- expected: ~156

-- AWS-* primary keys (excluding any historical slug bulletins)
SELECT count(*) FROM "CVEMetadata"
 WHERE source='aws-security-bulletins' AND "cveId" LIKE 'AWS-%';
-- expected: ~156

-- GHSA aliases (AWS-2025-012 has GHSA-5x4f-fvv8-wr65)
SELECT count(*) FROM "CVEAlias"
 WHERE "discoveredFrom"='aws-security-bulletins' AND "aliasCveId" LIKE 'GHSA-%';
-- expected: ≥30

-- Reference coverage
SELECT count(*) FROM "CVEMetadataReferences"
 WHERE source='aws-security-bulletins'
   AND "cveId" IN (SELECT "cveId" FROM "CVEMetadata" WHERE source='aws-security-bulletins')
 GROUP BY "cveId" HAVING count(*) = 0;
-- expected: 0 rows

-- Service distribution: distinct from ALAS's (aws,ec2,instance) cohort
SELECT service, count(*) FROM "CritRecord"
 WHERE provider='aws'
   AND "critJSON"->'provider_advisory'->>'advisory_id' LIKE 'AWS-%'
 GROUP BY 1 ORDER BY 2 DESC;

-- Pending should be empty
SELECT "processingStatus", count(*) FROM "S3QueueObject"
 WHERE source='aws-security-bulletins-json-processor' GROUP BY 1;

Resume, notification and exit semantics

• Resume is per-record only: db.LoadProcessedHashes keyed on the bulletin id versus sha1(bulletinID | dateUpdated | sha256(rawHTML)[:8]). There is no BulkDataDumpTracker row for this processor — the full 2000-item directory listing plus every changed bulletin’s HTML is fetched on every run.
• Notification: notifier.Completed is called unconditionally at the end of the run (main.go:273) even when totalErrors > 0, and the process always exits 0. Per-item failures are recorded via notifier.RecordError but never escalate to Errored, so a run in which every HTML fetch failed still reports success and EventBridge sees a healthy task.
• No AI enrichment — no aienrich.Enricher is constructed.
• Reference / problem-type duplication: db.InsertReferences issues a bare ON CONFLICT DO NOTHING against a table with no natural-key unique index, so every re-store of a bulletin appends duplicate reference rows.

Risk Surface

S3 filenames

The unit of work is an API item plus a scraped HTML body, so the archived payload is the combined ParsedEntry marshalled by s3client.MarshalRecord and {filename} is {bulletinID}.json (main.go:220-233). The raw HTML is not archived. store-error is the only quarantine reason emitted — an HTML fetch failure has no payload bytes to store, so it is counted and reported only.

S3 Persistence

• Archive path: aws-security-bulletins/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/aws-security-bulletins-json-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.