Mageia JSON Processor Design

Overview

PropertyValue
SourceMageia Linux (advisories.mageia.org)
APIhttps://advisories.mageia.org/vulns.json + bugs.json indexes, /{ID}.json per advisory
ScheduleWeekly Thursdays 07:00 UTC (cron 0 7 ? * THU *)
Timeout4 hours (backfill), 60 min (daily)
Resources256 CPU / 512 MB
Source fieldmageia
VVD formatMAGEIA-{YYYY}-{seqnum}
GCVE formatGCVE-110-MAGEIA-{YYYY}-{seqnum}

Business Logic

Freshness Check

  • Uses BulkDataDumpTracker with source=mageia, frequency=86400s (24h)
  • Skipped in --backfill mode

Processing Flow

  1. Fetch Indexes — GET vulns.json (~1057 MGASA entries) and bugs.json (~1889 MGAA entries)
  2. Deduplicate — Merge indexes, remove duplicates by ID
  3. Filter — Daily: only entries modified in last 48h; Backfill: all entries
  4. Skip Known — Skip advisory IDs already in CVEMetadata (unless --all)
  5. Fetch Advisory — GET /{ID}.json per entry with 200ms rate limiting and 3x retry
  6. Parse — Unmarshal OSV v1.6.2 JSON into Advisory struct
  7. Map — Each advisory → MAGEIA-{year}-{seqnum} + CVESourceData
  8. Store — In a single transaction per advisory:
    • UpsertCVEMetadata (source=“mageia”)
    • InsertDescriptions (details field or summary fallback)
    • InsertReferences (advisory self-link + references[])
    • UpsertAffected + InsertVersions (package info + version ranges from events)
    • UpsertGcveIssuance + InsertGcveAlias (advisory ID + CVE IDs from related[])
    • InsertAliases (CVEAlias for related[] CVE IDs)
    • S3 artifact upload → Link → fileLinkId

VVD ID Generation

  • Deterministic: MAGEIA-{year}-{seqnum} where year = year from published, seqnum = numeric portion of advisory ID
  • Example: MGASA-2026-0057, published=2026-03-14 → MAGEIA-2026-57

Advisory Types

  • MGASA (Mageia Security Advisory): Security-related, usually has related[] CVE IDs
  • MGAA (Mageia Advisory): Bug-fix, typically no CVE IDs in related[]

Data Format (OSV v1.6.2)

Each advisory contains:

  • id, published, modified, summary, details
  • related[] — CVE IDs (may be absent for MGAA)
  • references[] — {type: “ADVISORY”|“REPORT”, url}
  • affected[] — {package: {ecosystem, name, purl}, ranges: [{type, events: [{introduced, fixed}]}], ecosystem_specific}
  • credits[] — {name, type, contact}

Architecture

graph TB EB[EventBridge: weekly Thursdays 07:00 UTC] --> ECS[ECS Fargate Task] ECS --> MAIN[mageia-json-processor] MAIN --> IDX1[GET vulns.json] MAIN --> IDX2[GET bugs.json] IDX1 --> MERGE[Dedupe + Filter] IDX2 --> MERGE MERGE --> FETCH[GET /{ID}.json per entry] FETCH --> PARSE[Parse OSV JSON] PARSE --> MAP[Map to CVESourceData] MAP --> TX[Transaction] TX --> CVE[UpsertCVEMetadata] TX --> DESC[InsertDescriptions] TX --> REF[InsertReferences] TX --> AFF[UpsertAffected + InsertVersions] TX --> GCVE[UpsertGcveIssuance + Aliases] TX --> S3[S3 Upload + Link/Artifact] CVE --> RDS[(PostgreSQL)] DESC --> RDS REF --> RDS AFF --> RDS GCVE --> RDS S3 --> S3B[S3 Bucket] S3 --> RDS

Data Mapping

erDiagram MageiaAdvisory ||--o{ CVEMetadata : "maps to" MageiaAdvisory { string id "MGASA-2026-0057" string published "ISO 8601" string modified "ISO 8601" string summary "short desc" string details "full desc" } CVEMetadata { string cveId "MAGEIA-2026-57" string source "mageia" string title "from summary" string sourceAdvisoryRef "MGASA-2026-0057" int datePublished "unix seconds" } CVEMetadata ||--o{ CVEDescription : has CVEMetadata ||--o{ CVEMetadataReferences : has CVEMetadata ||--o{ CVEAffected : has CVEMetadata ||--o{ CVEAlias : has CVEMetadata ||--o| GcveIssuance : has CVEMetadata ||--o| Link : "fileLinkId" Link ||--|| Artifact : has CVEAffected ||--o{ CVEAffectedVersion : has

Directory Structure

scripts/go-processors/
├── cmd/mageia-json-processor/
│   └── main.go
├── internal/mageia/
│   ├── types.go      (Advisory, IndexEntry structs)
│   ├── parser.go     (ParseIndex, ParseAdvisory, ID extraction)
│   └── mapper.go     (MapToSourceData, reference/affected mapping)
└── mageia-processor.design.md

S3 Persistence

  • Archive: ⚠ Not yet implemented — requires record reconstruction (DB row → canonical JSON).
  • Quarantine: ⚠ Not yet implemented — same reason.
  • Likely reasons when implemented: enrich-error

This is an enrichment processor; it reads from CVEMetadata rather than ingesting raw feeds, so there is no original payload to archive verbatim. See S3 Persistence Contract § Processors whose unit-of-work is not a file.