CERT-JP RSS Processor Design

Overview

Fetches advisory feeds from the Japanese Vulnerability Notes Database (JVN/JVNDB) published by CERT-JP (IPA / JPCERT/CC). Per-year historical feeds cover 2002–present. Both the English and Japanese language editions of each feed are fetched and merged, storing bilingual descriptions (lang="en" + lang="ja") per CVE. Two rolling feeds exist for incremental ECS updates.

Mapped records are stored with source="cert-jp" in CVEMetadata.

Scheduling

The soft deadline is checked between years, so a wide --start-year range stops at a year boundary rather than mid-year; the per-year tracker means the next run resumes at the year it did not reach. A full --start-year=2002 backfill run therefore needs several invocations (the local recipe unsets EXPECTED_DURATION_MINUTES, which leaves the hard-coded 55-minute default in force rather than removing the deadline).

Feed Structure (JVNRSS 3.1 / RDF)

The feed is RDF/RSS 1.0 (xmlns="http://purl.org/rss/1.0/") with security namespace extensions. The same XML schema is used for both English (/en/) and Japanese (/ja/) editions.

Feed URLs

Base URL: https://jvndb.jvn.jp

Package Layout

internal/certjp/types.go

RDF feed structs (Feed, Channel, Item, Reference, CVSS, CPE) and normalized Advisory.

Advisory includes JapaneseTitle and JapaneseDescription fields populated by MergeJapanese.

internal/certjp/parser.go

• ParseFeed(data []byte) — XML unmarshal; returns advisories + parse errors
• ParseItem(item Item) — normalize dates, extract CVE IDs from references, deduplicate CPEs
• FeedSHA256(data []byte) — hex SHA256 for dedup tracker
• CombinedHash(enHash, jaHash string) — SHA256(enHash|jaHash); falls back to enHash if JA unavailable
• MergeJapanese(enAdvisories, jaAdvisories []*Advisory) — copies JA title + description into matching EN advisories by JVNDB ID
• CVEIDFromID(id string) — regex ^CVE-\d{4}-\d{4,}$

internal/certjp/mapper.go

• MapAdvisory(adv *Advisory, rawJSON string) — returns one CVESourceData per CVE ID
• Emits lang="en" description always; also lang="ja" when JapaneseDescription is set
• Both descriptions are passed to InsertDescriptions in a single call (delete+insert atomicity)
• CVSS version mapping: "2.0" → cvssV2_0, "3.0" → cvssV3_0, "3.1" → cvssV3_1
• CPE parsing: supports CPE 2.2 (cpe:/a:vendor:product) and CPE 2.3 formats
• Affected hash: MD5(vendor|##|product|##||##|)

cmd/cert-jp-rss-processor/main.go

Main binary with flags:

ECS mode (start==end==currentYear): also processes rolling feeds (cert_jp_current, cert_jp_new).

Processing Flow

1. Parse flags; validate 2002 ≤ start ≤ end ≤ currentYear
2. Connect DB using a short-lived 30-second context (main.go:83), then run under a cancellable background context bounded by a soft deadline of EXPECTED_DURATION_MINUTES - 10 (default 55 min when the var is unset); init S3 uploader if S3_BUCKET_NAME set
3. For each year: call processFeedPair(enURL, jaURL, trackerSrc)
4. In ECS mode: also call processFeedPair for rolling feeds

processFeedPair per year/feed:

1. Fetch EN feed (required — returns error on failure)
2. Fetch JA feed (optional — log warn and continue English-only on failure)
3. Compute combinedHash = CombinedHash(enSHA256, jaSHA256) — tracks both feeds
4. Skip entire year if tracker hash unchanged and !force
5. Parse EN advisories; parse JA advisories (if available)
6. MergeJapanese — copy JA description/title into matching EN advisories by JVNDB ID
7. Filter out advisories with no CVE IDs
8. Load resume set (cveId → sourceFileHash) for source='cert-jp'
9. Set adv.FileHash = combinedHash so per-advisory resume also tracks JA changes
10. Map each advisory with certjp.MapAdvisory → descriptions include both lang="en" and lang="ja"
11. Accumulate mapped records in batches of 100, then hand each batch to processBatch, which opens one transaction per record (not per batch) with a 2-minute timeout and up to 3 attempts — a slow or failed connection loses one advisory rather than the whole batch (main.go:305). Attempt 2+ re-runs strings.ToValidUTF8 on RawDataJSON when the error looks like SQLSTATE 22021
12. UpsertTracker(trackerSrc, combinedHash, processed)

Data Mapping

Error Alerting

Partial success exits 0 deliberately, so the per-year tracker survives for the next run. Note that notify.Completed publishes only Stats and does not flush the messages accumulated by RecordError (internal/notify/notify.go:213-233) — the count of failed records reaches Slack in the stats, but the per-record detail stays in the task logs. Per-advisory parse errors (main.go:215-217) are not counted at all.

Combined Hash / Tracker Design

The tracker for each year uses a single key cert_jp_{YYYY} with value SHA256(enHash + "|" + jaHash). This means:

• If only the EN feed changes → combined hash changes → full reprocess
• If only the JA feed changes → combined hash changes → full reprocess (picks up new Japanese)
• If JA feed is unavailable → CombinedHash returns enHash directly → no churn

The per-advisory FileHash is also set to combinedHash, so the resume set correctly tracks whether an advisory has been processed with both languages.

Decisions

• Skip non-CVE advisories: JVN publishes advisories for JVN IDs without CVE assignments. These are skipped since CVEMetadata requires a CVE ID.
• JA feed optional: if the Japanese feed returns an error, the processor falls back to English-only without failing the run (logged as warning).
• Bilingual in one transaction: both EN and JA descriptions are stored in a single StoreCVESourceData call. InsertDescriptions deletes then re-inserts, so both langs are always accurate and consistent.
• Per-advisory multi-CVE — no bundling aliases: when an advisory references multiple CVEs, every CVE gets its own full CVEMetadata row (same title, descriptions, metrics, affected, sourceAdvisoryRef), and no CVE↔CVE alias edges are written. A JVN bulletin listing several CVEs is co-listing them, not asserting that they are the same vulnerability. The JVNDB id alias is likewise emitted only for single-CVE advisories, where it is a legitimate 1:1 cross-prefix identifier (internal/certjp/mapper.go:58). Cross-source linkage is preserved by db.InsertAliases’ same-cveId backfill, which links (CVE-X, cert-jp) to every other source holding CVE-X.

Alias policy and JVNDB placeholder rows

Emitting the JVNDB id as an alias has a side effect: db.InsertAliases calls EnsureMinimalCVEMetadata for any non-CVE alias with no row of its own, so each single-CVE advisory also mints a content-free CVEMetadata row keyed (JVNDB-YYYY-NNNNNN, "cert-jp"). In production 63 of the 1,885 cert-jp rows are these placeholders — they carry no title, no rawDataJSON and datePublished = 0 by construction, and must be excluded from any field-coverage measurement of this source.

Artifact / Link provenance

Beyond CVEMetadata and its children, processOneItem writes to three shared tables when S3 is configured (main.go:362):

This is a second S3 path, distinct from the archive/quarantine contract below: it uses a raw PutBool closure and the legacy cert-jp/advisories/{hash}/{cveId}.json key layout rather than {source}/files/{sha256}/{filename}. Failures at any step are logged as warnings and do not fail the record.

Verification Queries

-- Check bilingual descriptions
SELECT d."cveId", d.lang, LEFT(d.value, 100) AS description
FROM "CVEDescription" d
WHERE d.source = 'cert-jp'
ORDER BY d."cveId", d.lang
LIMIT 40;

-- Lang distribution
SELECT lang, COUNT(*) FROM "CVEDescription" WHERE source = 'cert-jp' GROUP BY lang;

-- Advisories with both EN and JA descriptions
SELECT "cveId", COUNT(DISTINCT lang) AS langs
FROM "CVEDescription"
WHERE source = 'cert-jp'
GROUP BY "cveId"
HAVING COUNT(DISTINCT lang) = 2
LIMIT 10;

S3 Persistence

• Archive path: cert-jp/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/cert-jp-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: parse-error, store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.