FSTEC BDU Git Processor Design

Status: Live, but the upstream mirror has not committed since 2026-03-24 — see Upstream freshness Source: FSTEC-Dump — CIRCL’s git mirror of the FSTEC BDU Type: git (repo baked by the fstec-git-data stage, then pulled at runtime with pure-Go go-git) Source slug: fstec Schedule: Runs weekly on Wednesdays at 04:00 UTC (cron(0 4 ? * WED *)). CPU 256 / memory 1024 MB / expected_duration_minutes = 45.

Overview

Imports vulnerabilities from the FSTEC BDU (Bank of Data on Threats and Vulnerabilities) maintained by Russia’s Federal Service for Technical and Export Control. Records are stored as CVEMetadata(source="fstec") using the BDU identifier as cveId (e.g. BDU:2024-00001).

FSTEC scores aggressively and independently — every one of the ~83,000 rows carries a CVSS vector, including v2/v3/v4 vectors for CVEs that NVD left unscored — and it names Russian-market products and CVE cross-references no Western feed carries. That independent severity opinion and the BDU:*↔CVE alias edges are the reasons to carry it.

• Data source: https://helga.circl.lu/vulnerability-lookup/FSTEC-Dump.git
• Format: JSON files with Russian-language field names, one file per advisory
• File path pattern: dump/{year}/BDU:YYYY-NNNNN.json
• ECS schedule: Runs weekly on Wednesdays at 04:00 UTC (cron(0 4 ? * WED *)).
• Timeout: 45 minutes
• CPU: 256 units / Memory: 1024 MB

Architecture

[EventBridge cron(0 4 ? * WED *)]
    → ECS Fargate Task (fstec-git-processor)
        → git pull /data/FSTEC-Dump
        → Walk dump/{year}/*.json recursively
        → Parse → Map → DB batch upsert (100 files/tx)
        → Update tracker (fstec_advisory)

The git repository is baked into the container image at build time via the fstec-git-data Containerfile stage (shallow clone, --depth=1, with the HEAD SHA recorded in .baked-sha). At runtime processor.PullOrClone refreshes it — the final image stage is scratch, so there is no git binary and the pull runs through pure-Go go-git. --no-pull skips the refresh and reads the baked HEAD.

Upstream freshness. The only ingest path is CIRCL’s FSTEC-Dump git mirror, not bdu.fstec.ru directly. When that mirror stops committing the processor correctly reports repository unchanged and exits 0 every run — which looks healthy but means the data is frozen. Check BulkDataDumpTracker.lastProcessedAt for fstec_advisory against git ls-remote https://helga.circl.lu/vulnerability-lookup/FSTEC-Dump.git before concluding the feed is current.

Data Mapping

All FSTEC text fields are in Russian (lang=“ru”). The BDU identifier is used directly as cveId.

CVSS Score Parsing

FSTEC embeds numeric scores in the Уровень опасности уязвимости field using Russian text:

Высокий уровень опасности (базовая оценка CVSS 2.0 составляет 9,7)
Высокий уровень опасности (базовая оценка CVSS 3.0 составляет 8,3)

The parser uses regex CVSS\s+([\d.]+)\s+составляет\s+([\d,]+) and replaces the Russian decimal comma with a period.

Alias Handling

Идентификаторы других систем описаний уязвимости contains CVE IDs separated by commas, semicolons, or newlines. All CVE-* and GHSA-* patterns are extracted via regex and stored as CVEAlias records:

• primaryCveId = BDU identifier (e.g. “BDU:2024-00001”), primarySource = “fstec”
• aliasCveId = CVE identifier (e.g. “CVE-2022-27488”), aliasSource = “cve.org”
• discoveredFrom = “fstec”

State Mapping

Resume & Idempotency

• Per-file resume: SHA1 of each file stored in CVEMetadata.sourceFileHash. Unchanged files are skipped on re-runs.
• Tracker: fstec_advisory tracker stores the git HEAD SHA. If SHA unchanged and --force is false, the entire run is skipped.
• Delta filter: with a tracker present, processor.DetectChangedFiles narrows the walk to files touched in the last 3 days; a first run, --force, or history too shallow to diff falls back to a full scan of all ~83k files.
• Tracker is written unconditionally (deliberately — see the note below), before the errored-file exit check, so a transient DB error no longer condemns the next run to a full re-scan.

Soft deadline caveat. When EXPECTED_DURATION_MINUTES is unset the processor still installs an 80-minute soft deadline (defaultTimeout − 10 min). A local just go-fstec-git-backfill therefore stops after 80 minutes and still advances the tracker, so the files it never reached are not revisited until the upstream HEAD changes or --force is passed. On a repo this size a full local backfill should be run with --force and re-run until it reports zero remaining work.

Shared-table side effects

Beyond the CVE tables in the mapping above, the shared store path (processor.StoreCVESourceData → db.EnrichAffectedWithDependency) writes Dependency, DependencyRegistry, PackageVersion and PackageVersionCVE rows for each CVEAffected entry, and db.InsertAliases always runs — so every stored BDU record also emits the same-cveId cross-source alias edges, not just the CVE/GHSA aliases parsed out of the Russian identifier field. One CVEMetric per record is derived locally by cvss.DeriveV4FromDescription (containerType="vulnetix", metricType="cvssV4_0").

AI enrichment

Not wired. Unlike the OSV-pipeline processors, fstec-git-processor drives its own per-file loop and never constructs an aienrich.Enricher, so FSTEC rows receive no ATT&CK / CWE-inference / TreeSitter enrichment from this processor.

Transactions and retries

Each file gets its own transaction with a 2-minute timeout and up to 3 attempts (connection errors back off 1 s/2 s, other errors 0.5 s/1 s). There is no batch transaction and no per-file savepoint — --batch-size only controls how often the progress line is logged.

Local Backfill

# Using local FSTEC-Dump clone (default: ../../../FSTEC-Dump relative to vdb-manager root)
just go-fstec-git-backfill

# Custom repo path
just go-fstec-git-backfill REPO=/path/to/FSTEC-Dump

# Force reprocessing all files
just go-fstec-git-backfill FORCE=true

# Skip git pull (use existing data)
just go-fstec-git-backfill NO_PULL=true

# Run against production DB
just go-fstec-git-backfill TARGET=prod

ECS Infrastructure

• Container image: go-fstec-git-processor-{tag}
• ECS task family: go-fstec-git-processor
• EventBridge schedule: go-fstec-git-processor — cron(0 4 ? * WED *)
• CloudWatch log group: /ecs/vdb-scheduler/go-fstec-git-processor

Data Scale

As of mid-2026 the FSTEC dump holds ~83,000 advisories across 2011–2026 (all of them stored, with title, CVSS vector and affected product populated). A full scan of the tree — even one that skips every file on its sourceFileHash — takes hours at 2–3 ms per file, which is why the weekly run relies on the HEAD-SHA short-circuit and the 3-day delta filter rather than walking everything.

S3 Persistence

• Archive path: fstec/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/fstec-git-processor/{YYYY-MM-DD}/{reason}/{filename} ✓
• Failure reasons emitted: store-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.