digitalocean-fetch-processor

Status: Live (commentary enricher) Source: DigitalOcean Blog — Security topic Type: fetch (HTML scraping) Source slug: digitalocean

Overview

DigitalOcean has no PSIRT and no first-party CVE feed. The digitalocean.com/blog?topic=security page surfaces blog posts that occasionally comment on third-party CVEs affecting infrastructure DO runs on behalf of customers — kernel CVEs, OpenSSH (regreSSHion-class), Postgres, runc / containerd, etc. — and announce DO’s response on Droplets, Managed Kubernetes, Managed Databases, and friends.

This processor therefore is a third-party-analysis reference enricher, not an advisory producer. It mirrors the fastly-fetch-processor precedent exactly:

• It never creates CVEMetadata rows. A DO commentary post is too weak a signal to seed a canonical CVE record on its own.
• For each CVE-mentioning post, it inserts one CVEMetadataReferences row per (cveId, source) tuple already present in CVEMetadata (the post is a fact about the CVE, not about any particular ingesting source’s representation of it). Rows carry type="third-party-analysis" and referenceSource="digitalocean".
• CVEs not yet known to vdb-manager are silently skipped, counted as cvesUnknown.
• When a post names a DigitalOcean product mitigation, the processor optionally stages CRIT candidates with vex_status="not_affected" (the provider sits upstream of the vulnerable component) gated on --emit-crit and on a successful dictionary resolution for (digitalocean, service, resource_type).

Source

The marketing-compliance landing at digitalocean.com/security points to the Intigriti bug-bounty program and is not a feed. The operational status.digitalocean.com/history.rss is RSS but contains zero CVE references — incidents only.

Page Structure

Listing page — <a href="/blog/{slug}"> anchors are emitted server-side. The processor excludes /blog/tags/..., /blog/category/..., /blog/author/..., and /blog/topic/... aggregation paths.

Detail page — three signals of interest:

Parsing

internal/digitalocean/parse.go uses compiled regexp (no DOM tree — the Next.js server-rendered HTML is large but the signals are stable):

ParseListing returns absolute post URLs, excluding aggregation prefixes. ParseDetail returns ok=false when no CVE references are found — those posts are out of scope.

Storage

CVEMetadataReferences row shape per emitted reference:

CVEMetadataReferences has no unique constraint covering (cveId, source, url, referenceSource) — only plain indexes — so ON CONFLICT cannot dedupe. The insert is therefore guarded by an explicit WHERE NOT EXISTS (…) sub-select (main.go:352-360) and inserted is counted from tag.RowsAffected(), which makes a re-run over an unchanged post a true no-op. (fastly-fetch-processor still uses the bare ON CONFLICT DO NOTHING form and duplicates on every run — do not copy it.)

CVEMetadata writes: none. CVEAlias writes: none. CritRecord writes: yes, indirectly — see CRIT Staging below.

Identifier Policy

No DigitalOcean-issued identifiers exist; the processor extracts CVE references via CVE-\d{4}-\d{4,} and ignores everything else. Posts with zero CVE matches are not stored.

Incremental Strategy

• Freshness gate: BulkDataDumpTracker row keyed by source="digitalocean". When now - lastProcessedAt < frequencyMs the run exits via notifier.NoWork. --force bypasses the gate.
• Soft deadline: when EXPECTED_DURATION_MINUTES is set (ECS schedule), the loop stops cleanly at expected - 10 minutes. Local backfills unset the var (see feedback_backfill_no_deadline).
• Idempotency: re-running over the same posts produces zero new rows on the second pass — the WHERE NOT EXISTS guard on the reference insert dedupes on (cveId, source, url, referenceSource).

CRIT Staging

Optional, gated on --emit-crit. When a post body contains a keyword matching a DigitalOcean product, the processor resolves (digitalocean, service, resource_type) against the CRIT spec dictionary (ietf-crit-spec v0.3.x) and stages one CRIT envelope per (cveId, CVEMetadata-source) pair with:

Product keyword → (service, resource_type) map (slugs match the upstream CRIT spec dictionary):

Posts that name “load balancer” or “VPC” pass the keyword filter but fail the dictionary gate (no spec entry yet), so they produce a reference row but no CRIT envelope. Pure-commentary posts (no DO product keyword at all) likewise produce reference rows only.

S3 Persistence

The raw post HTML is content-addressed on success via uploader.ArchiveRecord(ctx, "digitalocean", payload). Failure-path quarantine is non-compliant against the S3 Persistence Contract; see the compliance matrix for the open item.

Slack Notifications

Standard envelope via internal/notify:

• Started at pipeline entry
• RecordError(detail) per per-post failure (fetch, source lookup, insert)
• NoWork when the freshness gate fires or every parsed post was out-of-scope marketing
• Errored when at least one post failed terminally (exits 1)
• Completed with {fetched, postsParsed, postsSkipped, postsFailed, refsInserted, cvesUnknown, critStaged}

Flags

ECS Schedule

Architecture

Key Files

Future Work

• Wire Quarantine calls on fetch/parse failures to close the S3 contract gap.
• Add YARA/Snort detection queries once a CRIT envelope is staged for real (currently pending_reason="query_in_development").
• Reassess if DigitalOcean ever publishes a structured advisory feed — that would supersede this commentary-only path.

S3 Persistence

• Archive path: digitalocean/files/{sha256}/{filename} ✓
• Quarantine path: failed-feeds/digitalocean-fetch-processor/{YYYY-MM-DD}/{reason}/{filename} ⚠ not yet wired
• Failure reasons emitted: fetch-error, parse-error

Uses s3client.Uploader from internal/s3client/uploader.go. Skipped when S3_BUCKET_NAME is unset (local dev).

See the S3 Persistence Contract for the full reason taxonomy.