zombie-sweeper-json-processor

Status: Live — the sweep itself is clean, but its own inline guard emits false zombie alerts (see Known defects) Source: CloudWatch Logs DescribeLogGroups / DescribeLogStreams over the /ecs/vdb-scheduler/ prefix, plus the ECS API Type: json (operational; no advisory feed, no database writes) Source slug: none — this processor writes nothing to PostgreSQL Schedule: Runs every 30 minutes (cron(*/30 * * * ? *)), 256 CPU / 512 MB, expected_duration_minutes = 10.

Overview

Sweeps every ECS task family under /ecs/vdb-scheduler/ for stale or previous-date zombie tasks. For each log group it lists log streams ordered by lastEventTime desc; any stream that is either

• idle for more than the staleness threshold (6 hours), or
• started before today’s UTC midnight and idle beyond the healthy window (10 minutes)

becomes a zombie candidate.

A candidate is only stopped once ecs:DescribeTasks confirms the task is still RUNNING. Only those confirmed kills publish a task.zombie_detected SNS event (Slack-bound) and call ecs:StopTask.

Why the liveness check is essential, not defensive. A task that finishes normally stops emitting logs, so its stream’s lastEventTime freezes and it becomes indistinguishable from a hung task by log recency alone. The previous-date rule then matches every historical stream from an earlier UTC day, and DescribeLogStreams returns up to 50 streams per family across ~156 families. Judging on log recency alone therefore issued StopTask against long-finished tasks on every single sweep — raising false task.zombie_detected alerts and consuming the shared ecs:StopTask rate limit that every processor’s inline guard also depends on.

Expect zombie_candidates to greatly exceed zombies_killed. That gap is the check working, not a fault.

If liveness cannot be established — DescribeTasks errors, or the permission is missing — the sweeper fails closed and kills nothing. A missed hung task costs one wasted schedule slot; a wrongful StopTask costs a false alert plus shared rate limit, and that was the cost being paid continuously.

The inline zombieguard.CheckFromEnv in each processor’s main() handles the common case (next cron fires while a healthy sibling is mid-run). This sweeper handles the edge cases: tasks whose previous instance crashed before reaching the inline check, long-interval (daily+) tasks whose zombie would otherwise sit unkilled for 24 hours, and tasks removed from task-manager.toml that still have something stuck in ECS.

Thresholds

The sweeper calls zombieguard.Detect(taskName, defaultStaleThreshold/2, …) — i.e. it passes 3 hours as the notional expected duration, and Detect derives the actual windows from it:

A stream idle less than healthyWindow marks the family as having a healthy sibling and is never touched.

Cluster and region resolution

ZOMBIEGUARD_CLUSTER_ARN is used when set. Otherwise the sweeper reads its own ECS task metadata (ECS_CONTAINER_METADATA_URI_V4 → /task) for the cluster ARN and derives the region from the task ARN. Outside ECS both paths fail and the run exits 1, which is why there is no just recipe for it.

Storage

Writes no database rows. This is an operational sweeper: it stops zombie ECS tasks and publishes SNS notifications (Slack-bound); it does not ingest advisory data.

IAM

The task role needs logs:DescribeLogGroups and logs:DescribeLogStreams on /ecs/vdb-scheduler/*, plus ecs:DescribeTasks and ecs:StopTask on task/vdb-scheduler/* (terraform/iam.tf, aws_iam_role_policy.ecs_task_zombie_detection).

ecs:DescribeTasks is required, not optional. Without it the liveness check fails, and because the sweeper fails closed on an unknown liveness it will stop killing zombies entirely rather than fall back to log recency.

Runtime budget

The whole run is bounded by a hard 5-minute context.WithTimeout, not by EXPECTED_DURATION_MINUTES. A real run scans 156 log groups in ~189 seconds (2026-08-06 08:30), so the margin to the hard timeout is about 40%. Each new processor family adds a DescribeLogStreams round-trip, so this is worth re-measuring as the fleet grows — a timeout aborts the sweep with exit 1.

Fixed 2026-08-06 — false kills of already-finished tasks

The defect. zombieguard.Detect reads only CloudWatch log-stream timestamps; it never asked ECS whether the task was still RUNNING. A run that exited cleanly looks identical to a hung one once its log stream goes quiet. Because the sweeper’s own inline check used a 5-minute expected duration (a 10-minute stale threshold) against a 30-minute cron interval, every run declared its own predecessors zombies.

Observed on 2026-08-06 08:30: two zombieguard: stopped zombie task lines with reason=stale-timeout and idle_for of 26m48s and 56m55s, both against sweeper tasks that had already completed successfully (1f1a0ceb… ran 08:00–08:03, 6907c5e4… ran 07:30–07:33) — and 6907c5e4… had already been “killed” by the 08:00 run. Each published a Slack-bound task.zombie_detected event, so the sweeper generated roughly 96 false alerts a day plus an equal number of pointless ecs:StopTask calls, which contributed to the ThrottlingException: Rate exceeded failures other processors logged from the same shared guard.

The sweep was subject to the same blindness: a daily task that finished yesterday keeps a log stream whose first event precedes today’s UTC midnight, satisfying previous-date for as long as it stays in the most recent 50 streams.

The fix. zombieguard.KillAndNotify was replaced by zombieguard.StopZombies, which resolves all candidate task ARNs, batches ecs:DescribeTasks (100 per call) and stops only those ECS reports as RUNNING. PENDING/PROVISIONING/ACTIVATING are skipped because log recency says nothing about a task that has not started; tasks absent from the response (aged out of stopped-task retention) are skipped as long gone. The unchecked kill path was removed rather than deprecated, so it cannot be reintroduced by a future caller — Check and the sweeper are the only two callers and both now go through StopZombies.

Behaviour on failure is deliberately asymmetric: if liveness cannot be established the sweeper kills nothing. A missed hung task costs one wasted schedule slot, whereas a wrongful StopTask costs a false alert plus shared rate limit — and the second was being paid continuously.

Run stats now separate the two populations: zombie_candidates (flagged by log recency) against zombies_killed (confirmed RUNNING), plus skipped_not_running, skipped_unknown, skipped_unresolved and stop_failed. A large candidate-to-killed gap is expected and healthy.

Regression cover is in internal/zombieguard/zombieguard_test.go, which pins the stream-to-ARN mapping, asserts that a malformed cluster ARN can never yield a guessable task ARN, and states the killable-status table with the reason each status is or is not killable.

S3 Persistence

Not applicable. This processor consumes no feed and produces no records — it reads CloudWatch metadata and calls the ECS API. There is no payload to archive and no record to quarantine, so the S3 Persistence Contract does not apply; its absence here is by design, not a compliance gap. Its own run output is a CloudWatch log stream, which is archived to S3 by the shared log-delivery Firehose like every other task’s.

Processing flow