Real time ICU telemetry with WebSockets, Redis and Supervisor: the failure modes we plan for

A monitoring dashboard looks like a web app, but it behaves like a streaming system. Data never stops, delays matter, and a silent failure is dangerous. We design these platforms from the failure list backward.

The shape of the pipeline

• Ingestion: device gateways post readings to an authenticated endpoint that writes to a queue and answers at once.
• Processing: workers normalize units, validate ranges and run the anomaly rules.
• Delivery: WebSocket channels scoped by unit and role push updates to dashboards and mobile.
• Alerting: a routing table decides who is told, and escalates if nobody acknowledges.

Why Redis and Supervisor

Redis queues give us fast, durable hand off between ingestion and processing. Supervisor keeps worker processes alive and restarts them if they die, which they will. The combination is boring, and in a clinical system boring is a feature.

Failure modes to test on purpose

• A gateway goes quiet. The system must show "no data" for that bed, not the last good value.
• A worker crashes mid batch. Retries must be idempotent so a reading is never counted twice.
• The queue backs up. Alerts need priority lanes, so a critical event is not stuck behind routine data.
• A WebSocket drops. The client reconnects and replays missed readings from a short buffer.
• The alert target is unavailable. Escalation moves to the next person after a timer.
• The database is slow. Writes are buffered, the live view reads from memory, and the audit log still records.

Auditability

Every alert records who saw it and when, and every view of a patient is written to an append only audit log. This matters for quality review as much as for compliance.

Load and failover testing

We replay a recorded peak at several times the expected rate, then kill workers, Redis and the WebSocket server one at a time while watching delivery. The test report goes to the client with the go live checklist.