nuha · Oct 2025 – May 2026 · Software Engineer
Real-Time Pipeline and Edge Fleet
Speech-to-speech held inside a 1.8-second budget, degrading gracefully instead of stalling. Setup time cut 80%, and production updates recoverable by rollback rather than a site visit.
1.8s
speech-to-speech budget
50+
Linux edge devices
80%
less setup time
Background
Context
nuha builds mmWave radar devices on Raspberry Pi that sense the room and hold a spoken conversation with the people in it. From October 2025 to May 2026 I was a software engineer in New York responsible for the fleet: 50+ Linux edge devices deployed on networks the company does not control.
I built the two Rust services on the real-time path. teddy-core is the one a person actually experiences: it takes the moment someone stops speaking and produces a spoken reply, orchestrating transcription, the language model and speech synthesis over a WebSocket connection. teddy-lab sits behind it, turning raw audio and sensor streams into the structured context that makes the reply specific to the room it is spoken in.
The whole round trip has a budget of 1.8 seconds. Past that a reply stops reading as conversation and starts reading as a machine thinking, which is a product failure even when every component technically worked.
Each device runs a systemd-managed radar agent I built, which talks to the mmWave sensor over UART and SPI, configures chirp profiles and streams frames and health telemetry to on-device and cloud services.
Constraint
The hard part
Every device needs an identity, a secure path back to the cloud, a way to receive commands, and updates that can never leave it unbootable in someone's home or facility.
Manual setup was the bottleneck. Provisioning a device by hand, opening remote access ad hoc and shipping updates without a rollback path does not scale to 50 devices, let alone the next 500.
Ownership
What I was responsible for
I owned the end-to-end device and cloud control plane: the overlay network, enrollment, the registrar API, command delivery, telemetry, audit logging and OTA. I also built the operator-facing Kitting UI and iterated on it directly with the people using it.
System
Architecture and flow
Two layers. On the real-time path, raw audio and sensor streams become structured context, which becomes a spoken reply, each stage holding a share of the budget. Underneath, every device joins a WireGuard overlay policed by nftables. Identity is issued at enrollment by the registrar, which exposes four endpoints: /enroll, /config, /telemetry and /commands, authenticated with Supabase JWTs and RBAC. Operators reach devices through Cloudflare Zero Trust SSO and SSH ProxyJump, never through an exposed port, and push updates through hawkBit to RAUC on the device.
Real-time path · 1.8s budget
Rust + Python
Rust
Fleet beneath it
50+ devices
Raspberry Pi · radar-agent · RAUC
nftables
Registrar · JWT + RBAC
- /enroll
- /config
- /telemetry
- /commands
device shadow · idempotent command queue · retry / backoff · audit log
Operators: Zero Trust SSO · SSH ProxyJump · Kitting UI · hawkBit
Rollback on every OTA update
The real-time path a spoken turn takes, and the fleet infrastructure underneath it. Every control call crosses the overlay and is logged.
Decisions
Key technical decisions
1
Overlay first, least privilege by default.
WireGuard gives every device a stable, authenticated address regardless of the network it sits on. nftables on top enforces who can talk to what, so a compromised or misbehaving device cannot reach beyond its role.
2
Issue identity at enrollment, not by hand.
The registrar API and a Supabase registry provision each device's identity, WireGuard keys and addresses automatically. Zero-touch enrollment turned setup from an engineer's afternoon into a step in the kitting process, and combined with Zero Trust SSO and ProxyJump it cut setup time 80%.
3
Commands are idempotent, queued and retried.
Devices go offline. The /commands endpoint feeds an idempotent queue with retry and backoff, and a device shadow records the last known state, so a command delivered twice or delivered late has the same effect as one delivered once, on time.
4
Make the deadline the contract, not the goal.
Every stage of the round trip gets a share of the 1.8 seconds, and the assembly step returns whatever it has when its share runs out rather than waiting for stragglers. Anything late is simply picked up on the following turn. Treating the deadline as a guarantee the service always honours, rather than a target it usually hits, is what keeps a slow dependency from turning into a silent pause.
5
Put the ML workers inside the binary.
teddy-lab runs as a single Rust binary with an embedded Python interpreter through PyO3. Rust handles what Rust is good at, streaming I/O and time alignment across audio and sensor streams; Python handles the model work. Because they share one process and one address space, passing a buffer between them costs nothing, where two processes would have meant serialising and copying on every tick of a real-time path.
6
One writer for shared state.
Several independent inference paths produce partial results, and all of them are merged by a single component before anything is published. Letting each worker write directly would have been simpler to build and would have produced races and flicker that are painful to reproduce, let alone debug.
7
Every update must be able to roll back.
hawkBit manages rollouts and RAUC applies them on the device with rollback support. A bad image is recoverable without a site visit, which is the property that makes fleet-wide updates safe to attempt at all.
Failure
Reliability and failure handling
- The pipeline degrades instead of blocking: a stage that overruns its share of the budget is dropped for that turn rather than making the person wait, and its result is used on the next one.
- Live processing is best-effort by design, while persistence runs asynchronously behind it, so a slow database write can never stall a conversation.
- Rollback-safe OTA: a failed update reverts rather than bricking the device.
- Idempotent command queue with retry and backoff, so intermittent connectivity does not cause duplicate or lost actions.
- Device shadow state gives the cloud a consistent view of each device even while it is unreachable.
- Audit logs on every control-plane action, carried over WireGuard, so operator changes are attributable.
- No exposed ports: remote operations go through Cloudflare Zero Trust SSO and SSH ProxyJump.
- The radar agent runs under systemd, so a crashed process restarts without intervention.
Outcome
Measured outcome
50+
Linux edge devices on the overlay
80%
reduction in device setup time
1.8s
speech-to-speech budget
2
Rust services on the real-time path
4
authenticated registrar endpoints
rollback
available on every OTA update
- Zero-touch enrollment replaced manual provisioning.
- Operators moved from ad-hoc SSH to a Kitting UI, runbooks and audit logs.
- Real-time vital-sign streaming from the on-device radar agent.
Learned
What I took from it
A control plane is mostly a set of promises about failure: what happens when the device is offline, when a command arrives twice, when an update is bad. Writing those down first made the API design almost mechanical.
Building the Kitting UI and then sitting with operators while they used it taught me more about the system's real edge cases than any log did.