Real-Time Fleet Telemetry Pipeline
Ingest GPS telemetry from 500 delivery vehicles and serve a live late-delivery dashboard - partition keys, idempotent writes for duplicate pings, the write-heavy versus read-heavy split, and caching the hot read.
Last updated:
Ingest GPS telemetry from 500 delivery vehicles and serve a real-time late-delivery dashboard. This is not an AI problem and the first useful thing you can do is say so. It is a streaming data problem, and the decisions that matter are the partition key, idempotency on duplicate and replayed pings, the split between a write-heavy path and a read-heavy one, and how much staleness the dashboard is allowed to show. Volume is low enough that the honest answer includes scoping the machinery down.
“500 vehicles is not a scale problem. The interesting decisions are the partition key, idempotent writes for pings that arrive twice or ten minutes late, and never letting the dashboard query the raw event store.”
Clarifying Questions (Ask These First ~5 min)
| Question | Why it matters |
|---|---|
| Ping frequency, and what is in a ping? | 500 vehicles at 1Hz is 500 events per second - small; at 10Hz with sensors it is a different system |
| How stale may the dashboard be? | Five seconds and five minutes are genuinely different architectures |
| Is history queried, and how far back? | Drives the cold store, retention and the replay story |
| How is late defined - predicted ETA or a promised window? | A threshold needs no model; a predicted ETA does |
| Do devices buffer and replay when they lose signal? | This guarantees duplicates and out-of-order arrivals, which is the real constraint |
| How many dashboard viewers, and do they all see everything? | Sets fan-out and whether per-region filtering matters |
| Does anything get actuated from this, such as customer notifications? | Adds genuine exactly-once concerns on that path only |
Architecture (Draw This)
Say this as you draw it
Key Components (30-second pitch each)
- 1The partition key is the whole question - Partitioning by vehicle_id spreads load evenly and preserves per-vehicle ordering but forces a shuffle for regional aggregates. By region gives cheap regional windows but hot partitions in dense cities. By shipment_id matches the business question - is this delivery late - and gives natural idempotency, which is why it wins here.
- 2Idempotent writes on a dedupe key - Devices buffer while out of coverage and replay on reconnect, so every ping can arrive twice or long after the fact. Key on vehicle_id plus device timestamp and upsert rather than insert, ordering by device time while keeping ingest time purely for lag monitoring.
- 3An explicit watermark and lateness policy - Decide the allowed lateness on each window and what happens to a ping arriving after the window closed - side output and correct the aggregate, or drop and count. Leaving this implicit is how dashboards develop numbers that quietly disagree with the warehouse.
- 4Split the write path from the read path - The write path is many small appends; the read path is a handful of aggregates read by many viewers. Materialise per-region aggregates on the write side and never let the dashboard scan the raw event store, or one popular view takes the pipeline down with it.
- 5Cache the hot read, and show the staleness - One cached aggregate per region with a one-to-five-second TTL turns any number of viewers into a fixed read load. Display the as-of timestamp so staleness is visible and understood rather than suspected and escalated.
- 6Keep the raw stream - Aggregates encode today's definition of late, and that definition will change. The raw archive is the only way to recompute history when it does, and it costs almost nothing at this volume.
What Separates a Strong Answer
- 1Choose the right partition key - Partitioning by shipment_id aligns with the business question and naturally supports idempotency, avoiding unnecessary data shuffling and errors.
- 2Implement idempotent writes - Using a dedupe key with vehicle_id and device timestamp ensures that replayed pings converge to the correct state, preventing data inflation and inaccuracies.
- 3Use a short-TTL cache - A cache with a 1-5 second TTL handles high read loads efficiently, ensuring the dashboard remains responsive and preventing system overload during peak times.
- 4Maintain a raw data archive - Keeping a raw data archive allows for historical analysis and future model retraining, providing flexibility and adaptability to changing definitions of lateness.
Partition Key Trade-offs
| Key | Gets you | Costs you |
|---|---|---|
| vehicle_id | Even spread across 500 keys, per-vehicle ordering | Regional aggregates require a shuffle |
| region | Cheap regional windows with local state | Hot partitions in dense cities; ordering is not per vehicle; regions change as the business grows |
| shipment_id | Matches the business question and gives natural idempotency | A single vehicle's events spread across partitions |
Write Path versus Read Path
| Property | Write path | Read path |
|---|---|---|
| Shape | Hundreds of small events per second, append-only | Tens of aggregate queries per second, many viewers |
| Optimise for | Throughput, ordering, durability | Latency, fan-out, cacheability |
| Store | Log plus time-series or key-value state | Pre-aggregated serving store behind a cache |
| Failure behaviour | Buffer at the device and replay | Serve last known good with a visible as-of timestamp |
3 Biggest Risks
- 1Replayed pings corrupting counts - A vehicle reconnecting after an hour flushes its buffer and every naive counter double-counts. Mitigated by a dedupe key and upsert semantics so replay converges to the same state rather than inflating it.
- 2Device clock skew - Device timestamps drift, and a badly skewed device can place events in windows that already closed or in the future. Mitigated by carrying both device and ingest timestamps, rejecting implausible skew, and alerting on devices whose skew is trending.
- 3Dashboard fan-out at the daily peak - Everyone opens the dashboard at the same time, and an uncached read path multiplies straight into the store. Mitigated by the short-TTL cache, pushing only state changes rather than full refreshes, and capping per-viewer subscriptions.
Google Stack: Cloud IoT-style ingest on Cloud Run or Pub/Sub HTTP → Pub/Sub → Dataflow (windowed aggregates) → Bigtable for hot state and BigQuery for the raw archive → Memorystore cache → dashboard on Cloud Run with server-sent events
Azure Stack: IoT Hub (device auth and buffering) → Event Hubs → Stream Analytics or Flink → Azure Cache for Redis hot state → Azure Data Explorer for time-series queries → ADLS raw archive → SignalR for dashboard push
AWS Stack: IoT Core (MQTT + device auth) → Kinesis Data Streams → Managed Service for Apache Flink → DynamoDB hot state + Timestream aggregates → S3 raw archive → API Gateway WebSockets for the dashboard
Other Options: Postgres with TimescaleDB and continuous aggregates, fronted by Redis - genuinely sufficient at 500 events per second and much cheaper to operate | MQTT broker plus ClickHouse where analytical queries over history dominate | Kafka and Flink only once the scaling triggers above are real.
Frequently asked questions
500 vehicles is tiny. Do you really need Kafka and Flink?
Honestly, no - and saying that first is the strongest move available. At this volume a single Postgres with a time-series extension and a materialised view handles it comfortably. Then name what would force the streaming stack: fifty thousand vehicles, a multi-second SLA, replay of a year of history, or several independent consumers.
How do you compute ETA?
Start with a routing provider's ETA adjusted by observed delay for that route and hour. A learned model needs months of labelled arrivals and is a later step. Either way the pipeline shape does not change - it is one more stateful operator emitting state changes.
What if a vehicle goes dark?
Absence is an event. A per-vehicle liveness timer emits a no-ping-for-N-minutes state change, and the dashboard renders stale vehicles distinctly rather than showing a last known position as though it were current. Silent staleness is the failure mode dispatchers never forgive.
Do you need exactly-once processing?
At-least-once delivery with idempotent upserts is the practical answer and what most production systems actually run. Reserve true exactly-once for the customer-notification path, where a duplicate message is visible to someone outside the company.
How long does history need to live?
Two tiers. Hot aggregates for the operational window - days to weeks - and raw events in object storage partitioned by hour for analytics and replay, with a retention policy set by the business rather than by the engineer who built it.
Where would AI genuinely help here?
ETA prediction, anomaly detection on route deviation, and a natural-language layer over the aggregates for dispatchers. None of them change the pipeline, and stating that the pipeline is the deliverable is the right answer to a question that was deliberately dressed up as an AI problem.