Told through a company I made up, called Wavelink. It is a messaging app, loosely modeled on how WhatsApp actually grew, with the name changed and the story simplified. The engineering problems and the numbers where I cite them are real.
Wavelink starts as a weekend project: a chat screen bolted onto an existing app. The first version has each client ask the server "anything new for me?" every three seconds. It works for a demo. At a few thousand users it is already indefensible: most requests come back empty, the database takes the same load whether anyone is typing or not, and a message sent right after someone's poll still takes up to three seconds to show up.
The fix that actually matters is not a smarter poll interval, it is dropping polling entirely. A chat app is not really a request-response problem. It is closer to two people holding a phone line open, where either side can talk at any moment. That reframing is what the rest of this is built on.
Once polling is off the table, each client needs a connection the server can push down at any time. A raw TCP socket, wrapped in TLS, kept open for as long as the app is running, is the simplest version of this. WebSockets are the same idea dressed to survive corporate proxies and browsers.
The part that is easy to miss: almost every one of these connections is idle almost all the time. A user opens the app, it connects, and then nothing happens for minutes. Wavelink's first production servers ran each connection as its own OS thread, which is fine at a few thousand users and falls over near ten thousand, because a thread's stack and scheduling overhead cost real memory and CPU even when it is doing nothing.
The fix is an event loop, or in Erlang's case a process model built for exactly this: a huge number of very cheap, independently scheduled units, each one just waiting on its own socket. WhatsApp built its backend on Erlang for this reason specifically, and later published that a single server, tuned, held a little over two million concurrent connections. That number is not from a bigger machine, it is from a runtime whose per-connection cost is a few kilobytes instead of a thread.
With a connection open, a message from A to B has to cross the server exactly once but never gets sent client to client directly, since either side can be behind a firewall or NAT that makes them unreachable from outside.
Writing to a per-recipient queue before trying to push is what makes this durable. If the push attempt is skipped and the server only tries to deliver to connected users, a message sent in the half second B's phone loses signal is gone. Writing first means delivery is a guarantee the queue makes, and pushing to an open connection is just an optimization for the common case.
The per-user queue is the whole answer to offline delivery: nothing special has to happen, because the message was already durably written there when it was sent. The interesting design question is what a client does the moment it reconnects.
Wavelink's client sends the id of the last message it has seen, and the server replays the queue from that point forward. This has a sharp edge: if the client stores that id wrong, or a bug resets it, the client either replays messages it already has or, worse, skips ones it never received. Wavelink learned this the hard way when a client-side bug persisted the wrong id after a crash, and a slice of users quietly missed messages for a week before anyone reported it, because a missing message produces no error, just silence.
Three separate facts, each true at a different point in the message's life, and each one needs its own acknowledgment traveling back to the sender.
Each ack is itself a small message traveling the same path in reverse, through the same per-user queue and push mechanism as the original message. There is no separate acknowledgment channel, which keeps the delivery guarantees identical for messages and for the acks about them.
Two failure modes show up together often enough that they get solved with the same mechanism. A flaky connection makes the client retry a send it is not sure went through, which risks a duplicate. A server restart mid-delivery risks a message arriving out of order relative to one sent moments later through a different server process.
The client assigns each message a unique id before it ever leaves the device. The server treats that id as the deduplication key: if the same id arrives twice, the second one is dropped, and the sender still gets back a "sent" ack either way, so a retry after a dropped ack is indistinguishable from a first attempt to the sender. Ordering within a single conversation is kept by having every message for that conversation pass through the same queue, so replays and pushes stay in the order they were written, without needing a global clock.
A group message is not one message, it is one message that has to become several, one per member. The design question is when that fanout happens.
Fanout on write is the more expensive option per send, one write per member instead of one, but it keeps the read path trivially simple: a client's "give me my new messages" logic never needs to know whether a message came from a person or a group. That simplicity is worth more than the write cost as long as group size stays bounded, which is exactly why messaging apps cap group size rather than letting it grow the way, say, a Twitter following list can. WhatsApp's own group limit has moved over the years, from 256 up to 1024, but it has always been a fixed ceiling rather than unbounded, for precisely this reason.
Wavelink's first attempt sent image bytes inline in the same message payload that carries text. It works until someone sends a ten megabyte video into a group of two hundred people, and the fanout from the previous section turns one upload into two hundred copies of ten megabytes moving through the messaging pipeline meant for kilobyte-sized text.
The fix is to take media out of the messaging path entirely. The client uploads the file to a separate blob store first and gets back a reference, and the chat message that actually fans out to every member is small: a reference, a thumbnail, and metadata. The two hundred recipients each fetch the full file once, on their own schedule, from storage built for that, not from the low-latency messaging pipeline. The messaging system's job stays "deliver a small, ordered stream of events," which is the job it was actually designed for.
Wavelink's connection layer eventually needs to hold millions of mostly-idle connections, and the honest question is how many of those one server can carry before the answer is just "add more servers."
WhatsApp published the answer they arrived at in detail: a FreeBSD box, tuned kernel parameters for socket buffers and file descriptor limits, and an Erlang process per connection, held a little over two million concurrent connections on a single machine. The interesting part of that story is not the final number, it is that most of the work was removing artificial ceilings the OS and runtime impose by default, not adding hardware.
Past a single machine's ceiling, connections get sharded by user id, usually with consistent hashing so that adding a shard does not reshuffle every existing connection's assignment. A user's messages always route through their assigned shard first, which is why the earlier delivery diagram labeled the first server "A's shard": that is a specific, addressable server, not a generic pool.
Once a message has been delivered and acknowledged, does the server still need it? Wavelink's answer, and WhatsApp's stated one, is no. Messages sit in a user's queue only until delivered, and undelivered messages expire after a bounded window, on the order of a month, rather than being retained indefinitely.
This is a real product tradeoff, not a free simplification. It means a user cannot install the app on a new phone and pull down five years of chat history from the server, the way an email client can. Wavelink pushes history onto the device instead and lets users back it up themselves, which keeps the server's storage bill flat as the user base grows, instead of growing with the full lifetime volume of every message anyone has ever sent.
One of the non-functional requirements above was that the operator should not be able to read message content even under compulsion. That requirement shapes storage more than it shapes any single component: if the server cannot decrypt a message, it also cannot search it, index it, or moderate its content server-side, which rules out a whole category of features other systems get almost for free.
The mechanism WhatsApp adopted, after partnering with Open Whisper Systems, is the Signal Protocol's double ratchet: each message uses a new derived key, so that even if one key is later compromised, past and future messages in the same conversation stay unreadable from it. The server's role shrinks to exactly what the earlier sections describe, moving encrypted bytes through queues in order, with no ability to look inside them. Delivery, ordering, and dedup all still work, because none of that needs the content, only the envelope: sender, recipient, message id.
"Online now" and "typing…" look like they belong in the same durable, ordered system as everything above. They do not. Presence is a snapshot of the current moment, not an event with lasting meaning: nobody needs to know that a friend was typing four hours ago the way they need to know a message existed four hours ago.
Wavelink keeps presence as ephemeral, in-memory state on whichever shard a user is connected to, pushed to the small set of people currently viewing that conversation, with no durable write and no delivery guarantee at all. If a "typing" event gets dropped on a bad connection, nothing retries it, because the next keystroke will send a fresher one in a second anyway. Treating presence with the same durability machinery as messages would have meant paying queue and storage costs for information that is often stale before it would even finish writing.