Told through a company I made up, called Northlane. It is a ride-hailing and delivery app, loosely modeled on how a company like Uber actually grows, with the name changed and the story simplified. The engineering problem and the numbers where I cite them are real.
Northlane is three years old. Rides in nine cities, food delivery layered on top, around 400 backend services owned by maybe 30 teams. On a Friday night during a big local event, the payments service started double charging riders on a small percentage of trips.
Nobody on call could tell what percentage, or which cities, or when it started. The payments service alone ran on 60 machines, and each one wrote its own log file. Finding every line for one affected trip meant SSHing into boxes one at a time and grepping for a trip id, hoping you picked the right box first. It took the on-call engineer close to two hours to confirm the bug and scope it. The fix itself took twenty minutes.
The postmortem had one real finding. The bug was ordinary, a retry that did not check whether the first charge had already gone through. What made it expensive was that nobody could see it happening. Northlane had metrics and dashboards, but nothing that let you ask "show me everything that happened to trip 88213" and get an answer in under an hour.
The team that came out of that postmortem wrote down what they actually needed before picking any tools. Two lists.
Everything below is Northlane building toward that list, one piece at a time, and what changed each time a piece landed.
First decision: how does a line of text get off a machine and into a place where it can be searched.
A wrinkle they hit almost immediately: on their newer hosts, systemd's journal is single threaded, and their busiest service, the one that streams live driver locations, was fast enough to back up behind it. Cloudflare hit the identical problem and has its own agent read from a Unix socket for high volume services instead of going through the journal at all. Northlane copied that.
With the agent in place, the two hour SSH hunt from the incident became a search box. Not a fast search yet, and not a cheap one, but a search.
The agent shipping logs somewhere is not the whole design. The real question is what happens when that somewhere is slow or unreachable, which happens more often than anyone likes to admit.
They picked a bounded local disk buffer on every machine. It absorbs a queue outage without the app ever noticing, and it survives the agent itself restarting, which a memory-only buffer would not. Blocking the service was ruled out outright, since that is the one outcome that turns a logging hiccup into a ride-booking outage.
Three months later, a routine upgrade to the message queue went wrong and took it down for twenty minutes. Every service kept running. The buffer filled, held, and drained once the queue came back. Nobody outside the platform team noticed. That was the first time the disk buffer earned its cost.
A different kind of failure showed up a few months after that. A bad deploy in the driver-matching service put it into a retry loop, and its log volume jumped to fifty times normal. That volume shared a pipe with every other service, so payments and trip-tracking logs started arriving late during the exact window someone would have wanted to look at them.
Cloudflare, who deal with this at a much bigger scale, use max-min fair share: every stream gets an allocation, streams under it pass through untouched, and only the ones over it get squeezed to fit what capacity remains. Northlane implemented a simpler version of the same idea.
The next time a service misbehaved this way, payments logs kept arriving on time, and the on-call engineer for driver matching had a smaller, sampled view of their own mess, which was still enough to diagnose it.
Sampling brings its own problem: once a stream is downsampled, any count or average built from it has to be corrected back up, or your dashboards quietly lie. The fix is to store the sampling rate alongside each kept event, so a count of 1 at a 1-in-100 rate is understood as roughly 100.
This is where Northlane's story stops and a real one is worth telling, because Cloudflare wrote up a mistake in exactly this step that is worth knowing about before you make it yourself. They sampled by keeping every nth event in arrival order, which looks the same as random sampling and is not. Their web responses tend to arrive largest first within a burst, since a full page renders before the small cached assets behind it. Taking every nth event kept the large responses more often than the small ones, and every size estimate downstream came out too high, quietly, until someone checked.
Northlane's engineers read that writeup while building their own sampler and shuffled before sampling from the start, before it ever became their own bug to find.
Between the agents and storage sits a queue, Kafka in Northlane's case, and it does one job well: it lets logs be produced faster than they can be indexed, for a while, without anything falling over.
The number worth arguing about is retention, since it is your outage budget in disk space. Northlane sized theirs around how long it would realistically take to fix a broken consumer at three in the morning, then added a wide margin, because that estimate is always optimistic when you make it in daylight. They landed on six hours, close to what Cloudflare has said publicly they target for the same reason.
Partitioning came with a smaller cost they didn't expect. They partition by service and host, same as Cloudflare, and their transport protocol only carries millisecond timestamps. A service logging faster than that has no guaranteed order within a millisecond, so "ordered logs" became "logs that are ordered almost all of the time," which they wrote down and told people, rather than letting someone discover it during an audit.
The first storage layer Northlane stood up was Elasticsearch, because it was familiar and the full text search was genuinely good. Within a year the cluster was the single largest infrastructure line item the platform team owned, ahead of the databases actually running the product.
| Approach | Indexes | Stored for 100 GB/day raw | Weak at |
|---|---|---|---|
| Inverted index Elasticsearch, OpenSearch |
every token | ~500 GB | write throughput, storage cost |
| Label index Loki |
labels only | ~30 GB plus a small index | high cardinality, ad hoc search |
| Columnar ClickHouse, Parquet on S3 |
sparse, sorted | ~30 to 50 GB | true full text, schema drift |
Most of what people actually asked of the system was filter by service and time range, then aggregate, which a columnar store does well without paying for a full inverted index on every write. Northlane moved onto ClickHouse for anything past a week old and kept a small, short-lived Elasticsearch index for the truly recent, truly free-text searches. Cloudflare made close to the same move, off a 90-node Elasticsearch cluster onto ClickHouse, and Uber did as well. The storage bill dropped by more than half.
A well-meaning engineer added the rider's user id as a label on every trip log line, reasoning that it would make "show me this rider's history" a fast lookup. It shipped on a Thursday. By Friday the label index had gone from a few hundred stable values to several million, and query latency across the whole dashboard had climbed noticeably.
It was caught before it became an incident, but only because someone happened to check the dashboard's load time that week. The fix was to move the user id into the line body instead of a label, where finding it costs a scan rather than an index entry. The rule that came out of it: a label is for a field with a small, known set of values, service, region, level. Anything that can grow without bound stays in the body.
Most of what Northlane stores is old, and most of what gets read is recent. One tier of storage for everything was either too expensive to hold years of history or too slow for the searches people ran fifty times a day.
Debug logs age out at seven days. Payment logs, the ones the whole project started over, sit in cold object storage for seven years to satisfy the regulator, and are read maybe once a quarter. Same pipeline, two completely different retention rules, set on purpose rather than by default.
A short list of things Northlane learned by living with the system rather than by designing it.
Northlane is invented, sized to be relatable rather than impressive. It's worth knowing what this looks like at the far end of the same curve, at a company that publishes real numbers.
Cloudflare's logging pipeline moves close to a million log lines a second across their network, and the wider data pipeline behind their analytics was handling 706 million events a second and 107 GiB/s of compressed data as of December 2024, about 100 times their 2018 figure. In 2024 they also replaced syslog-ng, which had run their collection layer for years, with the OpenTelemetry Collector. None of their stated reasons were about speed. It is written in Go instead of C, so more of their own engineers can work on it, it was easier to build against their internal cryptography libraries, it exposes its own Prometheus metrics, and it let them stop running separate agents for logs and for traces. A mature team's biggest infrastructure migration in years, justified almost entirely by how much easier the result is to operate.
That is the same order of priority Northlane's much smaller team ended up with: correctness and operability first, raw throughput a distant second, because throughput was rarely what was actually breaking.