Backend & Data
Database sharding — splitting data across machines
A single database server can be made surprisingly powerful — but there's a ceiling. At some point the data won't fit on one disk, or the write traffic outruns one machine's CPU. Indexes and read replicas buy time, but they don't help a firehose of writes. Sharding is the answer: split the data itself across many machines, each owning a slice. The elegance and the pain both come from one decision — how you decide which row goes where.
Partition by a shard key
You pick a shard key — a column like user_id — and a rule that maps each key to a shard. Every row lives on exactly one shard, and every query uses the key to find the right machine. Split it well and each shard holds a manageable share of the data and traffic; split it badly and you've built a distributed system with all of the cost and none of the benefit.
How the mapping is chosen
- Range-based — users 0–3M on A, 3–6M on B. Simple, and great for range scans, but prone to hot spots (all the newest, busiest users pile onto the last shard).
- Hash-based — hash the key and use that to pick a shard. Spreads load evenly and avoids hot spots, at the cost of efficient range queries.
- Directory-based — a lookup table maps keys to shards. Most flexible, but the directory itself becomes a thing to scale and protect.
Too much data for one machine? Split it. A shard key decides which machine each row lives on. Choosing that key well is the whole game — a bad one just moves your bottleneck onto one unlucky shard.
What sharding costs you
Sharding trades a scaling wall for a pile of new complexity. Queries that need data from many shards (a global sort, a join across users) become slow scatter-gather operations. Cross-shard transactions are hard — the clean all-or-nothing guarantee of a single database is exactly what you give up. And a badly chosen key causes hot shards: one machine melts while the others idle. Rebalancing later — moving rows to add a shard — is one of the genuinely painful operations in this business.
So shard when you must, not because it sounds impressive. Push a single machine, add read replicas and indexes first, and reach for sharding when writes or raw data size truly exceed one box. When you do, obsess over the shard key — it's the one decision that's brutal to change once you're live, and it dovetails with the same consistency-vs-availability trade-offs every distributed system has to make.