Scaling Out & Statelessness
Why your server must not remember anything locally if you want a second one.
40 minDifficulty 3/5scaling · architectureAI-writtenWritten by a model on 16 August 2026 and not yet read by a person. Checked automatically: schema, the pedagogical rules the hand-written material is held to, and every diagram parsed for real.
Before this
Why this exists
A server stores each user's shopping cart in a plain in-memory JavaScript object. It works perfectly — until traffic grows and a second server is added behind a load balancer, and now half of every user's requests randomly land on the server that has NEVER SEEN their cart, because it lives only in the first server's memory. Scaling out doesn't just require more machines; it requires the application to stop assuming there's only one.
The mental model
Scaling VERTICALLY means making one machine bigger — more CPU, more RAM. Scaling HORIZONTALLY means adding more machines and spreading the work across them. Vertical scaling has a hard ceiling (there's a biggest machine money can buy) and doesn't survive that one machine failing; horizontal scaling has no such ceiling, but only works if any request can genuinely be handled by any server — which requires the server itself to hold no unique, irreplaceable state.
How it works
Statelessness is the precondition, not a nice-to-have
A stateless server treats every request as fully self-contained — everything needed to handle it (identity from a token, data from a shared database) is either in the request itself or fetched from a shared external store, never held in that specific server's own memory between requests. This is what makes it safe for a load balancer to send any request to any server interchangeably.
Session data moves from local memory to a shared store
Instead of `req.session = {...}` living in one server's process memory, session data is stored in something EVERY server instance can read and write — Redis, a database, or encoded entirely into a signed token the client holds (a JWT) and presents with each request. Any of these approaches lets any server correctly handle any user's request, which local in-memory storage structurally cannot.
Uploaded files and other local disk writes have the same problem as in-memory state
A server that writes an uploaded file to its own local disk has the same issue as one holding session state in memory — a request to DOWNLOAD that file, routed to a different server instance, finds nothing there. Files need to go to shared storage (like object storage such as S3), not any individual server's local filesystem, for the same reason session data needs to move to a shared store.
Horizontal scaling also buys resilience, not just capacity
With ten stateless servers behind a load balancer, one crashing means 10% capacity loss and health checks routing around it — with one big vertically-scaled server, that same crash means 100% downtime. This resilience benefit is often as valuable as the raw capacity gain, and it's a property vertical scaling can never provide, since there's only ever one machine to fail.
The mechanism
Requests arrive at a load balancer, which distributes them across a pool of identical, stateless server instances. Each server, having no local state of its own, reads any data it needs (session, user data, uploaded files) from shared external stores — a database, a cache like Redis, object storage — that every instance can access equally. Adding more capacity means adding more identical instances to the pool, with no coordination required between them, because none of them hold anything unique.
What people get wrong
- Horizontal scaling is just about adding more servers — the application code doesn't need to change.
- If the application holds any state locally (in-memory sessions, uploaded files on local disk), adding servers without addressing that state actively BREAKS things, since requests can now land on a server that never saw the relevant state. This is the most common horizontal-scaling failure mode: it works fine with one server in development and testing, then breaks unpredictably the moment a second instance is added in production.
- Vertical scaling is always a worse choice than horizontal scaling.
- Vertical scaling is simpler (no statelessness requirement, no load balancer needed) and genuinely appropriate for workloads that don't need to scale beyond what a single powerful machine can handle, or where the engineering cost of going stateless isn't justified yet. Treating horizontal scaling as automatically superior ignores the real complexity cost it introduces — statelessness, distributed session handling, load balancing — which isn't worth paying before it's actually needed.
- Storing sessions in a JWT (rather than server-side) automatically makes an application horizontally scalable with no other changes.
- JWTs solve the SESSION statelessness problem specifically, but any other local state — uploaded files on disk, in-memory caches unique to one instance, WebSocket connections tied to a specific server — still needs its own separate solution for true horizontal scalability. Fixing one source of state (sessions) can create false confidence that the whole application is now stateless, when other state sources are just as capable of breaking horizontal scaling.
When not to use it
- The workload has a hard ceiling that's well within what a single, sufficiently powerful machine can handle, and simplicity matters more than resilience.
- Vertical scaling — it avoids the real complexity cost of statelessness and load balancing when that complexity isn't actually needed yet.
- The application genuinely needs a specific request to always reach the same server, like a stateful WebSocket connection.
- Sticky routing for that specific connection type (see `load-balancing`), while still keeping everything else in the application stateless where possible.
Terms
- Vertical scaling
- — Increasing a single machine's capacity (more CPU, RAM) rather than adding more machines.
- Horizontal scaling
- — Adding more machines and distributing work across them, requiring the application to be stateless so any instance can handle any request.
- Statelessness
- — A server design where no request depends on data held only in that specific server's own memory or local disk between requests.
- Shared store
- — External storage (a database, Redis, object storage) accessible to every server instance equally, used to hold state that would otherwise live locally on one server.
In an interview
An application works fine with one server instance, but users randomly get logged out after a second instance is added behind a load balancer. What's the likely cause?
- session data is being stored in the server's local memory, not a shared store
- when a user's subsequent request lands on the OTHER instance, that instance has no record of their session
- fix: move session storage to something all instances share (Redis, a database) or use a self-contained token like a JWT
Can you recall it?
Why is statelessness a precondition for horizontal scaling rather than just a nice architectural property?