Skip to content
All writing

NOTES & IDEAS

Why My Real-Time App Broke When I Added a Second Server (and the Fix)

My live-comments feature worked perfectly on one server, then quietly broke for half my users on two. Here's the trap - and the small Redis fix - explained from scratch.

NestJSSocket.IORedisWebSocketsSystem Design

I built a blog called Writer's Heaven where comments show up live - you don't refresh the page, new comments just appear. It worked perfectly on my laptop. Then I ran it on two servers in production, and suddenly some people stopped seeing new comments. No error message. Nothing in the logs. It just quietly stopped working for half the users.

This turned out to be one of the most common traps in real-time apps. Let me walk you through it slowly - what was happening, why, and the small fix that solved it.

First, some background: how "live" comments work

To send live updates to a browser, you can't use normal requests (where the browser asks and the server answers). Instead you use a WebSocket - a permanent open connection between the browser and the server, like a phone call that stays connected. Either side can talk at any time.

I use a library called Socket.IO to manage these connections. Socket.IO has a handy feature called rooms: you can group connections together and message the whole group at once. So when someone opens "post 123," I put their connection into a room called post:123:

// when a reader opens a post, add their connection to that post's room
client.join(`post:${postId}`);

And when a new comment is created, I send it to everyone in that room:

// send the comment to every connection in room "post:123"
this.server.to(`post:${postId}`).emit('commentCreated', comment);

emit just means "send a message." So this line says: "send a commentCreated message, carrying the comment, to every connection in this post's room." On one server, this works flawlessly.

The problem: what "two servers" actually changes

When an app gets popular, one server isn't enough, so you run several copies of it (people call each copy an "instance" or just "a server"). A load balancer sits in front and spreads incoming users across them - some users land on Server A, some on Server B.

Here's the catch that breaks everything: each server keeps its list of rooms only in its own memory. The servers don't share that list with each other.

Picture this:

        Server A                          Server B
  room post:123 → [Alice]           room post:123 → [Bob]

Alice's phone call (WebSocket) is connected to Server A. Bob's is connected to Server B. They're both "in room post:123" - but A only knows about Alice, and B only knows about Bob. Neither server knows the other one has anybody in that room.

Now watch what happens when Bob comments:

  1. Bob's comment goes to Server B (the load balancer sent him there).
  2. Server B runs this.server.to('post:123').emit(...).
  3. Server B looks in its own memory for room post:123. It finds only Bob.
  4. So it sends the comment to Bob - and that's it.
  5. Alice is on Server A. Server B has no idea she exists. She gets nothing.

No crash, no error. Server B did its job correctly - it just didn't know about Alice. The comment never made it from one server to the other.

This is why it worked on my laptop: locally I only run one server, so everyone is always on the same one. The bug only appears once there's a second server.

The fix in one sentence

The servers need a way to tell each other "hey, a comment just happened." They need a shared messenger sitting between them. That messenger is Redis.

What is pub/sub?

Redis supports a pattern called pub/sub (publish / subscribe). The easiest way to picture it is a radio channel:

  • Publish = shout a message onto a channel.
  • Subscribe = tune into a channel and hear everything shouted on it.

The shouter doesn't need to know who's listening. It just broadcasts, and everyone tuned in hears it.

Now connect all your servers to the same Redis. When one server wants to send a comment, instead of only messaging its own users, it publishes the comment to Redis. Every server is subscribed, so every server hears it - and each one then forwards it to its own users.

So the new flow is:

Bob comments → Server B → publishes to Redis
                                  │
                  Redis shouts it to ALL servers
                                  │
        ┌─────────────────────────┴─────────────────────────┐
        ▼                                                     ▼
   Server A hears it                                   Server B hears it
   → sends to Alice ✅                                 → sends to Bob ✅

Redis is the shared messenger that lets the servers talk to each other.

The code, explained piece by piece

Socket.IO lets you swap in this Redis behavior using something called an adapter - think of it as "instructions for how messages travel between servers." We're going to give it Redis-based instructions. The official package for this is @socket.io/redis-adapter.

Here's the whole thing, and then I'll explain every part:

import { IoAdapter } from '@nestjs/platform-socket.io';
import { createAdapter } from '@socket.io/redis-adapter';
import { Redis } from 'ioredis';

export class RedisIoAdapter extends IoAdapter {
  private adapterConstructor: ReturnType<typeof createAdapter>;

  async connectToRedis(redisUrl: string): Promise<void> {
    const pubClient = new Redis(redisUrl);       // connection for publishing
    const subClient = pubClient.duplicate();     // connection for subscribing

    // wait until BOTH connections are actually connected
    await Promise.all([
      new Promise<void>((res) => pubClient.once('ready', () => res())),
      new Promise<void>((res) => subClient.once('ready', () => res())),
    ]);

    // build the Socket.IO-to-Redis bridge
    this.adapterConstructor = createAdapter(pubClient, subClient);
  }

  createIOServer(port: number, options?: ServerOptions): any {
    const server = super.createIOServer(port, options); // build the normal server
    server.adapter(this.adapterConstructor);            // plug Redis into it
    return server;
  }
}

extends IoAdapter - we take Socket.IO's default behavior and override just the parts we want to change. Everything we don't touch keeps working as normal.

Two Redis connections (pubClient and subClient). This looks odd - why two? It's a Redis quirk: once a connection starts listening (subscribing), Redis won't let it send (publish) on the same connection. It's busy listening. So you need a second connection that's free to publish. One talks, one listens. pubClient.duplicate() just makes a copy of the first connection's settings so you don't retype the address.

The scary await Promise.all([...]) block. Connecting to Redis takes a moment (it's over the network). If we start sending messages before the connection is ready, those first messages silently vanish. So we wait. Here's that block written out the long way so you can see what it means:

// "make a promise that finishes when pubClient is connected"
const waitForPub = new Promise<void>((resolve) => {
  pubClient.once('ready', () => resolve());
  // .once('ready', ...) = "when this connection fires its 'ready' event, run this"
  // resolve()           = "okay, this promise is now finished"
});

const waitForSub = new Promise<void>((resolve) => {
  subClient.once('ready', () => resolve());
});

// "pause here until BOTH promises are finished"
await Promise.all([waitForPub, waitForSub]);

The only trick here is turning an event ('ready') into something you can await. We wrap it in new Promise(...), and call resolve() inside the event listener to mean "done." Promise.all([...]) then waits for both connections at once. (In the original, res is just a short name for resolve.)

createAdapter(pubClient, subClient) - this builds the actual bridge between Socket.IO and Redis, using one connection to publish and the other to subscribe. We save it on the object (this.adapterConstructor) so the next method can use it.

The createIOServer method - this is where we actually attach Redis:

const server = super.createIOServer(port, options); // let the parent build the server normally
server.adapter(this.adapterConstructor);            // ← attach our Redis bridge
return server;

super.createIOServer(...) means "run the original version from the parent class" so we don't rebuild everything from scratch - we just take the server it makes and plug Redis into it.

Finally, switch it on when the app starts, before it begins accepting connections:

const redisAdapter = new RedisIoAdapter(app);
await redisAdapter.connectToRedis(process.env.REDIS_URL ?? 'redis://localhost:6379');
app.useWebSocketAdapter(redisAdapter);

The best part: nothing else changes

Here's what makes this elegant. Your original line -

this.server.to('post:123').emit('commentCreated', comment);
  • stays exactly the same. You don't rewrite any of your comment logic. The adapter works underneath it: it quietly turns that one emit into "publish to Redis, let every server hear it, and every server delivers to its own users." You wrote a ~30-line adapter once, and every real-time feature in the app instantly works across all servers.

What happens if Redis goes down?

Worth knowing, because it shows the trade-off. If Redis crashes, the live broadcasts stop working - servers can't talk to each other anymore. But the comments are still saved to the database, because saving and broadcasting are separate steps. So users lose the "appears instantly" magic, but nothing is lost: a page refresh loads the comment from the database. Real-time degrades to "refresh to see it," not "the app is broken." When Redis comes back, live updates resume on their own.

Takeaway

Real-time apps have a hidden assumption: that everyone is connected to the same server. That's true on your laptop and false in production. Once you run more than one server, each one only knows about its own users - so a message from one server never reaches users on another. Redis pub/sub fixes this by acting as a shared messenger that every server can publish to and listen on. Small change, and your app goes from "works on one server" to "works on any number of servers."