System Design: Google Drive — Collaborative Document Editing

Introduction

Google Drive is one of the most sophisticated distributed systems ever built. It solves a fundamental tension: strong consistency (all users see the same document, no edits lost) and low latency (typing must feel instant). Traditional systems cannot have both. Collaborative editing solves this with operational transformation and optimistic client updates.

Step 1: Requirements

Functional:

1. Create, edit, delete documents
2. Real-time collaborative editing — multiple users editing simultaneously
3. Presence — see who else is in the document, their cursor, selection
4. Version history — view and restore past versions
5. File storage — attachments, exports (PDF, DOCX)
6. Sharing and permissions
7. Full-text search across documents

Non-functional:

1. Low latency — typing feels instant (< 50ms perceived)
2. Strong consistency — all clients converge to the same document state
3. High availability — documents always accessible
4. Highly scalable — 100M+ daily active users
5. Durable — no edit ever lost

Step 2: Scale Estimation

100M daily active users
Average 10 documents per user
Average document size: 1 MB

Total storage: 100M × 10 × 1 MB = 1 Petabyte

Reads: 50M document opens/day = ~600 reads/second

Writes (the hard part):
10 edits/second per actively edited document
1M concurrently active documents
= 10M operations/second globally

Refined estimate:
→ Not all 1M docs are heavily edited
→ ~10% heavily edited = 100k docs × 10 edits/sec = 1M ops/sec
→ Range: 1–10M operations/second

Core insight: this is an extremely write-heavy system. Storage is 1 PB, but the architecture challenge is 10M character-level operations per second, not storage size.

Step 3: Why Not Store the Full Document Every Time?

Naive approach — save full document on every edit:
Document: 100 KB
1,000 edits in one session = 100 KB × 1,000 = 100 MB per session ❌

At scale:
1M active docs × 1,000 edits × 100 KB = 100 TB/day ❌

The right approach: store only what changed — an operation log — plus periodic snapshots.

Step 4: Operation Logs and Snapshots

Operation Log

Instead of saving the full document, store each edit as a small operation:

Op 1: Insert("H", position=0)            → 20 bytes
Op 2: Insert("e", position=1)            → 20 bytes
Op 3: Insert("Hello World", position=0)  → 30 bytes
Op 4: Delete(position=6, length=5)       → 25 bytes
Op 5: Insert("Beautiful ", position=6)   → 30 bytes

Each operation: 20–50 bytes
Full document save: 100 KB

Storage savings: ~3,000x smaller ✅
Complete edit history preserved ✅

Reconstructing a document:

Start from empty
Apply Op 1 → "H"
Apply Op 2 → "He"
...
Apply all operations in order → current state

Problem: 10,000 operations → replay all 10,000 on every open → too slow ❌

Snapshots

A snapshot is a full document state captured at a point in time.

Strategy:
→ Every 100 operations, take a snapshot
→ Store snapshot + operations after snapshot

Document has 10,450 operations
Latest snapshot at operation 10,400

To reconstruct:
→ Load snapshot at op 10,400 (one read)
→ Apply ops 10,401–10,450 (only 50 operations)
→ Current state ready in milliseconds ✅

Storage Architecture

operation_logs (Cassandra):
  document_id   → partition key
  operation_seq → sort key (auto increment)
  operation_type, position, content
  user_id, timestamp, client_version

Why Cassandra:
→ Up to 10M operations/second
→ Partition key = document_id
→ Append-only, never updated

snapshots (S3):
  document_id, snapshot_seq, content, s3_path

snapshot_metadata (PostgreSQL):
  document_id, snapshot_seq, s3_path, created_at
  → Quick lookup: "latest snapshot for doc X?"

Snapshot Trigger Logic

After every 100 operations for a document:
→ Snapshot Worker (Kafka consumer) reconstructs current state
→ Saves to S3
→ Records metadata in PostgreSQL
→ Deletes operation logs older than snapshot (no longer needed)

Storage lifecycle:
Operations after latest snapshot → Cassandra (hot)
Recent snapshots → S3 Standard
Old snapshots → S3 Infrequent Access
Very old versions → S3 Glacier

Version History

Never delete snapshots — keep all of them
Each snapshot tagged with timestamp

To view "document last Tuesday":
→ Load snapshot closest to that time
→ Apply operations between snapshot and requested time
→ Render that version

Named versions:
→ User names a version: "Final draft before review"
→ Stored as special snapshot, never auto-deleted

Step 5: Presence

Presence = real-time awareness of who else is in the document right now.

When you open a Google Doc:

→ Colored avatars at top right
→ "Raj is viewing", "Priya is editing section 3"
→ Priya's colored cursor moving in real time
→ What text she's currently selecting

What presence includes:

1. User list — who is currently in the document
2. Cursor position — where each user’s cursor is
3. Selection — what text each user has selected
4. Typing indicator — is someone currently typing
5. User color — unique color per user for cursor and selections

How it works:

User opens document
→ WebSocket connection to Presence Service
→ Presence Service records in Redis (TTL 30 seconds):

Key: "presence:doc123"
{
  "u456": { name: "Raj", cursor: {5,12}, color: "#FF5733" },
  "u789": { name: "Priya", cursor: {12,4}, color: "#33FF57" }
}

Raj moves cursor:
→ Client sends update via WebSocket
→ Presence Service updates Redis
→ Broadcasts to all other users in document
→ Other clients render Raj's cursor at new position

Every 10 seconds: heartbeat refreshes TTL
No heartbeat → TTL expires → user considered gone
→ Automatic cleanup without explicit disconnect handling

Presence is ephemeral state — Redis with TTL, not a database. If presence data is lost, users reconnect and repopulate it. Availability over consistency.

Step 6: Operational Transformation (OT)

The hardest concept in collaborative editing.

The Problem

Document: "Hello World"
           0123456789...

Raj:   Insert "Beautiful " at position 6 → "Hello Beautiful World"
Priya: Delete "World" (positions 6–10)   → "Hello "

Both happen simultaneously, both based on the same original state.
How do you apply both without losing either?

Naive Approach Fails

Apply Raj first:  "Hello World" → "Hello Beautiful World"
Apply Priya next: Delete positions 6–10 → deletes "Beaut"
Result: "Hello iful World" ❌

Priya intended to delete "World"
After Raj's insert, "World" moved to positions 17–21
Priya's operation used stale position information

OT Solution

Rule: if operation B was generated concurrently with operation A, transform B against A before applying it.

Transform Priya's delete against Raj's insert:
→ Raj inserted 10 characters at position 6
→ Priya's delete target was positions 6–10
→ Shift Priya's delete by +10 characters
→ Transformed delete: positions 16–20

Apply both:
"Hello World"
→ Apply Raj: "Hello Beautiful World"
→ Apply transformed Priya: delete 16–20 → "Hello Beautiful " ✅

OT Rules Summary

Insert vs Insert:  A before B's position → shift B forward by A's length
Insert vs Delete:  A before B's delete range → shift B's range forward
Delete vs Insert:  A deletes before B's position → shift B backward
Delete vs Delete:  Overlapping ranges → adjust B to delete only what A didn't

OT in Practice

Raj sends:   Op_A = Insert("Beautiful ", pos=6), version=v5
Priya sends: Op_B = Delete(pos=6, len=5),         version=v5

Server receives both:
→ Apply Op_A first → document becomes v6
→ Transform Op_B against Op_A → Op_B' = Delete(pos=16, len=5)
→ Apply Op_B' → document becomes v7

Broadcast:
→ Send Op_B' to Raj (transform and apply locally)
→ Send Op_A to Priya (transform and apply locally)
→ Both clients at v7 with same content ✅

Step 7: Optimistic Client Updates

What makes Google Docs feel instant.

Without Optimistic Updates

Raj types "H":
→ Send operation to server
→ Wait for confirmation (50–200ms round trip)
→ Display "H"

60 words/minute = ~5 characters/second
5 × 200ms = 1 second lag behind typing ❌

With Optimistic Updates

Raj types "H":
→ IMMEDIATELY display "H" locally
→ Simultaneously send operation to server
→ User sees instant response ✅

Server confirms:
→ If agrees → nothing changes, user never noticed

Server transforms operation:
→ Client rolls back local state
→ Applies server-confirmed state
→ Happens in milliseconds — user barely notices

Three States

State 1 — Committed:
Operations confirmed by server. Definitive truth. Shared by all clients.

State 2 — In-Flight:
Sent to server, not yet confirmed. Applied locally optimistically.
Might need transformation when server responds.

State 3 — Pending:
Not yet sent (still typing). Applied locally. Will be sent soon.

Client always displays: Committed + In-Flight + Pending
Server eventually catches up and confirms.

Conflict Resolution During Optimistic Updates

While Raj has In-Flight operation:
Server sends transformed operation from Priya

Raj's client:
1. Roll back In-Flight and Pending operations
2. Apply Priya's server-confirmed operation
3. Re-apply Raj's operations transformed against Priya's
4. Update display

Happens in milliseconds — tiny flicker at most, usually invisible.

Step 8: Handling 10M Operations Per Second

The Write Pipeline — Three Parallel Paths

When Raj types a character, the Document Service does three things simultaneously — not one sequential chain through Kafka.

[Raj types "H"]
      ↓
Local OT engine applies optimistically — Raj sees "H" instantly ✅
      ↓
WebSocket → Document Service
      ↓
Document Service:
1. INCR Redis seq counter → seq: 10451 (~0.5ms)
2. Apply OT transformation if needed (~0.5ms)
3. Simultaneously:

   Path 1 — Broadcast (Redis PubSub, fast):
   → Publish to Redis channel "doc:doc123"
   → All WebSocket servers with users in doc123 receive (~1ms)
   → They push to connected collaborators
   → Priya sees "H" in ~2–5ms total ✅

   Path 2 — Persistence (Kafka, async):
   → Publish to Kafka "document.operations"
   → Operation Persister batches → Cassandra (~100ms)
   → Snapshot Worker monitors operation count
   → Does not slow down broadcast

   Path 3 — ACK to Raj:
   → Return sequence number (~2ms)
   → Raj's operation moves from In-Flight → Committed

Why not broadcast through Kafka? Kafka adds 5–15ms and is built for durability, not sub-millisecond signaling. Priya would see updates in ~10–50ms — acceptable, but unnecessary. Google Docs uses a fast path (Redis PubSub) for collaborators and a slow path (Kafka) for persistence.

Why Redis PubSub for Broadcast — Not Kafka

This is the same pattern from the Notification System design (Lesson 11):

Lesson 11 — Notification System:
Notification Service → Redis PubSub "user:u456" → SSE Server → User

Google Drive:
Document Service → Redis PubSub "doc:doc123" → WebSocket Servers → Collaborators

Same pattern, different channel granularity:

Lesson 11: channel per USER   (one user per channel)
Google Drive: channel per DOCUMENT (all collaborators on one channel)

The routing problem:

1,000 WebSocket servers handle 50M connections
Raj → WebSocket Server 347
Priya → WebSocket Server 892
Kumar → WebSocket Server 156

Document Service receives Raj's edit.
Which servers hold Priya and Kumar? Document Service doesn't know ❌

Redis PubSub solves it:

When user opens document:
→ Their WebSocket server subscribes to Redis channel "doc:doc123"

Raj types "H":
→ Document Service publishes to "doc:doc123":
  { operation, seq: 10451, userId: "raj", excludeUserId: "raj" }

Redis PubSub delivers to ALL subscribed WebSocket servers:
→ Server 347 (Raj) — skips, excluded
→ Server 892 → pushes to Priya ✅
→ Server 156 → pushes to Kumar ✅

If Priya misses one operation, her client detects the gap via sequence numbers and requests missing ops from the server. Message loss on PubSub is acceptable; persistence is guaranteed by Kafka.

Kafka Consumers (Persistence Only)

Consumer 1 — Operation Persister:
→ Batch writes to Cassandra (100 ops per batch)
→ 10M individual writes → 100k batch writes ✅

Consumer 2 — Snapshot Trigger:
→ Count operations per document
→ Every 100 operations → trigger snapshot creation

Batching — Key to Scale

10M operations/second hitting Cassandra directly → struggles ❌

Batch 100 operations for same document → single Cassandra write
10M / 100 = 100k Cassandra writes/second → handles easily ✅

Batching adds ~100ms delay to persistence
→ Acceptable — client already has optimistic update
→ User never waits for persistence

Scaling Document Service — The Real Bottleneck

Redis being fast does not help if Document Service is the choke point. Every operation passes through Document Service before it reaches Redis.

Redis is not embedded in Document Service. They are separate fleets communicating over the network:

Wrong mental model:
  Document Service EC2 runs Redis inside it

Correct:
  Document Service fleet (200 × c5.2xlarge)
  Redis Cluster fleet (20 × r5.2xlarge, memory-optimized)
  Each fleet scaled independently

Step 1 — Scale Document Service horizontally

Single EC2 (Document Service): ~50,000 WebSocket messages/second
10M ops/sec ÷ 50,000 = 200 instances needed

Load balancer routes by document:
hash(documentId) % 200 → which instance handles this doc

Why same document → same instance?
→ Random routing: two instances process seq 5 simultaneously → conflict ❌
→ Sticky routing: Instance 7 handles ALL ops for doc123 → natural ordering ✅

Step 2 — Each instance handles its share

200 instances × 50,000 ops/sec = 10M ops/sec total
No Document Service bottleneck ✅

Step 3 — Redis Cluster handles aggregate Redis load

Each operation:
→ 1 Redis INCR (sequence number)
→ 1 Redis PUBLISH (broadcast)
= 2 Redis calls per operation

10M ops × 2 = 20M Redis calls/sec
Single Redis node: ~1M ops/sec
→ Redis Cluster with 20 nodes
→ hash(documentId) % 20 → shard sequence counters and PubSub
→ ~1M ops/sec per node ✅

Per operation on one Document Service instance:

Step 1: Redis Cluster INCR     → seq number (~0.5ms)
Step 2: Apply OT               → in memory (~0.5ms)
Step 3: Redis Cluster PUBLISH  → collaborators (~1ms)
Step 4: Kafka PRODUCE          → async, non-blocking
Step 5: ACK to client          → ~2ms total

Sequence Number Assignment

Operations must be applied in exact sequence
Otherwise documents diverge between clients

Redis atomic counter per document:
Key: "seq:docId" → INCR → next sequence number

10M INCRs/second:
Single Redis node: ~1M ops/sec → not enough
Redis Cluster sharded by document_id:
hash(documentId) % 100 → 100 nodes × 100k INCRs = 10M/sec ✅

Multiple Users Reading Simultaneously

Reads have no conflict.
Each client has local copy of document.
Server broadcasts operations to all clients.
Each client applies operations locally via OT.
All clients converge to same state.

Step 9: Complete Architecture

[CLIENT LAYER]
Browser/Mobile
→ Local OT engine
→ Optimistic updates
→ WebSocket connection
→ Offline operation queue

[REAL TIME LAYER]
WebSocket Gateway (1000 servers):
→ 50,000 connections each = 50M concurrent connections

Presence Service:
→ Who is in each document
→ Redis for presence state (TTL)
→ Broadcasts cursor/selection updates

[OPERATION PROCESSING LAYER]
Load Balancer:
→ hash(documentId) → routes to correct Document Service instance

Document Service (200 instances, ~50k ops/sec each):
→ Receives operations for its assigned documents
→ Assigns sequence numbers (Redis Cluster)
→ Applies OT transformation (in memory)
→ Path 1: Redis PubSub "doc:docId" → WebSocket servers (broadcast)
→ Path 2: Kafka "document.operations" → persistence (async)
→ Path 3: ACK to sender (~2ms)

Kafka (persistence only):
→ Topic: "document.operations"
→ Partitioned by document_id
→ All ops for same document → same partition → ordering guaranteed

[PERSISTENCE LAYER]
Operation Persister:
→ Batch writes to Cassandra (100 ops/batch)

Snapshot Worker:
→ Every 100 operations → snapshot to S3
→ Metadata in PostgreSQL

[STORAGE LAYER]
Redis Cluster (20 nodes, separate fleet) → seq counters, presence, PubSub, active doc cache
Cassandra        → operation logs (sharded by document_id)
PostgreSQL       → metadata, permissions, snapshot metadata
S3               → snapshots, attachments, exports
Elasticsearch    → full-text search
CDN              → static assets, thumbnails, exported files

Step 10: Conflict Resolution — Three Levels

Level 1 — OT (Operational Transformation):
→ Character-level concurrent edits
→ Both edits preserved
→ Real-time

Level 2 — Sequence Numbers:
→ Global ordering guarantee
→ Every operation has unique sequence number
→ All clients apply in same order
→ Documents converge

Level 3 — Last Write Wins (metadata only):
→ Document title, settings
→ Not character edits
→ Most recent change wins

Everything Connects

Key Takeaways

1. Operation log + snapshots — store 20-byte operations, not 100 KB full saves. Snapshot every 100 ops so reconstruction is fast. ~3,000x storage savings.
2. OT resolves concurrent edits — transform operations against each other so both edits are preserved. Positions shift when concurrent inserts/deletes happen.
3. Optimistic updates decouple perceived latency from server speed — display locally first, reconcile when server responds. Three states: Committed, In-Flight, Pending.
4. Presence is ephemeral — WebSocket + Redis with TTL. Cursors, selections, user list. Not stored in a database.
5. 10M ops/sec needs batching — Kafka for persistence (batched Cassandra writes), Redis PubSub for broadcast (~2–5ms to collaborators). Redis Cluster for sequence assignment. User never waits thanks to optimistic updates.
6. Redis PubSub + Kafka, not Kafka alone — PubSub for real-time collaborator updates (Lesson 11 pattern). Kafka for durability and snapshots (Lesson 8 pattern). Each tool does what it’s best at.
7. The core tension — strong consistency and low latency seem contradictory. OT + optimistic updates solve it: eventually consistent state that feels immediately responsive.

Answering the Core Questions

Part of the system design series.