Scaling PostgreSQL to Power 800 Million ChatGPT Users

Architecture Evolution: From Monolithic to Highly Available Sharded Clusters

OpenAI did not adopt sharding or migrate to NewSQL. Instead, it built a horizontally scalable state store around PostgreSQL 15+, using it exclusively for critical OLTP paths—including user session state, conversation metadata, API key management, and usage quota tracking—while rigorously avoiding long-running transactions and complex JOINs.

Four Core Scaling Strategies

• Tiered Replica Routing: Deployed geographically distributed read-only replicas (across AZs and regions), dynamically routing read traffic based on latency and load; the primary cluster handles only writes and strongly consistent operations (e.g., quota decrements).
• Multi-Level Caching: Layer 1: Redis Cluster with LRU + TTL for high-frequency session metadata; Layer 2: in-process Guava Cache for user-level configs and token bindings; Layer 3: PgBouncer connection pooling with transaction-scoped reuse.
• Fine-Grained Rate Limiting: Leveraged pg_stat_statements and OpenAI’s internal metrics pipeline to enforce real-time per-user, per-API-key, per-model, and per-token rate limits at the application layer—with thresholds synced to shared memory (avoiding per-request DB lookups).
• Physical Workload Isolation: Split services by SLO into isolated PostgreSQL clusters: chat_sessions (low-latency, high-QPS), billing (strong consistency, low-QPS), and audit_logs (write-heavy, append-only)—each with distinct hardware profiles and WAL configurations.

Key Metrics & Constraints

• Peak write throughput: >120,000 TPS on primary;
• P99 read latency on replicas: <12ms (cross-region <45ms);
• Cache hit rates: >92% at Redis layer, >87% at application layer;
• Max connections per cluster: hard-capped at 32,000 via PgBouncer to prevent connection storms.

Deliberately Excluded Approaches

• Refused storing full conversation history in PostgreSQL (offloaded to TimescaleDB + object storage archiving);
• Rejected logical replication in favor of physical replication (due to WAL parsing overhead and consistency risks);
• Avoided external distributed transaction coordinators (e.g., CockroachDB or Vitess), preserving single-cluster ACID semantics.