OpenAI Reveals Codex Agent Loop and PostgreSQL Scaling for 800 Million Users

OpenAI has disclosed the underlying "Agent Loop" architecture powering its Codex programming assistant and detailed how a single-node PostgreSQL database, supported by 50 read replicas, manages peak traffic from 800 million global users. These revelations follow the emergence of Anthropic's Claude Code, an AI assistant capable of reading, modifying, and testing code within a terminal environment, which has garnered significant attention in the AI programming community.

Sam Altman, CEO of OpenAI, speaking at a conference, representing the company's official announcements.

OpenAI's disclosures, shared via its official blog and CEO Sam Altman on X, directly address the escalating competition in AI programming tools.

Understanding the Agent Loop

The Agent Loop is described by OpenAI as the operational "brain" behind tools like Codex CLI, enabling them to understand user intent, interact with models, and execute tasks. Unlike simple question-and-answer systems, the Agent Loop functions as a closed system incorporating observation, thought, action, and feedback.

Diagram illustrating the 'Agent Loop' with interconnected phases: Observation, Thought, Action, and Feedback.

For instance, when a user inputs a command such as "Add an architecture diagram to the project's README.md," the process unfolds in several steps:

1. Prompt Construction: Codex first builds a detailed prompt, informing the model of its identity, available tools (e.g., shell commands, file operations), environmental context (current directory, shell), and the user's specific instruction.

2. Model Inference: The constructed prompt is sent to OpenAI's ResponsesAPI. The model then processes this information, determining the necessary action, such as executing cat README.md to inspect the current README file.

3. Tool Call: Codex executes the model's chosen command locally, retrieving the content of README.md.

4. Result Feedback: The output from the command is appended to the original prompt and fed back to the model.

5. Loop Continuation: The model re-evaluates the updated prompt, potentially generating a diagram or writing to the file, and continues this observe-think-act-feedback cycle until the task is completed and a "done" message is issued.

This iterative process allows the AI to plan its actions, correct errors, and verify results, transforming it from a conversational agent into an independent problem-solver.

Optimizing Agent Performance

OpenAI highlighted two key optimizations crucial for Agent development:

1. Cost Reduction through Prompt Caching: Agent Loops typically resend the entire conversation history with each iteration, leading to quadratically increasing costs for longer interactions. OpenAI employs a prompt caching strategy that reuses previously computed results for unchanged portions of the prompt (system instructions, tool definitions, historical conversations). This reduces the cost growth from quadratic to linear. However, any modification to the prompt prefix—such as changing models, altering permissions, or reordering tool lists—invalidates the cache. OpenAI acknowledged an early bug where unstable tool list ordering led to frequent cache invalidation.

2. Context Window Management via Compaction: Large models have limited context windows. Extensive inputs, like large log files, can quickly fill this window, causing the model to "forget" earlier information. To address this, Codex uses a /responses/compact interface to summarize conversation history when token limits are approached. This "compaction" process generates encrypted_content that retains the model's implicit understanding of the original conversation, allowing the Agent to maintain its intelligence over extended tasks without losing critical details.

Abstract visualization of a central database pillar with 50 radiating data streams, symbolizing massive PostgreSQL scaling.

PostgreSQL's Extreme Scaling

OpenAI also revealed that its infrastructure, supporting 800 million ChatGPT users globally and millions of queries per second, relies on a single-node PostgreSQL primary database complemented by 50 read replicas. This architecture, according to OpenAI engineers, leverages several key technologies:

• PgBouncer Connection Pool Proxy: This proxy significantly reduces database connection overhead, decreasing average connection establishment time from 50ms to 5ms.

• Cache Locking Mechanism: To prevent database overload during cache misses, only one request is permitted to query the database and repopulate the cache, while other requests wait. This avoids "cache avalanches."

• Cross-Region Cascading Replication: Read requests are distributed globally across replicas, optimizing read performance.

The core principle behind this design is read-write separation and extreme optimization of the read path, as read requests in applications like ChatGPT vastly outnumber write requests.

Visual metaphor showing a wide, busy path for read requests and a narrow, steady path for write requests, illustrating database separation.

Additional optimizations include:

• Query Optimization and Load Isolation: Complex queries were simplified, and intricate logic was shifted to the application layer. Requests are also categorized into high-priority and low-priority streams, handled by dedicated instances to prevent performance degradation.

• High Availability and Failover: The primary database operates in High Availability (HA) mode with a hot standby. All read traffic is directed to replicas, ensuring read-only service availability even if the primary database fails.

Despite these optimizations, OpenAI acknowledges that this architecture is approaching its physical limits due to PostgreSQL's Multi-Version Concurrency Control (MVCC) limitations, which can lead to write and read amplification, and Write-Ahead Log (WAL) replication pressure, which increases with the number of replicas.

To overcome these challenges, OpenAI is migrating sharded, high-write workloads to distributed systems like Azure Cosmos DB and testing cascading replication to support over 100 replicas by allowing intermediate replicas to forward WAL to downstream replicas.

Competitive Landscape

OpenAI's detailed disclosures on the Agent Loop and its PostgreSQL scaling strategy are seen as a direct response to the growing capabilities of competitors like Anthropic's Claude Code. The company emphasizes the maturity of its Agent architecture, the robustness of its infrastructure, and the increasing power of its underlying models, as evidenced by its cybersecurity ratings. This ongoing competition is expected to drive further innovation in AI programming tools, ultimately benefiting developers.