Vercel CEO Warns of AI Hallucination Deploying Unrelated Code to Production

Vercel CEO Guillermo Rauch has issued an urgent warning after an AI programming agent, powered by Anthropic's Claude Opus 4.6, hallucinated a GitHub repository ID and deployed a student's assignment to a corporate production environment. The incident highlights a new class of AI-driven security risks, according to Rauch.

Guillermo Rauch, CEO of Vercel, in a professional setting.

The event began when a Vercel user discovered an unfamiliar open-source library in their project list. Vercel, a popular front-end deployment platform used by over 6 million developers, hosts sites for entities such as OpenAI and Perplexity. Rauch, who also led the creation of the Next.js framework, described the incident as "extremely terrifying."

An investigation revealed that the unexpected deployment was not the result of a hacker or an internal human error. Instead, an AI programming agent used by the team was responsible.

Unpredictable AI Errors

Rauch detailed the incident on social media, explaining that the AI agent, while executing a deployment task, needed a GitHub repository ID to interact with Vercel's API. Rather than querying GitHub for the correct ID, the AI "hallucinated" a nine-digit number. This fabricated ID coincidentally corresponded to a real GitHub repository containing a university student's course assignment. Consequently, this unrelated code was deployed to a corporate team's environment.

Visual metaphor comparing logical human code flow to chaotic AI hallucination.

Rauch emphasized that this type of error differs fundamentally from human programming mistakes. Human errors, such as typos or incorrect parameters, are typically traceable and logical. In contrast, the AI agent did not report an error or indicate uncertainty; it confidently proceeded with the fabricated ID.

"Even the smartest large models today, when they make mistakes, they make mistakes in ways that are unpredictable to humans," Rauch stated. He noted that AI's confidence level remains consistent whether its output is correct or a hallucination, making such errors difficult for downstream systems to detect.

The Threat of Code Poisoning

While the mistakenly deployed code was harmless in this instance, the incident has raised concerns within the developer community about "Package Hallucination Attacks." This scenario involves attackers creating numerous seemingly benign but malicious public repositories on GitHub. They then wait for an AI agent to hallucinate an ID that matches one of these malicious repositories, leading to the deployment of harmful code.

Digital landscape with malicious code packages, symbolizing 'Package Hallucination Attacks'.

This represents a new form of supply chain attack. Unlike traditional supply chain attacks that require direct infiltration or social engineering, this method exploits AI's tendency to fabricate non-existent package or repository names. Security researchers have identified this as a significant risk, where AI's suggestions, or even AI agents themselves, could inadvertently install malicious packages into projects.

Securing AI Agents in Production

The incident underscores the growing risks as AI's capabilities expand from providing code suggestions to directly executing commands and deploying to production environments. When AI gains API call permissions and command-line execution capabilities, the consequences of hallucination become far more severe.

A subsequent blog post from Vercel's security team highlighted that most AI agents currently lack sufficient security isolation between the agent and the code it generates. This means that AI-generated code can access sensitive assets like keys, file systems, and production infrastructure. Prompt injection attacks or AI hallucinations could lead to severe outcomes.

Digital padlock representing security, with code streams attempting to breach it.

Rauch advised development teams to implement robust security measures:

• Utilize security-designed integration tools: Employ tools like Claude Code, which are built with permission boundaries for AI coding, rather than connecting AI to unrestricted command lines or APIs.

• Implement guardrails: Introduce checks between the AI and real systems. For example, verify that a repository ID originates from a legitimate API query, not an AI hallucination, and confirm deployment permissions.

• Adhere to the principle of least privilege: Grant AI agents only the minimum necessary permissions for their tasks. This is more challenging to implement with AI agents due to their uncertain behavior compared to traditional software development.

Vercel's technical blog post further described a threat model where an attacker could embed a prompt injection into a log file, instructing an AI agent to steal SSH keys and AWS credentials. The recommended solution involves establishing strict security boundaries around AI agents, generated code, and infrastructure, controlling AI's access to files, execution, and network.

Separately, The Information reported that OpenAI is developing an internal code hosting platform, potentially challenging Microsoft's GitHub. This development follows an increase in GitHub outages since late 2025, with five outages occurring in February 2026 alone. This suggests a potential restructuring in the programming infrastructure landscape.

Rauch's warning serves as a critical reminder for developers: "Even the smartest models have failure modes that are fundamentally different from human logic." Teams using or planning to use AI agents must prioritize security guardrails to prevent potentially catastrophic deployments.