Airbnb says AI wrote 60% of the code produced by its engineers in the first quarter of 2026. As coding agents take on more work, software teams must decide how to verify code generated by a process they cannot fully predict.
The use of AI in software development is becoming a challenge for testing. In May, Airbnb said AI generated roughly 60% of the code its engineers produced during the first quarter of 2026. CEO Brian Chesky described teams using agents to complete work that once required far more engineering capacity. Other technology companies are making similar changes.
At Google I/O on May 19, Google introduced Gemini 3.5 and expanded Antigravity, its agent-first development platform. These tools can take on longer coding tasks and coordinate specialised subagents. “People often assume that if AI can write code quickly, testing it should also be easy, but that’s not always true,” says Rahul Sanjay, an AI infrastructure and quality engineering architect with more than 17 years of experience.
A failed test does not always mean failed software
GitHub discussed this problem in a May 6 technical post about testing autonomous agents. Traditional tests usually expect software to follow a known path and produce an expected result. Agents are less predictable because they can take different routes through an IDE, a browser, or a cloud environment. A loading screen may appear in one run but not another, or the agent may use a different menu or command.
GitHub found that this variation can cause a test to report a failure even when the agent completed its task. The company describes an independent “Trust Layer” designed to validate the outcome without requiring an identical sequence of actions. “An AI agent can complete a task successfully but take a different route from the one the automated test expected,” Rahul says. “The test then fails because the steps were different, even though the final result was correct.”
The reverse can also happen. A test may pass because the application completed a transaction or displayed the expected screen while missing a problem with the underlying data. A payment, for example, might be processed under the wrong business rule. Teams therefore need to check the result, the stored data and the rules applied during execution.
Tests also need to separate mandatory outcomes from incidental actions. Saving the correct data may be mandatory, while waiting two seconds instead of one is usually irrelevant. “I don’t necessarily need the agent to follow exactly the same steps every time,” Rahul says. “If it reaches the correct result safely, that is usually what I care about, but security rules, data integrity, authorisation and important business rules must always be correct.”
Using AI to test AI
AI is already being used to generate test cases, repair broken automation and review code changes. This raises another question: if one AI system writes the code and another decides whether it is correct, how independent is the verification? “I wouldn’t depend on one AI model to both create the code and decide whether that code is safe,” Rahul says.
Static analysis can identify problems without running the code, while functional tests verify that the software behaves as expected. Security testing, runtime monitoring and business-rule validation provide further checks. AI can analyse these results, but it should not be the only system making the release decision. Human review may still be required for changes that affect sensitive data or core business processes.
The level of review should depend on the risk. An agent updating internal documentation does not need the same controls as one modifying payment or authentication code. “Especially in banking, a system can appear to work correctly while still having a problem with its data, security or business logic,” Rahul says. A technically successful action may still produce an unacceptable business result.
Code volume is a poor productivity measure
Airbnb’s 60% figure shows how much code AI can produce, but it says little about the quality of the software that reaches users. “I wouldn’t measure productivity by looking at how much code AI has written,” Rahul says. “I would look at whether the company is delivering better software faster and more safely.”
Teams can measure how long it takes a requirement to reach production, how many defects reach users and how often releases fail. They can also track the time engineers spend reviewing generated code and the system’s reliability after release. These figures show whether AI is reducing work or moving it to another part of the development process.
Faster code generation may lead to longer review queues, more regressions or test suites that engineers no longer trust. Coding agents can now work across repositories, terminals and development environments, giving them more ways to complete the same task. “The real measure is not how much code AI produces,” Rahul says. “It is how much reliable software the team can safely deliver to production.”
