Airbnb says AI wrote 60% of the code its engineers produced last quarter. Google is pushing development tools further into autonomous agents. Now software teams face a less glamorous question: how do you verify code when the process that created it is no longer predictable?
Software development’s AI boom is quickly turning into a software verification problem.
Earlier this month, Airbnb said that AI generated roughly 60% of the code produced by its engineers during the first quarter of 2026. CEO Brian Chesky described teams using agents to take on work that would previously have required considerably more engineering capacity. The company is hardly alone in pushing AI deeper into development.
Then came Google I/O. On May 19, Google introduced Gemini 3.5 and expanded Antigravity, its agent-first development platform, positioning agents less as coding assistants and more as systems that can independently navigate complex workflows. Google says Antigravity 2.0 can orchestrate specialised subagents, while Gemini 3.5 Flash is designed for long-running coding and agentic tasks.
The pitch is productivity. The complication is verification.
If an engineer uses AI to write ten times more code, the organisation does not automatically gain ten times more capacity to understand whether that code behaves correctly. And as coding tools move from generating snippets to carrying out multi-step tasks, some of the assumptions behind conventional software testing start to look less reliable.
“Code generation is becoming cheaper very quickly, but confidence in that code is not getting cheaper at the same rate,” says Rahul Sanjay, an AI infrastructure and quality engineering architect who has worked on AI-assisted testing and enterprise automation systems. “The bottleneck starts moving. You spend less time producing an implementation and more time proving that the implementation is actually safe to release.”
The test can fail even when the agent succeeds
GitHub described a version of this problem on May 6 in a technical post about validating autonomous agents. Traditional software testing tends to assume that correct behaviour is repeatable: provide the same conditions and the software should follow an expected path towards an expected result.
Agents complicate that model. A coding agent interacting with an IDE, browser or cloud environment can take different routes to complete the same task. A loading screen might appear in one run and disappear in another. Timing can vary. The agent may choose one sequence of actions today and another tomorrow, while reaching the same valid outcome in both cases.
GitHub’s engineers found that this variability can produce an awkward result: the agent completes the task successfully, but the test still reports a failure because the execution no longer matches the sequence the validation system expected. The company argues that agentic validation needs to focus more heavily on essential outcomes rather than rigid paths. GitHub calls this an independent “Trust Layer” for agentic behaviour.
That sounds like a testing detail, but it gets at a larger problem.
Conventional automated testing works particularly well when engineers can specify a known sequence. Open a page, locate an element, click it, check the result. The more autonomy a system receives, the harder it becomes to assume that every correct execution should look identical.
“The test case has traditionally encoded both the goal and the route to the goal,” Sanjay says. “With agents, I think those two things have to be separated much more carefully. You may still need strict constraints around security, data integrity or the final system state, but you cannot necessarily treat every variation in the execution path as a defect.”
For quality engineering teams, that means deciding which parts of a workflow are mandatory and which are incidental. Saving the correct data may be mandatory. Waiting two seconds rather than one probably is not. An agent choosing a different menu or command could be irrelevant if the result is identical.
That distinction becomes harder as the number of possible actions grows.
AI testing AI creates its own trust problem
The obvious answer is to put more AI into testing. That is already happening.
AI systems can generate test cases, repair broken automation, analyse changes to a codebase and identify areas that may require additional validation. Developers can use one model to produce code and another to review it.
But there is a circularity here that software teams cannot ignore. If AI writes the implementation and AI also decides whether the implementation is correct, where does independent verification come from?
The problem is not necessarily that AI is incapable of testing AI-generated software. In many cases it may be highly effective. The question is how much confidence an organisation should place in a verification system that may share some of the same blind spots as the system that produced the code.
Sanjay argues that the answer is unlikely to be a single universal AI “judge.”
“You want different signals,” he says. “Static analysis can tell you one thing. Functional execution tells you something else. Runtime behaviour, security controls and business rules provide other evidence. AI can help connect those signals, but I would be cautious about replacing all of them with one model making a pass-or-fail decision.”
That is particularly relevant in environments such as banking, automotive systems or large enterprise platforms, where a technically plausible result is not necessarily an acceptable one. A payment can execute successfully while violating a business rule. An application can return the right screen while storing the wrong data. A generated fix can solve one bug while creating a security problem somewhere else.
Verification therefore has to extend beyond asking whether the application ran without crashing.
Quality engineering is moving closer to risk management
None of this means deterministic testing is disappearing.
Most software still contains enormous amounts of behaviour that should be predictable. APIs need to return the right values. Authentication rules need to work consistently. Database operations cannot become “approximately correct” because an AI agent happens to be involved.
What changes is the boundary around those tests.
A team may use strict assertions for data integrity and security while allowing greater flexibility in how an agent navigates an interface. It may validate the final state of a workflow rather than every intermediate action. For higher-risk changes, it might require human review regardless of what automated systems conclude.
The more agents participate in development, the more testing begins to resemble a problem of allocating confidence.
Not every generated line of code deserves the same level of scrutiny. A formatting change and a modification to payment logic carry very different risks. An agent updating internal documentation is not equivalent to an agent changing authentication code.
Software teams have always made these distinctions informally. AI coding makes it increasingly necessary to encode them into the development process itself.
The productivity numbers tell only half the story
The headline numbers around AI coding are difficult to ignore. Airbnb’s claim that AI generated 60% of its engineers’ code in a quarter offers a glimpse of what software production can look like when agents become a normal part of the workflow. Google’s latest developer tools point in the same direction, with multiple agents increasingly able to work on complex tasks under human supervision.
But lines of generated code are an incomplete measure of productivity.
Software has value when it reaches production, works reliably and continues working after the next change. If faster code generation produces larger review queues, more regressions or test suites that teams no longer trust, some of the apparent productivity gain simply moves elsewhere in the pipeline.
This is why verification may become one of the more important, if less spectacular, parts of the agentic development stack.
The first generation of AI coding tools was judged largely on whether they could produce useful code. The next generation is being asked to take actions across repositories, terminals, browsers and development environments.
That raises the bar considerably.
The central question for engineering teams is no longer only whether an agent can complete the task it was given. It is whether the organisation can tell, with enough confidence, when that task was completed correctly.
