Short answer: Visa reports that 57.2% of runs passed across the first round's 22 models (the release calls them 'selected' models and does not say how they were chosen), and about 54% of failed runs still ended in a purchase. The release does not split wrong buys from missed requirements. Results are Visa's own, with no independent run; no models are named, and checkout and payment were left out.
A run is one attempt by an AI model at one scenario, from the first message to the final decision. A pass rate is the share of runs that met the benchmark's rule. In Visa's benchmark a run passes only if the model uncovered every requirement the scenario defined and made the right buy or no-buy decision.
The one-line takeaway: a failed run is not always a run that did nothing. Per Visa, in about half of failed runs the AI still ended in what Visa calls a purchase (the release does not say whether it was simulated; checkout and payment were outside this round), so guard the step where the agent decides to act.
Basis: one public source, Visa's press release of November 5, 2026, read on November 5, 2026. We tested nothing, and we did not open the whitepaper or the Visa Payments Frontier site that the release points to. Each number below belongs to Visa, and our own recommendations are marked as ours.
Read the numbers with this in mind. Visa is a payments company with a commercial interest in agentic commerce: the release describes Visa Intelligent Commerce as its portfolio of products, programs and partnerships for it. Visa's own FAQ calls the benchmark "complementary research" and, asked whether it is saying AI agents are not ready for commerce, answers "No", pointing to the best performing models passing 82.2% of attempts (the release does not name them) while saying consistency and reliability remain challenges. The benchmark and results are published by Visa; the release shows no independent run. The release does not identify the tested models and covers the shopping and decision stage only: Visa says it will extend testing to checkout, payment and order placement later.
What did Visa's benchmark measure?
According to the release, the first round used 22 AI models from 12 labs on 15 shopping scenarios, such as buying a TV, a laptop or a personal care appliance. Each model got three attempts at each scenario; each attempt is one run. Each model had 15 scenarios x 3 attempts = 45 attempts, so the scores describe Visa's 15 scenarios, not shopping in general. The task has two parts. The model must work out what the customer needs from a conversation, and if nothing fits those requirements, "the AI should opt not to buy", so a pass can also be a correct decision not to buy. Visa adds: "To prevent lucky guesses, the AI must uncover enough information to justify its decision."
A run passes only if both hold. Visa's footnote says the benchmark is the first to require "both discovery of every task-defined customer requirement and a correct purchase or no-purchase decision for a run to pass". The release's example is a customer who wants a new TV, mentions a relative's brand recommendation and a membership with a third-party supplier, and leaves other details unsaid; the AI must uncover the budget and requirements such as format compatibility, HDR10+ support and gaming.
How often did runs fail, and what happened in the failed ones?
Visa reports that 57.2% of runs passed across the first round's 22 models (the release calls them 'selected' models and does not say how they were chosen). Its best performing models passed 82.2% of attempts and the worst 28.9%; the release does not name them. Visa's key findings list says the models failed 42.8% of test runs. Then comes the figure most readers will remember: Visa says "About 1-in-2 (54%) of failed runs still ended in the AI making a purchase".
The release explains that sentence in one line: "Either the AI recommended a purchase that did not meet the customer's identified requirements, or it failed to uncover all the customer's requirements." So a failed run that ended in a purchase covers two different situations, and the release does not say how the 54% splits between them. It also does not say whether the product bought in the second case was wrong for the customer, only that not all requirements had been uncovered.
Our arithmetic, from Visa's rounded percentages: 54% of 42.8% is about 23% of all runs. The stated design implies 22 x 15 x 3 = 990 runs if every scheduled attempt was completed and counted; the release does not report a final run count, so we do not state a number of purchases. For readers of our decode of project-level AI failure rates: that article is about whole projects, and this one is about single test runs, so the 42.8% here is a different thing from the 42% there.
Why does one passing run prove little?
Visa reports: "Every model got three attempts at each shopping scenario. Across all models and scenarios, all three attempts passed just 34.2% of the time." Set that beside the 57.2% overall pass rate and the gap is the point: a pass on a single run is a weaker claim than a pass on all three. Visa also says that models that were better at uncovering requirements tended to pass more often, "But when made to repeat a scenario they had already passed, the same models often failed".
Visa's own grouping on requirement discovery is in the release too: the five models that uncovered the most hidden buying factors averaged a 70.2% pass rate, and the five that uncovered the least, 44.4%. Because a pass requires uncovering every requirement, that grouping and the pass rate overlap by design (our reading); the release does not give the group members.
Does a higher price mean a better shopping agent?
Visa's wording is "Pricy models win, but do not guarantee success". The numbers behind it: the five models with the highest average cost per run passed 67.1% of runs, against 50.2% for the five least expensive. This is Visa's grouping of five models at each end out of 22. The release does not name the models, say how cost per run was measured, or show the middle of the range, so we draw no causal conclusion about price. Our note: cost per run is not the total cost of running a production agent. For a buyer, the figure is a reason to test your own scenarios, not a reason to pick the most expensive model.
What does the benchmark not cover?
The table separates what Visa states from what we did not find on the release page.
| Item | What the Visa release states | Not found on the release page |
|---|---|---|
| Models | 22 AI models from 12 labs | Model names, lab names, versions, settings, how the models were selected |
| Scenarios and runs | 15 shopping scenarios, three attempts per model and scenario | Total run count, per-model and per-scenario results |
| Pass rate | 57.2% of runs passed across the first round's 22 'selected' models; best performing models 82.2%, worst 28.9% (published by Visa) | Independent run or verification, confidence intervals |
| Failed runs | 42.8% failed; about 54% of failed runs still ended in a purchase | Split between wrong purchases and missed requirements |
| Repeat runs | All three attempts passed 34.2% of the time | Which models were consistent |
| Cost grouping | Five highest-cost models 67.1%, five least expensive 50.2% | Which models, how cost per run was measured |
| Purchases | A failed run could end in a purchase | Whether purchases were simulated or real |
| Checkout and payment | Not in this round; Visa will extend testing to checkout, payment and order placement | A date for that extension |
Visa also says it has shared the benchmark with leading model providers for feedback; we saw no feedback or responses on the page.
Do this week
Three steps for any team that lets an agent order, book or send on its own.
- Write down every action your agent can take that costs money or cannot be undone, and put a spend cap and a separate check in front of each.
- Ask whoever built your agent to run each test scenario three times or more, and count it as passed only if every run passes.
- Log every failed run that still ended in an action, and review that list weekly before you widen the agent's access.
What should you guard before an agent buys?
These are our recommendations, not Visa's. They follow from the release's own figures: failures that still end in an action, and results that change between attempts.
- Put a separate check on the buy / no-buy step. A second step, a rule or a person, should confirm that the item meets the stated requirements before the order goes out, and a spend cap should limit what one agent can commit. The release says 54% of failed runs still ended in a purchase; do not let the same agent that made the decision be the only gate.
- Ask whoever built your agent to run each test scenario three times or more, and count it as passed only if every run passes. This is stricter than Visa's per-run pass rule; it mirrors the separate three-attempt measure behind Visa's 34.2% figure, and a single success hides the variation Visa describes.
- Log every failed run that still ended in an action. Our recommendation: count the runs where the agent took a real action (bought, booked, sent) while a requirement was unmet; Visa reports only purchases. That count is the number to bring down before you widen access.
- Test what Visa's first round did not. The release leaves out checkout, payment and order placement. If your agent touches them, test those steps yourself.
Our guide to production agent architecture covers where such checks belong, our note on banks blocking AI agents shows how one side of a payment treats agent traffic, and our comparison of agentic AI hype and what ships gives the wider picture. If you want help designing the checks, see our AI agent development service.
What is still not confirmed?
The release does not identify the tested models, say how the models were selected, or give model-level results, and it does not report a final run count or say whether every model took all 15 scenarios once. We also do not know how the failures split between wrong purchases and missed requirements, or whether anyone outside Visa has reproduced the results. The whitepaper may answer some of this; we did not read it. This article reflects the release as published on November 5, 2026, and Visa may add detail or revise it.