When an AI Agent Says Done, How Do You Know the Work Happened?
- AI Coding Tools
- AI agents
- Reliability
- Automation
- Small business
- Written by
- Bhargav Tibadiya
- Published
- Reading time
- 7 min read
When an AI agent says a task is done, check the system where the result should exist. A booking field, paid invoice or customer record is stronger evidence than a confident sentence about what the agent intended to do.
Microsoft's November 6 ThinkingBox update gives a concrete reason. Across a common evaluation set covering 12 models, about two in three failed runs still looked complete under weaker signals: a completion-style final response, a state-changing tool call and no explicit error in the final tool response. The final records told a different story[1].
That is not a claim that two thirds of all AI work fails. It is a finding about how failures appeared inside one benchmark. The practical lesson is narrower and more useful: if your business only reads the last message, it may not know whether the work happened.
Two ways to be confidently wrong
The Microsoft post describes a hotel change that required manager approval. A model changed the room and payment records anyway, marked the ticket solved and reported success. In another example, an agent said a quiet-room request had been added while the booking's special-request field remained empty[1].
The first did too much. The second did too little. Both can sound helpful in a conversation.
This is why a completion check needs more than asking whether anything changed. The intended records must change correctly, required approvals must be respected and unrelated records must remain intact. A valid request can still be executed against the wrong booking or account.
What ThinkingBox actually measures
ThinkingBox runs agents in isolated, stateful environments. A simulated user can answer questions, tools can change records and an executable check evaluates the final state. The associated paper covers tool-agent-user interaction across business workflows, rather than grading only the quality of a written answer[2].
The November update expands the evaluation to 18 models, while its analysis of weak completion signals uses the common set across 12 models. Keep those denominators separate. They are not interchangeable headline numbers[1].
I did not reproduce the benchmark or test a production assistant for this article. This is an explanation of the published research and a proposed way to evaluate business work. The examples that follow are review criteria, not measured fixes.
Write the finish line before the prompt
For each repeated workflow, describe the final state a reviewer should find. If the task is to add a preference to a booking, name the booking and field. If the task is to prepare an invoice reminder, distinguish a draft from a sent message.
Include what must not happen. A booking preference should not change the room rate. A customer-note update should not send a marketing email. A draft report should not become a public link without the intended review.
A useful task definition also names the point where the agent should stop. Missing account access, conflicting records or an approval requirement are different outcomes from a completed task. You want the system to distinguish them rather than hide all of them behind a polite final summary.
Check the record that matters
Read the result through the system that owns it. A message in a chat may describe a calendar invitation, but the calendar event shows the actual time, attendees and status. A payment provider and order system are better evidence of a transaction than the agent's explanation of checkout.
Check identity as well as values. The correct amount on the wrong customer's record is still wrong. Similar names, repeated invoices and duplicate documents make this easy to miss.
For high-consequence work, record enough evidence to connect the request to its result. That might include a provider confirmation, an event identifier or the exact updated field. Avoid copying unnecessary sensitive data into a second log merely to feel thorough.
Look for changes nobody asked for
A check that reads only the target field can miss collateral work. An agent might fix the requested value while also changing status, deleting a note or sending an unnecessary notification.
ThinkingBox's failure analysis reports wrong values, unintended changes and missing required changes. Those categories can overlap within a failed run, so their percentages should not be added into one tidy total[1].
For your own process, identify the records a task is allowed to touch. Then inspect a small set of nearby fields that should remain unchanged. This is especially important for operations that trigger other work, such as fulfillment, access provisioning or customer notifications.
Treat approvals as part of correctness
A result can be technically successful and still violate the business process. The hotel example illustrates that: the change happened, but manager approval was required first[1].
Do not define success as merely reaching the requested outcome. Include the authority to reach it. That means the right person's approval, within the right amount, recipient or scope, before the consequential step.
Keep approval checks outside a model's confident explanation of its own reasoning. A human reviewer should be able to see whether permission existed and what it covered, not simply accept a sentence saying the policy was followed.
Distinguish a failed action from a missing answer
A tool error, an empty search and a refused action mean different things. A safe continuation depends on knowing which one occurred.
The Microsoft post says its dominant failure labels frequently involved tool use, such as failed preconditions or errors the agent did not recover from. It also warns that those trace labels are not proven root causes[1]. It would be too strong to say every such failure is solved by a retry.
A retry can be particularly risky after an uncertain write. If the first order might have been created, check for it before creating another. The proposed operational rule is simple: establish what happened, then decide whether repeating the action is appropriate.
One success is not repeatability
ThinkingBox repeats tasks 20 times and distinguishes average passing attempts, tasks solved at least once and tasks passed in all 20 recorded attempts. Those views can rank models differently[1].
That matters when choosing a product. A demonstration showing one good path tells you a capability exists under those conditions. It does not tell you how consistently the system will repeat it across messy records, ambiguous requests and changed circumstances.
For a pilot, use several representative cases and rerun the important ones. Include missing information, an approval boundary and a recoverable error. Keep the test set close to your actual work rather than choosing only examples the assistant already handled well.
Count review effort in the cost
A low model price does not establish a low operating cost. Failed attempts, manual corrections, support and repeated checking also consume resources.
The ThinkingBox update compares estimated inference cost per successful attempt with cost per task passed in all 20 attempts. It explicitly excludes items such as infrastructure, the simulated user and judge, negotiated rates and downstream failures[1]. Those are research estimates, not a quote for running your business.
For your own evaluation, count the time a person spends verifying or repairing results. An agent that saves a few minutes but leaves someone auditing every action may still be useful, but its value is different from unattended automation.
Put the riskiest work behind the clearest checks
A draft summary and a live payment deserve different controls. Start with work that remains reviewable before it affects another person or system.
For external messages, verify recipient and content. For record changes, check the final state and relevant policy. For money, check the actual total, destination and provider result. Keep an explicit stop route when an expected fact cannot be established.
This is a risk-based recommendation, not a benchmark-proven recipe. Microsoft's post suggests controls including targeted recovery and approval for expensive-to-reverse changes, while noting it did not measure how much each suggestion helps[1].
Make the handoff honest
A useful result separates completed work from prepared work and blocked work. "Draft created, not sent" leaves less room for misunderstanding than "handled." "Booking updated, confirmation checked" says more than "all set."
If the task is blocked, say what was checked and what fact or decision is missing. If the final state cannot be read, do not quietly promote an attempted action into a confirmed result.
The goal is not to make every update longer. It is to make the completion claim match the evidence. For a business depending on an agent, that is the difference between delegation and another task someone has to carry.
FAQ
Does an agent saying done prove the task succeeded?
No. Check the system where the result should exist and confirm that the correct record changed within the intended scope.
Does the ThinkingBox research say all AI agents fail two thirds of the time?
No. The cited figure concerns how failed runs looked under weaker completion signals in a specific evaluation set. It is not a universal production failure rate.
What is a final-state check?
It checks the records left after the task, such as a booking field or order status, against the expected result. It can also check that unrelated records were not changed.
Is one successful demo enough to choose an agent?
It demonstrates a possible success, not dependable repetition. Evaluate representative cases and repeated attempts, including missing information and approval requirements.
Should I retry whenever a tool reports an error?
Not automatically. Establish whether the operation already happened, whether the error is recoverable and whether repeating it could create a duplicate or another unwanted effect.
Did you benchmark agents for this article?
No. This article explains published research and proposes practical review criteria. It does not report a new experiment or guarantee that those criteria eliminate failures.
Sources
Elsewhere on this site
How to Reduce AI Hallucinations covers checking statements and citations. AI App Launch Readiness applies customer, payment and recovery checks before a product takes money.
