Computer-Use Agents: Why a Successful Demo Can Still Be an Unreliable System

Computer-use agents can now click, type, browse, edit files, operate desktop applications, and complete impressive multi-step tasks. That makes successful demos easy to understand and easy to overinterpret. A single completed workflow shows that the agent can succeed under those conditions. It does not show how often it succeeds, how it behaves when the environment changes, whether it verifies the result, or how safely it acts when the goal becomes ambiguous.

Why the demo is the easiest possible reliability test

A demo normally shows one trajectory that worked. The environment is known, the task is selected in advance, the operator can restart after a failure, and the audience sees the successful path. Production systems face a distribution instead: different pages, network conditions, account states, pop-ups, latency, UI changes, hidden state, permissions, interruptions, and users who describe goals imperfectly.

That distinction matters because computer-use agents operate through interfaces designed for humans rather than deterministic APIs. Their action loop depends on perception, state interpretation, planning, interaction timing, and environment response. Small changes can alter the trajectory even when the user goal is unchanged.

Microsoft Research's WAREX work makes the problem explicit: benchmark agents that look capable in controlled settings lose substantial task success when realistic web instability is introduced. The failure is not necessarily “the model became less intelligent.” The environment stopped being deterministic.

Capability, success rate, reliability, and safety are different claims

The Computer-Use Reliability Ladder

A useful way to evaluate computer-use systems is to move from one-off capability toward progressively harder reliability properties. Higher levels assume the lower levels but do not follow automatically from them.

Level 1 — Capability: the demo question

Capability asks whether an agent can perform the task at all. This is valuable. Computer-use systems have advanced rapidly, and modern agents can complete workflows that older systems could not execute reliably.

But capability is a weak deployment criterion. One successful run does not tell you whether the agent succeeds 95% of the time or 30% of the time, whether failures are harmless or destructive, or whether success depends on a lucky page state.

Level 2 — Repeatability: does the same task stay solved?

Computer-use trajectories are stochastic. Model outputs vary, pages load at different speeds, visual states change, and long workflows create many branching opportunities. A production test should therefore run the same task multiple times rather than treating one passing trace as representative.

Measure not only the average success rate but also the distribution of failure modes: wrong click, premature termination, missed confirmation, incorrect field, duplicate action, navigation loop, stale-state assumption, and false success report.

Level 3 — Environmental robustness: what happens when the web behaves like the web?

Real websites are not benchmark fixtures. Requests fail, elements load late, sessions expire, pages change, consent banners appear, servers return errors, and network conditions fluctuate.

WAREX evaluates this gap by injecting realistic web unreliability into existing benchmark environments and reports significant drops in task success. This is a critical production insight: a benchmark can measure task competence while under-measuring recovery from environmental instability.

Level 4 — Long-horizon control: success changes when the task becomes real work

Short tasks hide a class of failures that appear only after dozens or hundreds of actions: forgotten constraints, duplicated work, premature completion, missed state changes, cross-application inconsistencies, and accumulated small errors.

OSWorld 2.0 was designed specifically around long-horizon real-world workflows. Its tasks take human users a median of roughly 1.6 hours and require many more tool calls than earlier computer-use benchmarks. Under its primary completion metric, even the strongest evaluated systems remain far from complete task reliability.

WeaveBench reaches a similar conclusion from another angle. It evaluates hybrid GUI, CLI and code workflows and reports that the best evaluated model-runtime pairing passes only 41.2% of tasks. The important result is not one leaderboard number; it is that realistic cross-interface orchestration exposes failures hidden by simpler single-interface tasks.

Level 5 — State awareness: the environment can change underneath the plan

Long-running tasks often depend on hidden or changing state: an email arrives, a calendar changes, a form is submitted, a background process finishes, a browser session expires, a user modifies a file, or an external system changes availability.

Microsoft's SentinelBench argues that many long-running tasks should not be solved through continuous action at all. The correct behaviour may be to monitor, wait for an external event, then act when the state changes. This is a different capability from clicking faster or planning more steps.

A reliable computer-use agent therefore needs to distinguish actionable now, waiting for state, state changed, and assumption invalidated.

Level 6 — Outcome verification: did the action actually work?

An agent can execute an apparently correct sequence and still fail the task. A button click may not register. A form may reject hidden validation. A file may save to the wrong directory. A purchase may remain unconfirmed. A site may display a success-looking screen while the underlying operation failed.

OpenAI's current computer-use guidance explicitly recommends bounding and verifying the run instead of relying only on the model's final answer. Microsoft Research's work on computer-use verifiers reaches the same conclusion from evaluation: process and outcome need to be judged separately.

The Universal Verifier research reports that earlier verifier setups can produce high false-positive rates, while stronger rubric design and explicit separation of process, outcome, controllable failures, and uncontrollable failures substantially improve agreement with human labels.

Level 7 — Safe goal handling: the agent must know when not to continue

Computer-use agents are optimized to complete goals, but goal persistence can itself become a failure mode. An ambiguous request, impossible condition, contradictory instruction, suspicious webpage, or changed environment may require clarification or stopping rather than more action.

The BLIND-ACT benchmark studies this problem as Blind Goal-Directedness. Across the systems evaluated in that work, agents frequently continued pursuing tasks despite ambiguity, infeasibility, conflicting context, or other reasons to reconsider. The authors identify patterns such as execution-first bias and request primacy.

This failure class matters because a highly capable agent can make a bad situation worse faster. Reliability therefore includes a policy for when not to act.

The Demo-to-Production Stress Test

Before deploying a computer-use workflow, take the successful demo and systematically remove the assumptions that made it easy.

Benchmark success has a validity boundary

A benchmark score is a conditional statement. It is valid for a particular model, harness, environment, task set, judge, tool interface, step budget, retry policy, date and evaluation method.

The number becomes misleading when those conditions disappear from the claim. “Agent X scores 80%” is weaker than “Agent X scored 80% on benchmark Y under environment Z with judge J and step budget N.” The second statement preserves the boundary that tells you whether the number transfers to your application.

Process success and outcome success must be scored separately

WeaveBench reports that outcome-only grading can materially overestimate computer-use performance because an agent may produce an apparently successful artifact through a shortcut or fabricated evidence. The verifier must inspect the trajectory and deliverables, not merely the final claim.

Production reliability is a distribution, not a single pass rate

A useful production evaluation samples the dimensions that actually vary in your environment. For a browser workflow, that might include account age, locale, viewport, page version, network quality, authentication state, existing cart state, cookies, pop-ups, user permissions and whether a human interrupts the run.

Reliability needs a failure budget, not perfection

No production system is perfectly reliable. The useful engineering question is which failures are acceptable, detectable and recoverable. A failed attempt to sort a local folder is not equivalent to sending the wrong email, purchasing the wrong product or changing an account setting.

Classify actions by consequence and reversibility. Low-impact reversible actions can tolerate more autonomy. High-impact, externally visible or hard-to-reverse actions need stronger confirmation, state verification, authorization and post-action checks.

A practical computer-use reliability matrix

What to log for a computer-use failure

• User goal and explicit constraints.
• Model and harness version.
• Environment and application versions.
• Screenshots or structured observations relevant to the failure.
• Actions taken with timestamps.
• Tool, click, keyboard and navigation results.
• State transitions and waiting periods.
• Approval, refusal or handoff events.
• External errors and network failures.
• Final observable environment state.
• The agent's reported outcome.
• Verifier result and whether the failure was controllable by the agent.

The crucial comparison is between reported success and observable success. A system that cannot distinguish those two will eventually accumulate false positives in production.

Security is part of reliability for computer-use agents

Computer-use agents do not merely read untrusted content; they can act after reading it. That turns prompt injection, malicious page content and phishing into execution-path risks.

OpenAI's current computer-use guidance recommends isolating the environment, allow-listing sites and actions, treating screen content as untrusted, confirming consequential actions, bounding the run and verifying the actual outcome. ChatGPT agent similarly uses confirmations, prompt-injection monitoring and supervised modes for sensitive contexts.

The architecture principle is broader than any one provider: content observed by the agent must not be allowed to redefine the user's authority. A webpage can provide data. It cannot grant permission to send data elsewhere, purchase something, change credentials or override the task boundary.

What would change this answer?

The reliability gap would narrow if computer-use models became robust to long horizons, dynamic state, UI variation, environmental failures and ambiguous goals across representative production distributions. Better native state APIs, standardized machine-readable interfaces and stronger verifier infrastructure could also reduce the amount of fragile GUI interaction required.

The deployment threshold also changes with task consequence. A 70% success rate can be useful for a supervised low-risk research task and unacceptable for an autonomous financial or destructive workflow. Reliability must therefore be evaluated against the cost of each failure class, not one universal pass-rate threshold.

Limitations

The cited benchmarks evaluate different environments and should not be ranked against one another as if they measured the same thing. WAREX stresses web unreliability; WeaveBench targets hybrid long-horizon work; OSWorld 2.0 targets realistic long workflows; BLIND-ACT focuses on goal handling under ambiguity and infeasibility.

Benchmark results also age quickly. Model, harness and verifier improvements can materially change scores within months. The durable lesson is therefore the evaluation method: vary conditions, separate process from outcome, verify external state, and preserve the boundary around each performance claim.

Conclusion

Computer-use agents are already capable enough to be useful. That is exactly why the evaluation question has changed. The challenge is no longer only whether an agent can click through a workflow. It is whether the system remains dependable when the clean demo conditions disappear.

Treat one successful run as evidence of capability. Then test repeatability, environmental robustness, long-horizon control, state awareness, outcome verification and safe goal handling. A production computer-use agent is not the one that can complete the demo. It is the one whose failure boundaries are known, measured and controlled.

FAQ

Glossary

Primary sources and further reading