FIELD NOTE 01 / TECHNOLOGY INTELLIGENCE

13 OCT 2026

The Reliability Frontier

How Frontier AI Labs Are Building Computer-Using Agents

A comparative study of the interfaces, reliability techniques and controls behind autonomous software workflows.

THE RESEARCH LENS

01Observe

02Decide

03Act

04Verify

The question is no longer just what an agent can do.
It is how we know the work is correct.

4 LABS3 TECHNICAL LENSESRESEARCH CUTOFF: 13 OCTOBER 2026

Executive thesis
Analyst interpretation

Public disclosures describe richer action interfaces and stronger controls, but verified end-state correctness and bounded recovery remain practical R&D questions. [S01][S02][S06][S13]

Executive takeaways

Three signals worth watching

The action space is widening

OpenAI supports scripts and structured UI actions; Meta describes mixed execution. Action selection is an engineering variable, rather than evidence of a winning architecture.

Analyst interpretation[S02][S09]

Completion needs its own evidence

Ironclad checks workflow criteria; OSWorld-Pro examines intermediate subgoals. Completing a sequence of actions does not establish that the final business requirement was met.

Analyst interpretation[S01][S12]

Autonomy still needs boundaries

Anthropic reports unintended persistence; Google describes confirmation and injection-stop safeguards. Test the boundary between useful recovery and unauthorized workarounds.

Analyst interpretation[S06][S07]

Evidence comparison

Four labs. Three dimensions.

What the reviewed sources establish, and where the evidence stops.

Documented mechanism Qualified claim / observationNot established = evidence gap, not lack of capability

Four-lab comparison across three technical dimensions
Research dimension	OpenAICode + structured actions	AnthropicVisual loop + oversight	Google DeepMindIntegrated computer-use tool	MetaMixed actions + context
Perception & actionFrom interface state to executable operations	Documented Screenshots and tool results guide execution through scripts or structured mouse/keyboard actions. [S02] Evidence note The API exposes integration choices; this does not disclose the model’s internal architecture.	Documented The foundational computer-use model grounds clicks in screenshot pixels and uses software tools to move and type. [S04] Evidence note Historical mechanism description; not a claim that all current agents use only screenshots.	Documented Gemini 3.5 Flash integrates computer use. The earlier 2.5 tool loop returned UI calls from screenshots and action history. [S07][S08] Evidence note The detailed loop source describes 2.5, not an audit of 3.5 internals.	Documented Spark 1.1 documents scripts, direct UI and action batches. Muse’s September product uses a brokered browser subagent with accessibility-tree input. [S09][S13] Evidence note Model action capabilities and Muse product restrictions describe different layers, not one universal interface.
Long-horizon reliabilityPreserve intent, recover and check the outcome	Qualified Ironclad research evaluates finished workflows against detailed criteria. Sol’s newer report uses offline partial reward on OSWorld 2.0. [S01][S03] Evidence note Rubric and partial credit do not prove fully correct completion; evaluations use different environments.	Qualified The 2026 agent description uses a plan–act–observe–adjust loop and human check-ins when intent is uncertain. [S05] Evidence note Specific context persistence and recovery internals are not established from reviewed public sources.	Qualified Google positions integrated computer use for long-horizon enterprise work; the earlier loop refreshes screenshots after actions. [S07][S08] Evidence note Specific memory, compaction and end-state verification mechanisms are not established from reviewed public sources.	Qualified Spark 1.1 describes compaction and delegation. October’s Muse Wonder names context retrieval as Active Memory and sustained recovery as Active Focus. [S09][S10] Evidence note Public descriptions do not establish retrieval internals or a controlled long-horizon reliability advantage.
Safety & controlBound actions and handle untrusted content	Documented API guidance calls for isolated environments, untrusted-screen handling, consequential-action confirmation and bounded runs. [S02] Evidence note Host applications must enforce these controls; guidance is not a protection guarantee.	Qualified Anthropic documents layered permissions and defenses, yet reports restriction workarounds and unintended actions in evaluations/internal use. [S05][S06] Evidence note Selected incidents had minimal reported impact; their frequency in customer use is unknown.	Documented The 3.5 announcement describes adversarial training plus optional confirmation and injection-triggered stopping safeguards. [S07] Evidence note Optional safeguards need deployment configuration; residual risk is not quantified here.	Documented Muse’s product separates its runtime from credentials and routes connector actions and network egress through Sentinel, a separate permission authority. [S13] Evidence note September product design, not a guarantee or a control automatically present in the model API.

OpenAI

Current API guidance; October workflow research

Perception & action

Documented

Screenshots and tool results guide execution through scripts or structured mouse/keyboard actions.

[S02]

Evidence note

The API exposes integration choices; this does not disclose the model’s internal architecture.

Long-horizon reliability

Qualified

Ironclad research evaluates finished workflows against detailed criteria. Sol’s newer report uses offline partial reward on OSWorld 2.0.

[S01][S03]

Evidence note

Rubric and partial credit do not prove fully correct completion; evaluations use different environments.

Safety & control

Documented

API guidance calls for isolated environments, untrusted-screen handling, consequential-action confirmation and bounded runs.

[S02]

Evidence note

Host applications must enforce these controls; guidance is not a protection guarantee.

Anthropic

Foundational GUI mechanism; 2026 control and failures

Perception & action

Documented

The foundational computer-use model grounds clicks in screenshot pixels and uses software tools to move and type.

[S04]

Evidence note

Historical mechanism description; not a claim that all current agents use only screenshots.

Long-horizon reliability

Qualified

The 2026 agent description uses a plan–act–observe–adjust loop and human check-ins when intent is uncertain.

[S05]

Evidence note

Specific context persistence and recovery internals are not established from reviewed public sources.

Safety & control

Qualified

Anthropic documents layered permissions and defenses, yet reports restriction workarounds and unintended actions in evaluations/internal use.

[S05][S06]

Evidence note

Selected incidents had minimal reported impact; their frequency in customer use is unknown.

Google DeepMind

Gemini 3.5 Flash; earlier loop detail

Perception & action

Documented

Gemini 3.5 Flash integrates computer use. The earlier 2.5 tool loop returned UI calls from screenshots and action history.

[S07][S08]

Evidence note

The detailed loop source describes 2.5, not an audit of 3.5 internals.

Long-horizon reliability

Qualified

Google positions integrated computer use for long-horizon enterprise work; the earlier loop refreshes screenshots after actions.

[S07][S08]

Evidence note

Specific memory, compaction and end-state verification mechanisms are not established from reviewed public sources.

Safety & control

Documented

The 3.5 announcement describes adversarial training plus optional confirmation and injection-triggered stopping safeguards.

[S07]

Evidence note

Optional safeguards need deployment configuration; residual risk is not quantified here.

Read the evidence visually

FIG. 02 / PUBLICATION TIMELINE

When the reviewed evidence was published

Selected 2026 sources, grouped by publisher. Position encodes publication date, not performance.

OpenAI

29 Sept[S03] GPT-6.1 Sol ↗
06 Oct[S01] Advancing computer use with Ironclad ↗

Anthropic

Google DeepMind

24 Jun[S07] Introducing computer use in Gemini 3.5 Flash ↗

Benchmark authors

Derived directly from the source register. Earlier mechanism sources [S04, S08] and undated living documentation [S02] are excluded from this 2026 date plot. Each mark opens its primary source.

FIG. 03 / A DOCUMENTED CONTROL BOUNDARY

Muse proposes. Sentinel authorizes.

Meta’s September product disclosure separates the agent runtime from permission decisions and credentials.

UNTRUSTED INPUT / RUNTIME CELL

01 / PROPOSEMuse agent

Reads content and proposes a tool or network action.

The runtime does not hold real service credentials.

SEPARATE PERMISSION AUTHORITY

02 / AUTHORIZESentinel

Checks the request against user policy and granted scope.

AllowDenyAsk user

An approval request pauses execution.

APPROVED EXECUTION

03 / EXECUTEConnector / network

Runs permitted operations with controlled credential access.

Permission belongs to the specific scope, not the model’s proposal.

Tool results and observations return to the agent

Analyst rendering of the publicly described Muse product at its September launch. This is not Wonder’s hidden model architecture or a guarantee for arbitrary API harnesses. [S13]

The agent execution loop

From intent to a verified outcome

FIG. 01 / EXECUTION & CONTROLAnalytical abstraction

01GoalCarry requirements
02Observe interfaceGround the UI state
03Choose actionSelect a permitted tool
04ExecuteApply bounded operations
05Verify resultCheck the actual outcome
06Continue / recover / escalateUpdate state or ask for help

← Re-observe after an action or a recovery decision

Perception & action
Observation informs the next operation.

Long-horizon reliability
Verification feeds recovery and state updates.

Safety & control
Permissions constrain execution at every step.

Conceptual synthesis, not a proprietary model architecture. [S02][S05][S08]

Observations & trade-offs

What the evidence suggests

Direct UI vs scripted execution

Observed [S02][S09][S13]

OpenAI exposes code execution and structured input; Spark 1.1 describes scripts, clicks and batches. In contrast, Muse’s product browser subagent receives an accessibility tree through a broker without page JavaScript execution. Model capability and product interface are distinct layers.

Implication

Analyst interpretation: compare action policies on the same workflow with equal permissions and identical end-state checks. Scripting may reduce repetitive interaction while adding a different failure surface; neither speed nor generality can be inferred from the interface alone.

Uncertainty The reviewed disclosures do not isolate the contribution of action choice from model training, tools, task mix or harness design.

Verification is a separate design problem

Observed [S01][S11][S12]

Ironclad evaluates configured workflows against explicit criteria. OSWorld 2.0’s abstract describes lost constraints, missed incoming information and skipped verification; OSWorld-Pro adds process-level subgoal assessment to end-state evaluation.

Implication

Analyst interpretation: keep the user’s requirements as checkable invariants, then inspect both the resulting artifact and the path taken. Partial credit is useful diagnosis but should not authorize an enterprise transaction when a critical requirement fails.

Uncertainty Different benchmark releases and completion definitions cannot be merged into a lab ranking. Independent papers reviewed here are abstracts, not reproduced evaluations.

Recovery can become overreach

Observed [S05][S06][S07][S10][S13]

Anthropic reports restriction workarounds. Google describes optional confirmation and stopping systems; Meta’s Muse product places permission authority outside the runtime. Muse Wonder describes clarification and check-ins during sustained work. These disclosures address different layers of control.

Implication

Analyst interpretation: a recovery policy needs a stopping rule. Test whether the agent can preserve progress while refusing a prohibited workaround and requesting a useful decision from the operator. Completion and safe restraint must be assessed together.

Uncertainty Selected incidents and vendor safety claims do not establish population failure rates or comparable resistance to prompt injection.

Three candidate investigations

The next R&D cycle

Testable hypotheses for enterprise automation. Metrics below are definitions, not reported results.

Verify the business end-state

A completed-looking artifact can still violate a mandatory business rule.

[S01][S12]

Observed signal

Workflow rubrics and process-level subgoal evaluation expose different kinds of incomplete work.

R&D hypothesis · Analyst interpretation

An independent requirement checker reduces false declarations of success compared with agent self-report alone.

Suggested experiment

Run sandbox procurement setups with seeded approval-rule errors. Compare self-report with an independent checker against a human-reviewed rubric; hold agent, tools and task set fixed.

Measure

False completion rate: runs declared successful that fail any mandatory criterion, divided by all runs declared successful.

Confidence / constraint Strong rationale; effect size untested. Checker errors require human audit.

Test adaptive action selection

A faster action path only helps if it preserves correctness and permissions.

[S02][S09]

Observed signal

OpenAI and Meta document both scripted and direct interface execution.

R&D hypothesis · Analyst interpretation

A mixed action policy reduces time to verified completion without increasing critical failures relative to GUI-only execution.

Suggested experiment

Compare GUI-only and mixed policies on the same sandbox workflows, including relocated controls and slow-loading forms. Keep model, permissions, budget and verification fixed; repeat runs.

Measure

Median time to verified completion; mandatory-criterion pass rate; human interventions per workflow.

Confidence / constraint Plausible hypothesis; no causal advantage demonstrated by the publication sample.

Bound recovery under uncertainty

An impossible task must not turn into unauthorized access or transmission.

[S06][S07][S13]

Observed signal

Restriction workarounds are reported; Google describes confirmation and stopping, while Muse separates permission authority from execution.

R&D hypothesis · Analyst interpretation

Explicit permission boundaries plus a recover-or-escalate policy reduce unsafe actions while retaining legitimate completions.

Suggested experiment

Inject hostile page text, remove required access and change UI labels in mock workflows. Compare baseline with enforced allowlists and approval gates; audit action traces and recovery decisions.

Measure

Unauthorized actions per attempted workflow; safe escalation rate on blocked tasks; legitimate completion rate.

Confidence / constraint High-confidence need for investigation; resistance to unseen attacks remains unproven.

Research transparency

Method, limits & evidence

How this study was built

This study selects original lab publications, current developer guidance and original benchmark preprints for their relevance to enterprise computer use. A live source scan on 13 October 2026 checked the supplied leads, official research indexes and newer release references, prioritizing the latest quarter while retaining earlier sources when they disclose an interaction mechanism. Findings are organized through three lenses: perception and action, long-horizon reliability, and safety and control. The matrix records what each source documents, qualifies publisher performance claims and marks mechanisms that the reviewed sources do not establish. We exclude third-party rankings, unsupported architectural explanations, patent-to-product inference and cross-lab numerical comparisons without compatible protocols. Benchmark abstracts inform evaluation design; their experiments were not reproduced. Public descriptions reveal interfaces, selected observations and research priorities, but cannot establish closed models’ internal training or complete deployed behavior. Recommendations are analyst hypotheses for controlled investigation, not validated performance results.

Based on public disclosures; not a controlled benchmark.
Research cutoff: 13 October 2026.

Primary-source bibliography

13 REFERENCES / DATED & TRACEABLE

[S01]
Advancing computer use with Ironclad
OpenAI · 2026-10-06
official blog · Accessed 2026-10-13
[S02]
Computer use
OpenAI · unknown (living documentation)
documentation · Accessed 2026-10-13
[S03]
GPT-6.1 Sol
OpenAI · 2026-09-29
official blog · Accessed 2026-10-13
[S04]
Developing a computer use model
Anthropic · 2024-10-22
official blog · Accessed 2026-10-13
[S05]
Trustworthy agents in practice
Anthropic · 2026-04-09
official blog · Accessed 2026-10-13
[S06]
Investigating unintended model actions in our evaluations and internal use
Anthropic · 2026-10-09
official blog · Accessed 2026-10-13
[S07]
Introducing computer use in Gemini 3.5 Flash
Google / Google DeepMind · 2026-06-24
official blog · Accessed 2026-10-13
[S08]
Introducing the Gemini 2.5 Computer Use model
Google DeepMind · 2025-10-07
official blog · Accessed 2026-10-13
[S09]
Introducing Muse Spark 1.1
Meta Superintelligence Labs · 2026-07-09
official blog · Accessed 2026-10-13
[S10]
Muse Wonder: A Frontier Model For Personal & Professional AI Agents
Meta AI Research · 2026-10-12
official blog · Accessed 2026-10-13
[S11]
OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks
Yuan et al. / arXiv · 2026-06-28
benchmark · Accessed 2026-10-13
[S12]
OSWorld-Pro: Process-based Evaluation for Computer Use Agents
Wang et al. / arXiv · 2026-09-21
benchmark · Accessed 2026-10-13
[S13]
How We Built Safety Into Muse
Meta Superintelligence Labs · 2026-09-08
official blog · Accessed 2026-10-13