UI-Mate Advancing Open-Weight Foundation GUI Agents
with In-Context Demonstrations

Strong general computer use — plus one demonstration when the instruction alone is not enough.

Tencent HY Frontier

Technical Report Preprint Code Weights Try App Watch Demos
01 — Overview

What UI-Mate is

An open-weight foundation GUI agent: environment-grounded training plus in-context demonstration learning — show a procedure once instead of spelling every convention in a prompt.

OSWorld-Verified 77.0% Open weights · Competitive performance
WindowsAgentArena 66.2% Open weights · Competitive performance
OSWorkerBench 41.0% / 76.9% Strict / progress · +17.7 / +24.5 vs base
+ ONE DEMO 17.2% → 35.4% OSWorker self-demo · strict success
UI-Mate teaser: demo-guided execution and benchmark results
General computer use plus demo-guided execution. One same-task demonstration lifts GameDev, OSWorld-subset, and OSWorker self-demo scores without turning the agent into a replay script.
02 — Demos

See it run

General CUA on a real macOS desktop. DemoCUA clips coming next.

UI-Mate

Drop the recording into assets/demos/ to replace this placeholder.

03 — Approach

Three pieces

A closed-loop data and training stack, demonstration-guided execution, and an office benchmark that isolates what a demo adds.

01

Environment-grounded training

Closed-loop data flywheel: task & environment construction, filtered rollouts, capability-tree rebalancing, then SFT + online RL.

UI-Mate data flywheel
02

In-context demonstrations

Record once, then turn it into a captioned subtask workflow. The live screenshot stays authoritative — the demo is guidance, not a script.

Offline Capture and structure a demonstration
1

Record demo

Native recorder · before/after screen for every action

2

Pair evidence

Before / after screens + raw clicks, keys, text

3

Caption each step

VLM: observation · intent · action · verify

4

Group subtasks

71 events → 6 subtasks with completion criteria

5

Review & save

Human edit → reusable demo in the library

parsed into a subtask plan
Online Demo-in-the-loop at every step

Live screenshot

Authoritative live pixels

Harness Workflow Hook

Exposes only the active subtask

Current: Write sheet

Agent

Decides from screen + guidance

Desktop

Real mouse & keyboard

Harness keeps the checklist; the run ends after one or two subtasks complete.

03

OSWorkerBench

100 long-horizon office tasks across 41 apps. Paired demos isolate what one demonstration adds beyond baseline skill.

OSWorkerBench construction pipeline
04 — Results

Open-weight CUA — and more reliable with one demo

General computer use on public benches, then the lift from one demonstration.

General CUA · instruction only

Open-weight comparison. OSWorkerBench reports strict success / progress.

Model Size OSWorld-Verified WindowsAgentArena OSWorkerBench
UI-Mate-27B 27B 77.0% 66.2% 41.0% / 76.9%
UI-Mate-9B 9B 66.2% 61.7% 34.0% / 66.6%
Kimi-2.6 1T-A32B 73.1% 63.3% 40.7% / 72.4%
Qwen3.6-27B 27B 52.5%∗ 47.1% 23.3% / 52.4%
ScaleCUA-Qwen3.5 9B 68.7% 38.1% 16.3% / 38.3%

Effect of one demonstration

Average task score, without vs. with demo.

Without demo With one demo
05 — App

Run UI-Mate on your Mac

Apple Silicon macOS app. Download the DMG, then follow the usage guide for permissions, model setup, and demos.

Download DMG · 0.2.4 Usage & deploy guide

App requires Apple Silicon Mac · Intel Macs cannot run the client

06 — Citation

Cite this work

If UI-Mate is useful in your research, please cite the technical report.

BibTeX 
@article{uimate2026,
  title         = {UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations},
  author        = {Tencent HY Frontier Team},
  journal       = {arXiv preprint arXiv:2608.15930},
  year          = {2026},
}

Update the entry with the arXiv identifier and full author list once the report is public.