05 — App

Usage & deploy

Install the macOS app, grant permissions, connect a model endpoint, then run tasks or record a demo.

1. Before you start

Machine	Apple Silicon Mac (M1/M2/M3/M4) for the desktop app. Intel Macs cannot run the client.
System	macOS 13 Ventura or later
Disk	About 2 GB free
Model service	Required — the app has no built-in model. See Deploy.

Download for macOS

Open the DMG and drag UI-Mate.app into Applications.
Launch from Applications — do not run from inside the DMG.
If Allow applications from is set to App Store only, switch it to App Store & Known Developers.

Allow applications from setting — System Settings → Privacy & Security → Security.

2. Grant macOS permissions

Screen Recording and Accessibility are required. Use the in-app buttons — do not add the app manually in System Settings.

Permission	UI label	Why
Screen Recording	Screen Recording	See the screen; record demos
Accessibility	Accessibility	Click and type; capture your demo actions

Tap Open Screen Recording → Allow.
Tap Open Accessibility → enable UI-Mate.
Fully quit with ⌘Q and reopen so the banner clears.

Permissions banner — Red banner when permissions still need review.

Screen Recording permission — Enable Screen Recording from the in-app prompt.

3. Connect a model service

Open Settings (gear). Changes save automatically. Fill Model Configuration, then Test Connection.

Field	What to enter
Agent type	UI-Mate (default, for our trained models)
Endpoint	Service URL, usually ending in `/v1`
Model	Served model name — must match the server exactly
API Key	Leave blank if unused
Provider	`openai_compatible`

No endpoint yet? Ask whoever runs the model service, or follow Deploy. For a quick dry-run only: OpenRouter + Agent type Kimi — not the project model.

Model Configuration — Agent type UI-Mate or Kimi.

4. Run a task & record a demo

First task

New Conversation
Describe the outcome in the input box.
Press Enter. Avoid fighting the mouse; Stop / Esc to cancel.

Teach with a demo

Demo Library → Record Demo, perform the flow, then Stop Recording (⌘⇧S).
Process until status is Reusable.
Apply (or /demo) → mode becomes demo-in-the-loop, then send the task.

The demo is guidance, not a frame replay — the live screen stays authoritative.

Main tabs — Conversation · Previous Tasks · Demo Library

demo-in-the-loop — Input mode after applying a reusable demo.

Demo recording — Demo Library and recording flow.

5. Quick FAQ

Symptom	Fix
Permissions banner stuck	⌘Q and reopen; remove old app copies from Applications, then re-grant.
Test Connection 404 model not found	Model name must match the server exactly.
404 with correct model	Endpoint must end with `/v1` for UI-Mate.
Cannot start a new task	Only one run at a time — Stop first.
Apply does nothing	Process the recording until it is Reusable.

What “deploy” means

The app is an OpenAI-compatible HTTP client. Deploy = run a VLM service, then fill three fields in Settings.

Field	Notes
Endpoint	Base URL, usually ending in `/v1`
Model	Served model name — exact match
Agent type	UI-Mate for our trained models · Kimi for Kimi-family endpoints

OpenRouter Kimi — zero deploy

Fastest path to try the UI. Not the project model.

Agent type	Kimi
Endpoint	`https://openrouter.ai/api/v1`
Model	`moonshotai/kimi-k2.6`
API Key	Your OpenRouter key

GPU 27B — vLLM service

Team / eval setup on a multi-GPU box. Example launch:

vllm serve "$MODEL_PATH" \
  --served-model-name UI-Mate-27B \
  --trust-remote-code \
  --chat-template-content-format openai \
  --limit-mm-per-prompt '{"image":10,"video":0}' \
  --tensor-parallel-size 2 \
  --data-parallel-size 4 \
  --max-model-len 32768 \
  --host 0.0.0.0 --port 8080

--limit-mm-per-prompt image=10 must cover history screenshots.
App Endpoint: http://<GPU_IP>:8080/v1 · Model = served name · Agent UI-Mate.
Do not expose unauthenticated vLLM to the public internet.

Local Mac 9B — MLX 4bit

Offline on Apple Silicon. Convert, serve, then point the app at localhost.

python -m mlx_vlm.convert \
  --hf-path /path/to/9b-checkpoint \
  --mlx-path ~/models/UI-Mate-9B-mlx-4bit \
  -q --q-bits 4 --q-group-size 64

KV_BITS=4 PREFILL_STEP_SIZE=1024 \
python -m mlx_vlm.server \
  --model ~/models/UI-Mate-9B-mlx-4bit \
  --port 8080

Endpoint	`http://127.0.0.1:8080/v1`
Model	`~/models/UI-Mate-9B-mlx-4bit`
Agent type	UI-Mate

Local must-tunes: image detail balanced_1280, keep 1–2 screenshots.

Optional · MLX prefix-cache patch

Stock mlx-vlm barely hits prefix cache on multi-turn agent loads. Download the patch pack, unzip, apply — about 3× prefill and ~2× end-to-end step time.

Download patch zip

pip install mlx-vlm==0.6.8
unzip mlx-vlm-gui-agent-cache.zip
cd mlx-vlm-gui-agent-cache && ./apply.sh

export APC_ENABLED=1
export APC_GROWING_IMAGES=1
export MLX_INLINE_IMAGE_POSITIONS=1
export MLX_VLM_PER_IMAGE_VISION_CACHE=1
export MLX_VLM_VISION_CACHE_SIZE=20
export APC_EXACT_CACHE_ENTRIES=8
export APC_EXACT_PREFIX_GUARD_TOKENS=64

Patch targets mlx-vlm 0.6.8; features stay off until those env vars are set when starting the server.
After patching you can keep more screenshots; still prefer balanced_1280. Prefer 6-bit quant when memory allows.
Confirm in server logs: Prefill completed shows cached_tokens > 0.