Computer-Use GUI Agent: Post-Training and Self-Evolving Evaluation
Problem
Automating business workflows on web and desktop interfaces means a GUI agent has to search, fill forms, move across pages, and submit results over many steps, with high demands on action correctness, error recovery, and long-horizon stability — a base model is far from usable out of the box. At the same time, task types, page states, and action paths keep changing, so manual task-by-task review is expensive and slow to reveal capability gaps. The project therefore has two halves: training the agent (post-training) and measuring it reliably (automatic evaluation with a self-evolving benchmark).
1. Post-training: SFT → offline step-RFT → online GiGPO → multi-platform MOPD
Multi-source SFT trajectories and mixing. Four trajectory pipelines — teacher trajectories, switch-policy correction trajectories, hint-guided rollouts (rewritten into CoT afterwards), and rejection sampling — with mixing experiments to choose the data blend. Teacher-only SFT alone gives a 9.7% gain.
Switch-policy correction trajectories. A weak policy first takes a wrong action, then a strong policy takes over and corrects it, covering fail-and-recover situations. The loss is masked on the erroneous steps that were later corrected, so the model does not imitate them:
where
Offline step-level GRPO (step-RFT). With the trajectory context
A rule-based verifier scores format, action type, and arguments in three stages; a click
earns 1 inside the golden box
Initialized from the SFT checkpoint with the highest share of correction trajectories, step-RFT adds a further 4%.
Verifiable fine-grained reward. Task progress is the main term, with penalties for side effects, false completion, aborting after success, and overrunning the step budget (schematic):
A false-completion correction on wrong AnswerSheet answers stops the agent from collecting reward by merely claiming to be done.
Online GiGPO two-level advantages + DAPO clip-higher. GiGPO normalizes at two levels. At the
episode level, the
At the step level, steps from different trajectories that share the same state
The policy update uses DAPO’s asymmetric clipping (clip-higher,
Compared with trajectory-level GRPO, the two-level advantage gives much finer-grained credit assignment.
Multi-platform MOPD: merging platform specialists into one agent. Operating systems differ
widely in UI elements, shortcuts, and interaction conventions, and mixing every platform into one
RL run makes them interfere. So after GiGPO, a separate GiGPO specialist is trained for each of
macOS, Linux, Windows, and Web —
In practice the student optimizes a surrogate with training–inference importance sampling, where
The token-level advantage is the specialist’s log-probability ratio plus an outcome-reward
advantage — here the verifiable reward and GiGPO advantage
The student thus absorbs each specialist’s platform-specific behavior while keeping task completion as the global objective, ending with one model that covers all four platforms.
2. Automatic evaluation and a self-evolving benchmark
A closed loop of trace evaluation → benchmark self-evolution: each task is decomposed into atomic scoring points, judged one by one by an LLM-as-Judge, and routed to a simple or a complex evaluation path by task type.
- Windowed point judging and complex-task checks. The trace is cut chronologically into evidence windows; points are judged in batches per window and aggregated by group. For information-seeking tasks, only page facts the agent actually observed are extracted, catching answers that are right but whose evidence was invented. For looping tasks, the entities acted on are extracted per window, deduplicated across steps, and the execution count is checked, avoiding both missed and double counts.
- Benchmark self-evolution. Four mutation operators turn seed tasks into harder ones along the solution chain; capability constraints built from real-device traces keep every new task achievable. Candidates are written into the benchmark only after real-device reachability checks and independent review, so the benchmark keeps growing on its own.
Results
| Metric | Result |
|---|---|
| Success on the 200-task expert-solution benchmark | 31.3% → 49.2% (SFT → step-RFT → GiGPO) |
| Gain from teacher-only SFT | +9.7% |
| Gain from step-RFT | +4% |
| LLM-as-Judge agreement with human labels | 94.3% (complex tasks 83.2%) |
Formula references: GRPO from DeepSeekMath (Shao et al., 2024), GiGPO from Feng et al. (2025), MOPD from the MiMo-V2-Flash technical report (Xiaomi, 2026), and DAPO from Yu et al. (2025). Equations marked “schematic” are structured descriptions of this project’s design, with coefficients omitted.