GuirlVG: Incentivize GUI Visual Grounding via Empirical Exploration on Reinforcement Learning
By: Weitai Kang , Bin Lei , Gaowen Liu and more
Potential Business Impact:
Teaches computers to understand screens with less training.
Graphical user interface visual grounding (GUI-VG), a core capability for GUI agents, has primarily relied on supervised fine-tuning (SFT) of multimodal large language models (MLLMs), which demands extensive data curation and significant training costs. However, as MLLMs continue to advance and even cover GUI domains during pretraining, the necessity of exhaustive SFT post-training becomes increasingly questionable. Meanwhile, recent successes of rule-based reinforcement fine-tuning (RFT) suggest a more efficient alternative. Despite this promise, the optimal manner of applying RFT for GUI-VG remains unexplored. To bridge this gap, we introduce GuirlVG, a reinforcement learning-based GUI-VG method built on a systematic empirical study and a novel stabilization technique. We find that naive application of RFT underperforms the SFT baseline, motivating a deeper exploration. First, we decompose RFT into its core components and analyze the optimal formulation of each. Second, we propose a novel Adversarial KL Factor that dynamically stabilizes training to mitigate reward over-optimization. Third, we further explore the training configurations of RFT to enhance effectiveness. Extensive experiments show that GuirlVG, with only 5.2K training samples, outperforms SFT methods trained on over 10M samples, achieving a 7.7% improvement on ScreenSpot, a 17.2% improvement on ScreenSpotPro, and 91.9% accuracy on ScreenSpotV2.
Similar Papers
Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning
Artificial Intelligence
Helps computers understand and click on screen buttons.
A Survey on GUI Agents with Foundation Models Enhanced by Reinforcement Learning
Artificial Intelligence
Helps computers understand and use apps like people.
InfiGUI-G1: Advancing GUI Grounding with Adaptive Exploration Policy Optimization
Artificial Intelligence
Helps computers understand screen buttons from pictures.