Qualifying Exam

Qualifying Exam Evolving Multimodal Agents through Knowledge-grounded Simulation and Synthesis
Export Event Save iCal Send to Google Calendar Send to Yahoo Calendar Send to Outlook Live Send to Microsoft Outlook Save iCal Preserve formatting in description (only supported in some calendar applications) Close Wednesday, April 08, 2026, 10:00am - 11:30am

Speaker: Kai Mei

Bio

Location : CBIM 22

Committee: 

Professor Dimitris N. Metaxas

Professor Hongyi Wang

Professor Ruixiang Tang

Professor Mingsum Kim

Event Type: Qualifying Exam

Abstract: The development of Large Language Models (LLMs) catalyzes the long-term planning and decision-making capabilities of multimodal agents in sophisticated digital environments (such as computer-use scenarios). Multimodal agents navigate digital environments by perceiving visual states and executing discrete actions to fulfill complex user intents. However, the advancement of these agents is severely bottlenecked by a reliance on human-in-the-loop supervision, which is expensive, difficult to scale, and insufficient for capturing the diversity of real-world interaction traces. We argue that multimodal agents are able to continuously evolve with limited human supervision if their knowledge can be well grounded to specific environments. Specifically, we first propose the Retrieval-augmented World Model (R-WoM), a framework that enables evolution by grounding agent simulations in external, environment-specific tutorials. This grounding stabilizes long-horizon reasoning and reduces compounding hallucinations, yielding relative performance improvements of up to 23.4% on realistic benchmarks (OSWorld and WebArena). Second, we introduce GUIDED, which drives the evolution of multimodal perception and action through the self-distillation of internal coding knowledge into executable trajectories and programmatic verifiers. By utilizing Group Relative Policy Optimization (GRPO) on these synthesized signals, the agent bootstraps its own performance, surpassing human-annotated baselines in efficiency and success of task completion. Moving forward, we plan to extend this evolving paradigm by continuously refining the world models of multimodal agents to help them adapt to more unseen environments.

Organization: 

Contact  Professor Dimitris Metaxas

Zoom Link: https://rutgers.zoom.us/my/km1558?pwd=MUF0QldkeFRqTWtnbXZBYlcxeVREQT09