HexaShell: Exploring the Limits of VLM Coding Agents as Robot Agents

Researchers propose HexaShell, a system in which a coding agent built on a vision-language model drives a dual-arm robot through a single command line; the framework only converts the agent's commands into robot motion. After each failure, the agent reads the log, identifies the cause, and writes or fixes small tool programs. The tools use only the robot's own observations, determining the relative positions of objects through triangulation and, when necessary, moving a wrist camera to obtain a second viewpoint.