Multimodal large models may replace robot-specific policies

No training needed, no sim-to-real gap, just a prompt and a vision. Sentdex points out that every task in robotics today is a complex combination of locomotion, manipulation, vision, planning, and more, and fails once it deviates from training data or simulation characteristics. As a result, many demos are conducted only under perfect lighting and carefully placed objects, and never land under unknown real-world conditions—every deployment is a unique edge case. In the example, a general-purpose multimodal large language model (DSV4.1F) completed a robot task it was never specifically trained for, using an unseen camera, unlearned servo motors/motors, an environment it had never handled, and a morphology it had never been tested on. The video is at 10x speed and is currently slow, but this is already the slowest state at this level of intelligence. Sentdex says that the more he uses general-purpose multimodal large models, the more he realizes that specialized methods such as ACT and VLA may be mere supporting acts and distractions, and will eventually be swallowed by general models that do robotics incidentally. He does not think it will always be large language models, but the fact that large models can complete the most challenging robot tasks shows that... —Sourccey functionality is continuously improving.