Skip to content
#Perception and localization
Oct 22Thu
  1. arXiv cs.RO55

    AFFORD2ACT studies the capability ceiling of robot manipulation policies with sparse 2D keypoint input

    AFFORD2ACT (Affordance-Guided Keypoints to Act) studies how much manipulation capability can be retained when only sparse 2D keypoints are used as input: an off-the-shelf language-conditioned affordance model locates task-relevant regions, selects up to 15 object keypoints plus 4 end-effector keypoints, compresses the policy input to 19 2D keypoints (38 coordinates), and then uses lightweight gating to decide which keypoints to focus on during execution.

  2. arXiv cs.RO38

    Extending graspable-part segmentation to multi-object images

    A study proposes introducing a laser pointer into a vision-based portable pipeline for target disambiguation in multi-object scenes, to support grasp selection and preshaping in upper-limb prostheses. The method uses a mixed reality strategy to reuse existing high-quality segmentation masks to generate training images and retrains a deep convolutional neural network. Experimental results show target disambiguation accuracy improves by up to 9% over existing solutions while computational cost remains unchanged.

  3. arXiv cs.RO55

    AOV-GS: Active Open-Vocabulary Semantic Mapping with 3D Gaussian Splatting

    AOV-GS is an active 3D Gaussian Splatting mapping system that uses frozen foundation models to guide exploration, without requiring manually annotated segmentation masks or closed semantic categories. In its coarse exploration stage, it re-ranks geometric candidate views by the novelty of CLIP mask embeddings computed from SAM instance segmentation, biasing exploration toward semantically informative regions; a lightweight scene language field fitted offline directly aligns the collected CLIP embeddings, enabling open-vocabulary text-query segmentation.

  4. arXiv cs.RO55

    RoPT: A Robust 6DoF Object Pose Tracking Framework for Video

    RoPT is a 6DoF object pose tracking framework for robotic interaction that couples visual perception, temporal kinematics, and global re-localization into a closed loop to counter catastrophic divergence in monocular pipelines under fast motion. Its visual module tracks object masks to handle partial and severe occlusions; a kinematic factor graph regularizes frame-to-frame updates, predicts forward poses, and drives parallel multi-hypothesis verification to avoid local minima; and an active re-localization module gives the system failure awareness and deterministic self-recovery.

  5. arXiv cs.RO58

    RISE: Test-Time Motion Steering for Perceptive Humanoid Fall Recovery

    RISE is a perceptive humanoid fall-recovery method that uses task objectives at inference time to steer how the robot stands up. It learns a recovery vocabulary that links intended movements to motor actions through sensory feedback; a visual diffusion planner learns from executions of this vocabulary to predict motion intent as future body states, and task costs steer that prediction toward a desired heading, rise speed, or path, after which motion latents are generated accordingly. Simulations demonstrate diverse recoveries, terrain adaptation, and test-time steering without retraining, with a real-robot demo on the Unitree G1.

  6. arXiv cs.RO50

    Object- and Task-Aware Active Perception Framework for Mobile Manipulators in Semi-Static Environments

    An arXiv paper proposes an object- and task-aware active perception framework that lets a high-degree-of-freedom mobile manipulator use its hand camera to actively inspect uncertain or safety-critical regions during navigation. The framework uses a probabilistic next-best-view (NBV) formulation and is integrated with hierarchical task model predictive control (HTMPC), exploiting kinematic redundancy to adjust viewpoints while preserving the higher priority of base-trajectory tracking. Real-world tests showed that, compared with passive or non-task-aware baselines, the system improved map consistency and reduced task-related map entropy and navigation risk.

  8. arXiv cs.RO51

    OMAR: Open-World Multi-Attribute Reasoning for Scene-Conditioned Shelf Organization

    An arXiv paper proposes OMAR, an open-world multi-attribute reasoning pipeline for inferring a shelf's latent organization and ranking placement positions for new objects without user preference instructions. A vision-language model first proposes object labels and attributes such as content and packaging type, which are grounded through segmentation, depth, and multi-view association; an LLM then reasons over an instance-level 3D scene graph to produce scene-consistent placement rankings, followed by geometric and kinematic feasibility checks before execution.

  9. arXiv cs.RO47

    Reading the Room: Scale-ambiguous Sparse Semantic VSLAM Map Registration to Floor Plans of Partially Built Environments

    Reading the Room (RR) proposes a global-local method for registering scale-ambiguous sparse semantic VSLAM maps to floor plans of partially built environments. Its core is a single robust, differentiable, class-balanced semantic-geometric energy function that runs through the entire pipeline. Evaluated on 33 sequences from 3 construction sites, it improves mean absolute trajectory error by 4.5 times over the strongest baseline in the general unconstrained setting, rising to 7.6 times when anchor correspondences are provided.

  11. arXiv cs.RO55

    ContactFlowFilter: An Online Contact Estimation Method for Robotic Arms Based on Flow Matching and Particle Filtering

    The paper proposes ContactFlowFilter for estimating the number, locations, and force vectors of active contacts on a robotic arm equipped only with joint torque sensors and a base force/torque sensor. The method uses a flow matching model as an auxiliary proposal distribution for a particle filter, leveraging the particle filter's recursive structure to capture temporal dependencies while tracking multimodal beliefs and detecting contact addition and removal events.

  12. arXiv cs.RO56

    A2I-Calib++: A Noise-Robust Active Calibration Framework for Whole-Body Multi-IMU Systems on Legged Robots

    The paper proposes A2I-Calib++, a kinematics-driven, noise-robust active calibration framework for multi-IMU systems mounted on legged robot bodies. It can estimate temporal offsets, extrinsic rotations and translations, and inertial biases without requiring artificial calibration targets or complex parameter tuning.

  14. arXiv cs.RO60

    DepState-JEPA: Depth-Grounded State Representation Learning for Visual Perception and Manipulation

    The authors propose DepState-JEPA, a dual-stream RGB-D joint-embedding predictive architecture that connects spatial understanding with action-conditioned prediction through a shared latent state, DepState. Built on V-JEPA 2.1, the framework uses depth both as a complementary observation stream and as the grounding for metric 3D space, and constrains state learning with ray-conditioned readouts, object-trajectory supervision, and calibrated forward kinematics.

  15. arXiv cs.RO51

    inr2action: Encoding Robot Episodes as Neural Implicit Representations for Recall and Action

    The paper proposes the inr2action framework, which fits a neural implicit representation to each robot manipulation episode starting from a shared meta-learned initialization; the resulting weight delta serves as memory, with a size that does not vary with episode length and stays fixed from one hundred frames to one hundred thousand frames.

  17. arXiv cs.RO49

    VeriCalib: Verifiable Target-Based LiDAR-Camera Calibration

    VeriCalib formulates target-based LiDAR-camera extrinsic calibration as a verifiable estimation problem and outputs a calibration state indicating whether to accept the result or collect more data. It uses a structured covariance to decouple rotation and translation accuracy, guiding data collection toward the weakest directions, and uses in-view and cross-view checks plus the minimum detectable deviation to define the smallest fault detectable in each direction.

  18. arXiv cs.RO38

    Real-Time Ground Normal Vector Estimation for Spherical Mobile 3D Mapping Using LiDAR-Inertial Odometry

    The study proposes two RANSAC-based ground normal vector estimation methods: single-scan estimation and estimation based on a LiDAR-inertial odometry (LIO) global 3D map, for odometry in a spherical mobile mapping system (SMMS). Evaluated on diverse real-world datasets against terrestrial laser scanner ground truth, the LIO method significantly outperforms the single-scan method in accuracy and robustness. Integrating ground normal vector estimation into the motion model improves pose estimation and supports autonomous navigation in hazardous environments.

  20. HuggingFace Daily Papers67

    FocusVLA: A VLA Model with Future-Intent-Guided Dynamic View Routing

    FocusVLA proposes a stage-aware multi-view routing framework that predicts future interaction intent from historical actions and uses it to dynamically coordinate global and local views during action generation. On RoboTwin 2.0, its average success rate improves by 10.10 percentage points over the strongest baseline, π_{0.5}; in real-world evaluation, the success rate rises from 71.7% to 85.0%. The framework's intent prediction and dynamic routing modules can be integrated into existing VLA frameworks without modifying the core architecture.

  21. Reddit r/robotics49

    Home RGB + iToF depth build: noise reduced but straight walls still bow 12–14 cm

    A Reddit user built a home RGB + iToF depth setup using a 320×240 epc660-class iToF engineering sample, a fixed Pi camera, and Ouster reference scans. After raising exposure from 500 to 2,000 µs, filtered jitter dropped from 65 to 6 mm and raw jitter from 85 to 28 mm, but straight walls still bow 12–14 cm, and three modulation frequencies report 4.14, 3.92, and 3.97 m for the same wall.

  22. 差评X.PIN · 微信公众号57

    Differential Review's On-Site Visit: How Shengtong Energy and Qiten Robotics' Metaverse Intelligent Inspection Enables Inspection Robots to Work Across Plants

    Differential Review traveled to Chongqing to speak with engineers from Shengtong Energy and Qiten Robotics, and introduced the Metaverse Intelligent Inspection solution for high-risk industrial inspection. The solution first uses reconstruction technologies such as 3D Gaussian to build a digital factory on a computer, and a large model automatically identifies instruments.

Oct 21Wed
  1. HuggingFace Daily Papers47

    Slot-JEPA: An Object-Structured Prediction Interface for Latent World Models

    Slot-JEPA decomposes JEPA's dense patch representations into stable semantic slots and time-varying spatial masks, providing a reusable object-structured interface. Built on this interface, the Transition Memory Predictor (TMP) stores visual transitions as compact masks, retrieves context using slot statistics, and uses unbalanced optimal transport (UOT) to supervise mask prediction and evaluate candidate plans.

  2. 古月居 · 微信公众号58

    Xinming Shares Spatial Intelligence Chip and 3D Vision AI Module Solution at ROSCon China 2026

    Xinming gave a keynote at ROSCon China 2026, where Vice President Dr. Zhou Fan presented its spatial intelligence chip and 3D vision AI module. The self-developed chip integrates real-time 3D stereo vision perception, on-device AI, and SLAM on a single chip, delivering about 3.5 TOPS of on-device compute at under 1 W power consumption, with roughly 1 ms motion-to-display latency, and supports 6 cameras and multi-sensor fusion.

  3. arXiv cs.RO45

    Gibson Env: A Real-World Perceptual Virtual Environment for Embodied Agents

    Gibson Virtual Environment provides a perceptual training environment for embodied agents by virtualizing real spaces, and currently contains more than 1,400 floor spaces from 572 full buildings. Its built-in "Goggles" synthesis mechanism allows trained models to be deployed directly to the real world without additional domain adaptation. The environment subjects agents to physical and spatial constraints, and the paper demonstrates sample perceptual tasks learned within it; the code, dataset, and project website are openly available.

  4. arXiv cs.RO38

    Visual-Inertial Odometry for Ground Robots with Dynamic Camera-IMU Extrinsic Calibration

    The paper proposes a dynamic camera-IMU extrinsic calibration method based on the Denavit-Hartenberg convention, enabling online estimation of time-varying extrinsics by incorporating a robot kinematic model. A companion two-step initialization strategy efficiently estimates the initial joint angles, providing a reliable starting point for subsequent extrinsic parameter optimization. Simulation experiments show that the framework can effectively accommodate changes in camera-IMU extrinsics while maintaining competitive localization accuracy.

  5. arXiv cs.RO61

    ForeTac: A Predictive Contact Guidance Framework for Generative Robot Policies

    ForeTac is a predictive contact guidance framework for contact-rich manipulation that uses a represent-predict-evaluate-guide loop: it encodes the current tactile state, predicts future tactile latent trajectories conditioned on candidate action sequences, and evaluates each trajectory with a contact-quality objective learned from successful and failed physical interactions. At inference, gradients of this objective are backpropagated through the tactile predictor to guide action generation before execution, while all model parameters remain frozen.

  6. arXiv cs.RO45

    CoGaussian: A Gaussian-based open-vocabulary collaborative 3D scene understanding framework

    CoGaussian proposes an open-vocabulary collaborative semantic occupancy prediction framework that uses sparse 3D Gaussians aligned with foundation models to represent scenes as a compact representation for transmitting and fusing geometric and semantic information among multiple agents, no longer relying on dense 3D voxel annotations and no longer limited to predefined semantic categories. On the Semantic-OPV2V benchmark, the method outperforms existing open-vocabulary single-agent approaches.

  8. arXiv cs.RO45

    Giving Models a Finger: Developmentally Aligned Evaluation of Counting in Vision-Language Models

    A study of 9 vision-language models, 682 images, and 7 evaluation conditions found that models' exact-counting ranges differ by nearly two orders of magnitude (R80 from 3 to 150 dots). Using a training-free agent loop that lets models mark images to aid counting, the study found this method mostly failed as an intervention but produced auditable serial reasoning traces that help reveal how models arrive at answers. Experiments also showed that models often change answers when traversal order is reversed, and that a child-like advantage from regular arrangements appeared only under specific conditions.

  9. arXiv cs.RO55

    PanoGraphGS unifies panoptic segmentation and hierarchical 3D scene graph construction, releases NaviGS dataset

    PanoGraphGS integrates panoramic 3D Gaussian representation learning and hierarchical 3D scene graph construction into a single framework, learning semantic and instance features on shared Gaussian geometry to achieve cross-view-consistent panoptic prediction and novel-view panoptic segmentation.

  10. arXiv cs.RO55

    SynCalib: Breaking Dataset Shortcuts in LiDAR-Camera Calibration via Extrinsic-Controllable Synthesis

    The paper proposes SynCalib, a data generation framework that addresses overfitting and poor generalization caused by insufficient extrinsic diversity in training data for deep learning-based online LiDAR-camera calibration. Using omnidirectional image datasets annotated with depth ground truth, it synthesizes multimodal calibration data under strict extrinsic constraints. Experiments show that this pipeline significantly improves the generalization and robustness of learning-based calibration networks, and real-world evaluations across various camera-LiDAR configurations validate the zero-shot transfer capability of models trained solely on synthetic data; the data generation code, pretrained models, and online benchmark dataset will be made public.

  11. arXiv cs.RO44

    OmniVLoc: A Topo-metric Visual Localization System for Omnidirectional Cameras

    OmniVLoc proposes a surround-view visual localization system for omnidirectional cameras based on a topo-metric framework. It fuses global place recognition with monocular depth estimation to achieve multi-constrained localization, and solves a scale-aware pose graph and spherical bundle adjustment to recover metric-level poses. The system includes a hierarchical verification mechanism that filters erroneous outputs at each stage to improve robustness. Experiments on public datasets and a custom dataset validate its accuracy and reliability, and the source code and dataset will be released after the paper is published.

  12. arXiv cs.RO40

    Curvature as a Baseline: Dual-Color Stripe Markers Enable Monocular Pose Measurement of Cylindrical Objects

    To address the limited surface area on thin cylindrical objects that makes planar markers difficult to place, a dual-color stripe marker is proposed. It uses the cylinder's known curvature to improve monocular camera pose measurement accuracy, with the two-color boundary providing a long centerline and dots and printed edges providing features at different depths.

  14. arXiv cs.RO62

    REACT: VLM-Guided Contact Retargeting and Generative Partner Modeling for Humanoid-Human Interaction

    REACT is a framework for learning deployable humanoid-human interaction policies from human-human demonstrations, using semantic contact structure as the supervisory signal that links demonstration processing, interaction learning, and onboard control. It uses a VLM-guided contact modeling and retargeting pipeline to identify semantic contacts and transfer them onto humanoid-human geometry, then distills a privileged interaction policy into an egocentric depth policy that relies entirely on onboard sensing, and introduces a generative partner state modeling objective to predict partner motion from occluded viewpoints.

  15. arXiv cs.RO62

    MARE: An open-source lunar rover simulator with shadow ground truth

    The authors present MARE, an open-source lunar rover perception and autonomy simulation suite built on Unreal Engine 5 and ROS 2, comprising three parts: terrain generation, dual-physics lunar rover sensor simulation, and GNC mission control. The simulator provides stereo, lidar, event, and SPAD camera data along with pixel-level ground truth; at low solar elevation angles, the mean IoU between predicted shadows and rendered shadows is 0.79. Stereo depth analysis shows that pixel coverage in shadowed regions is lower than in sunlit regions.

  16. arXiv cs.RO55

    From Predictive Geometry to Action: Learning Complementary Predictive Representations for Embodied Manipulation

    The paper proposes a two-stage framework that learns complementary predictive representations from multi-view video clips to enhance embodied manipulation policies. In the pretraining stage, a frozen 3D foundation model provides dense geometric features, which are compressed into predictive geometric representations and optimized to preserve scene structure and its temporal evolution; action supervision is then applied to geometric transitions to extract a motor representation that emphasizes action-relevant transitions. In the downstream post-training stage, predictive geometry is transferred through geometry queries, and the motor representation guides intermediate features along the action-generation pathway. Future observations serve only as privileged supervision during training and are not needed at inference time.

  17. arXiv cs.RO57

    Tactile-ETA Achieves Fine Manipulation Through Tactile In-Context Learning

    The Tactile-ETA system brings tactile experience into embodied task agents through in-context learning, completing fine manipulation without updating model parameters. Across four simulated manipulation tasks, adding tactile observations to the same two expert demonstrations raised average success from 15.75% to 24.00%; on a real robot task, vision- and tactile-aligned demonstrations raised success from 0% to 60%. Experiments also show that performance depends on the sensory content and representation of the demonstrations, and different agent models differ markedly on the Lift Bottle task.

  18. arXiv cs.RO47

    Orthogonal Residual Learning for Adaptive Sideslip Angle and Cornering Stiffness Estimation

    An orthogonal residual learning method constrains a model corrector via null-space parameterization to be orthogonal to cornering stiffness sensitivity, preventing the corrector from absorbing stiffness-change information and suppressing online adaptation. In full-vehicle simulation, the method reduces sideslip angle error by 12.7% to 22.0% and maintains the same adaptation speed as an uncorrected filter under step changes in stiffness. On public real-vehicle data, a 430-parameter orthogonal corrector retained more response to nominal stiffness offsets than an unconstrained corrector of the same size.