[Research] Contact-Grounded Policy: Dexterous Visuotactile Policy with Generative Contact Grounding
Physical Demonstration: Real-world validation of the Allegro Hand V5 Plus retrofitted with Digit360 tactile sensors, performing Jar Opening and In-Hand Box Flipping.
Paper Abstract
Contact–rich dexterous manipulation with multi–finger hands remains an open challenge in robotics because task success depends on multi–point contacts that continuously evolve and are highly sensitive to object geometry, frictional transitions, and slip. Recently, tactile–informed manipulation policies have shown promise. However, most use tactile signals as additional observations rather than modeling contact state or how their action outputs interact with low–level controller dynamics. We present Contact–Grounded Policy (CGP), a visuotactile policy that grounds multi–point contacts by predicting coupled trajectories of actual robot state and tactile feedback, and using a learned contact–consistency mapping to convert these predictions into executable target robot states for a compliance controller. CGP consists of two components: (i) a conditional diffusion model that forecasts future robot state and tactile feedback in a compressed latent space, and (ii) a learned contact–consistency mapping that converts the predicted robot state–tactile pair into executable targets for a compliance controller, enabling it to realize the intended contacts. We evaluate CGP using a physical four–finger Allegro V5 hand with Digit360 fingertip tactile sensors, and a simulated five–finger Tesollo DG–5F hand with dense whole–hand tactile arrays. Across a range of dexterous tasks including in–hand manipulation, delicate grasping, and tool use, CGP outperforms visuomotor and visuotactile diffusion–policy baselines.
Physical Validation on Allegro Hand V5 Plus
As part of a highly sophisticated AI initiative by Meta Reality Labs Research, the proposed Contact-Grounded Policy (CGP) framework was fully validated and deployed on the Allegro Hand V5 Plus as the primary physical multi-finger robotic platform.
• Seamless Hardware Integration with Digit360 Tactile Sensors
Each finger of the 4-fingered Allegro Hand V5 Plus was retrofitted with Digit360, a cutting-edge vision-based high-density tactile sensor, to capture precise contact geometry and localized forces in real time.
• High-Fidelity Real-World Task Execution
By learning from human motion-capture demonstrations, the integrated system successfully executed highly complex manipulation tasks, including Jar Opening (unscrewing a sealed jar lid) and blind In-Hand Box Flipping (flipping a box mid-air within the palm).
Key Takeaway
By utilizing the Allegro Hand V5 Plus integrated with high-resolution tactile sensors, the system demonstrated significant improvements in success rates and robustness for complex manipulation tasks—such as jar opening and in-hand box flipping—compared to existing vision-motion baselines.
PREV
