T-Rex / Sticks: Visuotactile Policy Adaptation

Ongoing Research · Visuotactile Learning & Contact-Rich Control

Research Scope

In contact-rich manipulation tasks such as tight-tolerance insertion (Sticks task), visual observations alone cannot resolve sub-millimeter alignment errors or contact forces once parts engage. Building upon the open-source T-Rex framework, this research investigates how multimodal visual and tactile feedback can drive adaptive reinforcement learning policies on a Piper robot platform.

Technical Architecture

• Visuotactile Sensorimotor Fusion: Coupling global visual context ($\mathbf{o}_t^{\text{vis}}$) with local tactile shear and normal force fields ($\mathbf{o}_t^{\text{tac}}$) to infer contact states and geometric misalignment.
• Policy Adaptation via Reinforcement Learning: Training and adapting continuous manipulation policies to compensate for contact uncertainties during physical interaction.
• Hardware-Data Loop Integration: Establishing deterministic synchronization between continuous executed actions and recorded multimodal interaction trajectories on the Piper manipulator.