Retrieval Dexterity: Efficient Object Retrieval in Clutters with Dexterous Hand

Retrieving objects buried beneath multiple objects is not only challenging but also time-consuming. Performing manipulation in such environments presents significant difficulty due to complex contact relationships. Existing methods typically address this task by sequentially grasping and removing each occluding object, resulting in lengthy execution times and requiring impractical grasping capabilities for every occluding object. In this paper, we present a dexterous arm-hand system for efficient object retrieval in multi-object stacked environments. Our approach leverages large-scale parallel reinforcement learning within diverse and carefully designed cluttered environments to train policies. These policies demonstrate emergent manipulation skills (e.g., pushing, stirring, and poking) that efficiently clear occluding objects to expose sufficient surface area of the target object. We conduct extensive evaluations across a set of over 10 household objects in diverse clutter configurations, demonstrating superior retrieval performance and efficiency for both trained and unseen objects. Furthermore, we successfully transfer the learned policies to a real-world dexterous multi-fingered robot system, validating their practical applicability in real-world scenarios.

In this work, we have presented a novel approach to efficient object retrieval in cluttered environments using dexterous multi-finger hands. Our system demonstrates the ability to manipulate occluding objects strategically, exposing target objects for retrieval—a capability that significantly improves upon traditional sequential removal methods. Through careful design of our simulation environment and reinforcement learning framework, we have addressed key challenges including time efficiency, object diversity, and the complexity of high-dimensional control in contact-rich environments. Our experimental results, both in simulation and real-world settings, validate the effectiveness of our approach. The system successfully generalizes across diverse objects and achieves zero-shot transfer to real-world robots, demonstrating robust performance without additional training. This work represents a step toward more efficient and capable robotic manipulation in cluttered environments, though opportunities remain for future exploration, particularly in achieving fully autonomous operation through integration with advanced perception systems.