We introduce GEOTACT, a robotic manipulation method capable of retrieving objects buried in granular media. This is a challenging task due to the need to interact with granular media, and doing so based exclusively on tactile feedback, since a buried object can be completely hidden from vision. Tactile feedback is in itself challenging in this context, due to ubiquitous contact with the surrounding media, and the inherent noise level induced by the tactile readings. To address these challenges, we use a learning method trained end-to-end with simulated sensor noise. We show that our problem formulation leads to the natural emergence of learned pushing behaviors that the manipulator uses to reduce uncertainty and funnel the object to a stable grasp despite spurious and noisy tactile readings. We also introduce a training curriculum that enables learning these behaviors in simulation, followed by zero-shot transfer to real hardware. To the best of our knowledge, GEOTACT is the first method to reliably retrieve a number of different objects from a granular environment, doing so on real hardware and with integrated tactile sensing.
@article{xu2024geotact,
title={Tactile-based Object Retrieval from Granular Media},
author={Xu, Jingxi and Jia, Yinsen and Yang, Dongxiao and Meng, Patrick and Zhu, Xinyue and Guo, Zihan and Song, Shuran and Ciocarlie, Matei},
journal={arXiv preprint arXiv:2402.04536},
year={2023}
}We use the DISCO tactile finger (arXiv, video, blog post), a multi-curved optics-based tactile finger with all-around sensing coverage. It uses overlapping signals from light emitters and receivers embedded in a transparent waveguide layer. Its streamlined shapes allow smooth movement and its large sensing coverage is critical for manipulation inside granular media because contacts come from every angle.
The tactile observation is a a sequence of contact locations and net forces from the last ten steps.
Our policy often fails on rounded shapes (e.g., ball, tomato soup can) when deployed on the real robot. A successful grasp of these two objects is extremely sensitive to sensor error, due to their rounded shape. Our policy locates both objects well, but the gripper simply often squeezes them out during closing.
Another common failure case of our policy on the real robot is due to a thin layer of granules stuck between the target object and the finger. When this happens, due to the low friction coefficient with the granules, the object will slip out of the formed grasp.
See here for video demonstrations of the two failure cases above.
Our method is trained on a set of seven objects. It is then evaluated quantitatively (ten attempts on each object) on both the set of seven training objects and a set of six previously unseen objects. In additional testing, we also show that our method can work on an extended set of 22 unseen objects with at most three attempts per object.
Calculator
133.3g
Coke Can
13.1g
Cup
34.0g
EMG Armband
95.7g
Green Screw Driver
57.9g
Gripper
76.7g
Joy Stick
172.4g
Lego Toy
51.6g
Memory Card
43.5g
Mini Computer
99.9g
Mouse
101.5g
Oil Bottle
234.0g
Pen
4.5g
Spatula
33.3g
Sugar Box
470.0g
Tape Measure
117.9g
Toy Bolt
11.6g
Toy Bread
47.6g
Toy Eggplant
10.0g
Toy Lion
31.1g
Toy Power Drill
131.3g
Water Bottle
61.6g
We would like to thank Eric Chang and Amr El-Azizi for their support on the DISCO tactile finger. We would like to thank Zhenjia Xu for his help with the granular media simulation. This material is based upon work supported by the National Science Foundation under Award No. CMMI-2037101, and the Office of Naval Research under grant N00014-21-1-4010. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation or the Office of Naval Research.