Tactile-based Object Retrieval from Granular Media

*Equal contribution, 1Columbia University, 2Stanford University

GEOTACT retrieving buried objects from granular media.

35 objects (7 training + 28 unseen) that GEOTACT can successfully retrieve.


We introduce GEOTACT, a robotic manipulation method capable of retrieving objects buried in granular media. This is a challenging task due to the need to interact with granular media, and doing so based exclusively on tactile feedback, since a buried object can be completely hidden from vision. Tactile feedback is in itself challenging in this context, due to ubiquitous contact with the surrounding media, and the inherent noise level induced by the tactile readings. To address these challenges, we use a learning method trained end-to-end with simulated sensor noise. We show that our problem formulation leads to the natural emergence of learned pushing behaviors that the manipulator uses to reduce uncertainty and funnel the object to a stable grasp despite spurious and noisy tactile readings. We also introduce a training curriculum that enables learning these behaviors in simulation, followed by zero-shot transfer to real hardware. To the best of our knowledge, GEOTACT is the first method to reliably retrieve a number of different objects from a granular environment, doing so on real hardware and with integrated tactile sensing.



      title={Tactile-based Object Retrieval from Granular Media},
      author={Xu, Jingxi and Jia, Yinsen and Yang, Dongxiao and Meng, Patrick and Zhu, Xinyue and Guo, Zihan and Song, Shuran and Ciocarlie, Matei},
      journal={arXiv preprint arXiv:2402.04536},

Questions & Answers

What is your tactile sensor?

We use the DISCO tactile finger (arXiv, video, blog post), a multi-curved optics-based tactile finger with all-around sensing coverage. It uses overlapping signals from light emitters and receivers embedded in a transparent waveguide layer. Its streamlined shapes allow smooth movement and its large sensing coverage is critical for manipulation inside granular media because contacts come from every angle.

What is the tactile observation into your policy?

The tactile observation is a a sequence of contact locations and net forces from the last ten steps.

When does your policy fail?

Our policy often fails on rounded shapes (e.g., ball, tomato soup can) when deployed on the real robot. A successful grasp of these two objects is extremely sensitive to sensor error, due to their rounded shape. Our policy locates both objects well, but the gripper simply often squeezes them out during closing.

Another common failure case of our policy on the real robot is due to a thin layer of granules stuck between the target object and the finger. When this happens, due to the low friction coefficient with the granules, the object will slip out of the formed grasp.

See here for video demonstrations of the two failure cases above.

Individual Object Demonstrations
All videos are 1X speed.

Our method is trained on a set of seven objects. It is then evaluated quantitatively (ten attempts on each object) on both the set of seven training objects and a set of six previously unseen objects. In additional testing, we also show that our method can work on an extended set of 22 unseen objects with at most three attempts per object.

7 Training Objects

Cube (52.4g)

Long Cuboid (75.2g)

Ball (36.6g)

Pentagram (16.6g)

L-shape (98.2g)

Tomato Soup Can (335g)

Potted Meat Can (365g)

6 Unseen Objects

Clamp (35.7g)

Toy Banana (15.0g)

Utility Knife (217.4g)

Screw Driver (94.6g)

Chocolate Box (19.9g)

Coconut Water Can (67.3g)

22 Additional Unseen Objects with Limited Testing


Coke Can


EMG Armband

Green Screw Driver


Joy Stick

Lego Toy

Memory Card

Mini Computer


Oil Bottle



Sugar Box

Tape Measure

Toy Bolt

Toy Bread

Toy Eggplant

Toy Lion

Toy Power Drill

Water Bottle


We would like to thank Eric Chang and Amr El-Azizi for their support on the DISCO tactile finger. We would like to thank Zhenjia Xu for his help with the granular media simulation. This material is based upon work supported by the National Science Foundation under Award No. CMMI-2037101, and the Office of Naval Research under grant N00014-21-1-4010. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation or the Office of Naval Research.