TouchScale Brings 500 Hours of Human Vision and Touch to Robot Learning

Why visual data is not enough

A camera can show where an object is and how a hand moves, but it does not directly reveal whether contact has occurred, how much pressure is being applied, or whether an object is beginning to slip. These signals are particularly important for embodied systems that must manipulate objects rather than merely recognize them. TouchScale addresses this gap with a large-scale dataset of human interactions that combines first-person vision with touch.

The project introduces approximately 500 hours of contact-rich recordings collected with one unified wearable setup and a consistent protocol. The data covers about 2,000 predefined task descriptions, including everyday activities and more structured manipulation. Every recording temporally aligns three streams: egocentric RGB-D video, wrist-mounted RGB video, and dense full-hand tactile measurements from both hands.

What the experiments show

The authors evaluate whether increasing the amount of consistently collected visual-tactile data benefits both perception and robot control:

• Cross-sensor contact prediction: When trained on the full TouchScale dataset, a model reaches a zero-shot contact IoU of 0.383 on data from an unseen tactile sensor, compared with 0.134 when trained on prior tactile data.
• Action recognition: A visual encoder pretrained on TouchScale achieves the best accuracy among the compared visual-tactile datasets on three action-recognition benchmarks.
• Robot manipulation: Visual-tactile mid-training raises the average real-world success rate on four contact-rich tasks from 22.5% to 57.5%.

The study also reports an overall upward trend in tactile prediction and robot success as progressively larger TouchScale subsets are used, while keeping the sensing hardware and collection protocol fixed. This design makes the scaling comparison more informative than simply combining datasets produced under different conditions. At the same time, the results describe the evaluated settings; they do not establish that performance will improve linearly for every task or robot platform.

Why it matters

TouchScale’s contribution is not only its duration. Its synchronized and standardized recordings make it easier to study how visual context and tactile contact complement one another. Vision can provide information about the scene and the ongoing action, while touch supplies a direct signal about physical interaction. Together, they offer richer supervision for contact prediction, action understanding, and policy learning.

A 100-hour subset is currently available on Hugging Face under the dataset’s access conditions, while the complete 500-hour release is planned. The project also intends to release synchronized visual-tactile recordings and reconstructed object models. Future evaluation will need to clarify how well the data transfers across tasks, tactile sensors, and robot embodiments.

Source: Hugging Face Daily Papers