NeoteAI and Fudan Put Tactile Data at the Center of Embodied AI

Lead

A robot may visually recognize a cup, a plug, or a towel, but that does not mean it can hold, insert, or fold it correctly. Many failures in embodied AI happen in the final centimeter of interaction: the plug rubs against the socket edge, a plastic bottle is squeezed too hard, or a neatly folded towel falls apart at release.

The N0 series of reports from NeoteAI and Fudan University’s Institute of Trustworthy Embodied AI addresses this gap. Instead of treating touch as a secondary sensor, the work frames tactile perception as a core layer for robot data, policy learning, and world modeling.

Key points

• N0-Foundation builds a tactile data base. Tactile sensors often speak incompatible “languages”: gel deformation images, capacitance matrices, or six-axis force vectors. NeoteAI’s NeoData dataset aims to unify these signals at scale. It contains more than 30,000 hours of vision-tactile interaction data, about 1.4 million operation segments, 3.3 billion time steps, 8 billion RGB frames, and 10 billion tactile images. The data spans 450 real long-horizon tasks and six robot embodiments including Franka, Piper, and UR5e. The project has open-sourced 5,000 hours of data.

• NeoForce seeks a common tactile representation. The proposed model learns transferable, temporally structured tactile features across different hardware outputs. In practical terms, it helps downstream robot policies reason about where contact happens, how much force is applied, and whether lateral dragging or slip is occurring.

• N0-VTLA predicts future touch instead of only reading current touch. The reports argue that tactile signals are sparse and bursty, so simple fusion with visual tokens can make touch easy to ignore. Current tactile feedback is also inherently reactive. N0-VTLA instead predicts tactile evolution over the next 50 steps, allowing the action module to adjust earlier. In the reported plug-in task, success reached 85% versus 60% for a vision-only counterpart; in a key-pulling task, it reached 99% versus 35%.

• Failure data becomes useful. N0-VTLA also uses tactile information with post-deployment data and human-in-the-loop data to train a progress evaluation model. Combined with offline reinforcement learning, reported success rates rose from 50% to 95% for towel folding, 35% to 80% for backpack packing, and 20% to 75% for carton folding.

• N0-TWAM brings touch into world models. Rather than predicting only future video, N0-TWAM jointly models future video, tactile signals, and actions. It uses a mixture-of-experts design with asynchronous video, touch, and action experts, totaling 7.2 billion parameters. The report claims 84.5% average success in simulation and 46.3% average success across eight real-robot tasks.

Why it matters

The broader implication is a shift from vision-led robotics toward vision-touch fusion. Many manipulation failures do not come from poor object recognition, but from missing information about force, friction, contact, and slip. A scalable tactile dataset, unified representations, and predictive touch modeling could therefore become important foundations for dexterous and reliable robots.

Challenges remain: collecting tactile data is costly, generalizing across robot bodies and sensors is difficult, and real-time inference may add engineering constraints. Still, if the released data and models are widely reused, tactile perception may move from a niche research topic to a core infrastructure layer for embodied AI.

Source: QbitAI