Object-Agnostic Tracking for Dexterous Hands,
Enabling Rapid Physics-based Retargeting

Overview

Abstract

Retargeting human demonstrations into dynamically feasible dexterous hand trajectories requires resolving complex contact physics, yet current methods either omit physical constraints or rely on computationally expensive sampling-based optimization in simulation. Reinforcement learning has previously been used to achieve planning and control through contact dynamics; however, standard policies require object-specific training and fail to generalize across arbitrary geometries. This work reframes physics-based retargeting as a residual learning problem, where we demonstrate that zero-shot generalization across objects relies on structured state representations. Specifically, formulating the observation space to include a localized contact-error signal guiding each robot link to its target coordinate on the object surface enables a single, unified residual policy to successfully grasp novel, unseen object geometries. Rolling out this policy in simulation on a kinematic reference trajectory yields a physics-based retargeter, OATH. Operating at 0.4 seconds of compute per second of reference, OATH achieves 2.4× faster-than-real-time performance and a 60–350× reduction in computational cost compared to sampling-based methods. The system recovers 58% of held-out and 54% of unseen-object DexYCB trajectories while maintaining a sub-2 cm tracking error.

Method

Where the kinematic trajectory comes from

The output of the retargeting pipeline is , containing the robot and object states and , contact presence , and contact locations in the object and robot body frames , .

The key: contact-error observations

Results

Simulation

Against the baselines

Compute in seconds per second of reference, normalised to an RTX 5080. Bold is best in each row.

Hardware, open loop

BibTeX

@article{sigurdson2026oath,
  title   = {{OATH}: Object-Agnostic Tracking for Dexterous Hands,
             Enabling Rapid Physics-based Retargeting},
  author  = {Sigurdson, Solvin and Sriganesh, Prasanna and Borbon, Juan and
             Burdick, Joel and Zhang, Dingqi and Wang, Jiuguang},
  journal = {arXiv preprint arXiv:2610.14379},
  year    = {2026}
}

1. [1]C. Pan et al., SPIDER: Scalable physics-informed dexterous retargeting, 2026.
2. [2]B. Paliwal et al., Do as I do: Dexterous manipulation data from everyday human videos, arXiv:2606.19333, 2026.
3. [3]Y. Feng et al., A minimalist retargeting-guided reinforcement learning recipe for dexterous manipulation, arXiv:2607.11874, 2026.
4. [4]X. Zhu et al., Learning dexterous manipulation using contact wrench guidance from human demonstration, arXiv:2607.00033, 2026.
5. [5]J. Wu et al., TopoRetarget: Interaction-preserving retargeting for dexterous manipulation, arXiv:2606.16272, 2026.
6. [6]H. Pham and Q.-C. Pham, A new approach to time-optimal path parameterization based on reachability analysis, IEEE T-RO, vol. 34, no. 3, pp. 645–659, 2018.
7. [7]Y.-W. Chao et al., DexYCB: A benchmark for capturing hand grasping of objects, CVPR, 2021.