NeurIPS 2026
1Seoul National University 2RLWRLD
TL;DR HOPE predicts per-vertex contact and pressure on the MANO hand mesh from monocular RGB video — supervised mostly on gloved hands, yet transferring to bare-hand egocentric and in-the-wild video through heterogeneous force–contact supervision.
Estimating physical pressure from vision is essential for understanding contact-rich hand-object interaction. However, prior vision-based pressure estimation methods are largely limited to planar surfaces and single image input, making them difficult to apply to dynamic hand-object interaction with diverse objects. We instead formulate pressure estimation as a hand-centric video prediction problem with monocular video as input. This formulation predicts temporally evolving per-vertex normal pressure and contact directly on the hand mesh, yielding a unified output space independent of object shape and sensor layout.
Building on this formulation, we propose HOPE, a framework with two key components. First, we lift tactile-glove pressure, planar-sensor pressure, and distance-based hand-object contact annotations into a shared hand vertex space, allowing bare-hand contact data to regularize pressure learning where metric labels are unavailable. Second, we introduce a vertex-anchored video transformer that treats each vertex as a persistent token, aggregates visual features and hand pose over time, and uses a contact-gated pressure head to enforce that pressure vanishes without contact.
Experiments on OpenTouch, PressureVisionDB, and hand-object contact benchmarks demonstrate that HOPE supports a common hand-centric representation across object-pressure, surface-pressure, and contact-supervised HOI settings. Despite using metric pressure supervision primarily from gloved-hand videos, HOPE generalizes to bare-hand egocentric and in-the-wild videos, producing joint contact and pressure predictions beyond the scope of contact-only or planar-pressure baselines.
Monocular RGB in, per-vertex force and contact out. No tactile glove, no instrumented surface, no known object — and no ground truth to fit to. Arrow length is the aggregate force of each finger region; the same predictions are shown as a per-vertex pressure field and a contact mask.
Swipe the panel sideways to compare all four views.
Clips are held-out egocentric and in-the-wild videos; hand pose comes from an off-the-shelf reconstructor.
Each hand is predicted separately, the left one by mirroring the video through the same right-hand model.
One timeline, three views of the same output: the input frame, the hand mesh carrying the per-vertex field, and the force aggregated over hand regions. Drag the hand to orbit it, drag the plot to scrub.
Loading force data…
Aggregated forcesum of predicted vertex-level force
Pressure and contact are predicted as per-vertex fields on the MANO mesh. This shared output space is what allows heterogeneous supervision — two sensor-specific pressure sources and contact-only HOI data — to train a single model.
A single annotation space, shared across sensors and unifying pressure with contact. Tactile-glove taxels and planar-sensor readings are lifted onto MANO vertices on a common scale; thresholding that pressure yields contact, which joins distance-based contact from hand-object data.
VertexFormer learns the mapping between hand geometry and image evidence. Vertex self-attention relates vertices to one another, while vertex-to-patch cross-attention reads the full patch grid. Finally, temporal attention learns how each vertex evolves over time — pre-load, load, release.
The pressure head is gated by the contact head, p̂ = ĉ ⊙ p̃, building the physical prior that pressure vanishes without contact into the architecture rather than leaving it to be learned. This is what makes contact-only supervision pay off for pressure: those labels reach vertices no force sensor ever covered.
Evaluated across three settings that no prior method covers jointly: hand-object pressure with a wearable tactile glove, hand-surface pressure on an instrumented plane, and bare-hand contact.
Hand-object pressure
Our main target setting. The gap over baselines is largest at the vertex level: PressureVision predicts a 2D planar map and cannot transfer to mesh vertices, while HACO reaches high recall but over-predicts contact regions on gloved hands.
Generalization
Pressure supervision comes almost entirely from gloved hands (OT: OpenTouch). Adding contact-only hand-object data (HOI: DexYCB and ARCTIC) costs essentially nothing in-domain but takes bare-hand vertex-level contact on ARCTIC from 0.069 to 0.498 — a 7× improvement, and the reason HOPE transfers to the in-the-wild clips above.
The same vertex-space model is competitive on frame-level contact and achieves the lowest vertex-level force errors, despite predicting in a hand-centric space rather than the sensor's native 2D image space. PressureVision keeps an advantage on pixel-level IoU because projecting mesh vertices back to the pad introduces hand-reconstruction error that planar methods avoid.
@inproceedings{jeon2026hope,
title = {HOPE: Hand-Object Pressure Estimation from Monocular Videos},
author = {Jeon, Subin and Kim, Byungjun and Joo, Hanbyul},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
year = {2026}
}