A / Precise tool use · Sharpa
Power-Drill Actuation
Align the drill and press the trigger to tighten the target screw.
Real-world robot learning
HiFun: A Hierarchical Framework for Efficient
Functional Dexterous Manipulation Learning
Accepted to CoRL 2026
1 HKUST (Guangzhou)2 HKUST3 ETH Zurich
Overview
The full picture
Abstract
While multi-fingered dexterous robot hands have the potential to provide human-level dexterity, using them for functional manipulation remains difficult because task success depends on precise contact, coordinated arm-hand motion, and reactive recovery from execution errors. Precise contact depends on complex contact dynamics that are difficult to model accurately in simulation, motivating real-world learning. However, real-world demonstrations in the full arm-hand control space are costly and noisy because each trajectory must coordinate arm motion with high-DoF multi-finger contact. Small execution deviations can further cause slipped contact or premature force application, making recovery from off-nominal contact states important for robust execution.
To deal with these challenges, we propose HiFun, a hierarchical real-world RL framework for functional dexterous manipulation tasks. Our key insight is to decouple the control complexity into a functional hand skill for precise contact execution and an arm-hand coordination policy for arm control and hand-skill activation. This design enables effective real-time human intervention for reactive recovery learning and efficient data collection with Dynamic Movement Primitive (DMP) augmentation. Across six real-world tasks on two dexterous hands, HiFun achieves 98.3% average success within one hour of online training and remains robust to dynamic disturbances.
01 / Task challenges
Challenges in functional dexterity
Functional dexterous tasks demand precise contact, coordinated arm–hand motion, and recovery from small execution errors. Constrained tool retrieval brings these requirements together in a single task.
- 01
Precise contact
Place fingertips on the exposed handle and regulate force as contact changes.
- 02
Arm–hand coordination
Coordinate arm motion and multi-finger levering to create clearance and form a precision grasp.
- 03
Reactive recovery
Small slips or misalignment can interrupt the sequence, requiring contact or alignment to be restored before continuing.
02 / Method
Learn contact skills.
Coordinate their use.
HiFun separates learning how the fingers make contact from learning where and when to use that contact. First, residual RL refines a kinesthetic reference into a contact-aware hand skill. With this skill frozen, a coordination policy learns arm motion and skill activation, starting from DMP rollouts and improving through value-guided human-in-the-loop learning. At execution time, the two policies work together to approach the target and perform fine-grained contact.
03 / Real-world evaluation
Functional dexterous tasks.
Each task is evaluated over 50 trials with randomized object poses inside the trained workspace. Success requires completing the functional outcome.
B / Constrained space · Sharpa
Angle-Spreader Retrieval
Maintain fingertip contact to extract the tool from a tight tray.
C / Constrained space · Sharpa
Thin-Handle Retrieval
Lever an exposed handle into reach, form a precision grasp and fully extract it.
D / Contact-rich rotation · Sharpa
Power-Drill Bit Removal
Maintain multi-finger contact while loosening the chuck, then remove the drill bit.
Across hand embodiments
One framework, different dexterous hands
HiFun deploys on Sharpa and XHand. XHand tasks include pipette dispensing, constrained tool retrieval, drill operation, and faucet control.
E / Precise tool use · XHand
Pipette Dispensing
Three dispensing executions, side by side. The hand aligns the pipette with each vial and adapts as the vial position changes.
F / Constrained space · XHand
Thin-Shaft Tool Retrieval
1.5×
Use a fingertip contact sequence to fully retrieve a thin shaft embedded in foam.
Additional task · XHand
Power-Drill Operation
Coordinate tool alignment and trigger actuation while handling changes in the target position.
Additional task · XHand
Faucet Operation
Maintain finger contact while turning the faucet handle.
Across tasks and hands
Further deployments
Beyond nominal rollouts
Recovery & Generalization
HiFun adapts to position disturbances, object-size changes, and disruptions in fingertip contact.
2×
1×
Long-horizon manipulation
Multiple skills, one policy
Here, HiFun runs on XHand in a longer functional sequence. The policy turns the key, grasps the drill, and places it into the drawer by coordinating different hand skills. This shows that HiFun can deploy across dexterous hands and integrate multiple skills in one policy.
1.3×
04 / Learning challenges and design
Challenges & Insights
Learning these behaviors directly in the full arm–hand action space makes exploration and human correction difficult. HiFun structures the control interface and learning signals to make real-world learning more efficient.
01 / Challenge
High-dimensional exploration
Direct arm–hand RL must discover arm motion, coordinated finger contacts, and activation timing in a coupled action space. Successful contact sequences occupy only a small part of that space.
HiFun insight
Explore through contact-aware skills.
Learn finger execution separately, then let the coordination policy explore 6D arm motion and hand-skill activation.
02 / Challenge
Difficult finger-level intervention
Real-time correction of many coupled finger joints is difficult to deliver consistently, especially when human and robot hand morphologies differ.
HiFun insight
Make the skill the intervention interface.
Operators correct arm placement and skill activation through the learned hand skill, reducing the need to teleoperate individual finger joints.
Full-DoF HIL-SERL
7 sHiFun
7 s03 / Challenge
Ambiguous credit assignment
The same unsuccessful outcome can result from poor arm alignment or mistimed hand-skill activation. A terminal success signal alone does not distinguish these causes.
HiFun insight
Separate alignment and activation value learning.
Use distinct navigation and modulation targets, with action-subspace gradient routing, to guide arm motion and skill timing.
Method overview
Experimental evidence
04 / Challenge
Heterogeneous human supervision
Human corrections can be noisy, delayed, or hesitant. Their usefulness also depends on what the current policy would do in the same state.
HiFun insight
Imitate according to estimated advantage.
IAW weights each correction relative to the current policy action, giving stronger imitation weight to corrections with higher critic-estimated advantage.
Method overview
A critic trained on real-world outcomes predicts the return of each candidate action. Comparing the intervention with the current policy action in the same state estimates how much the correction improves on the robot’s own behavior. IAW gives higher-advantage corrections more imitation weight and limits the influence of low-advantage corrections, focusing learning on guidance expected to improve execution.
Learning contact skills and when to use them.
295/300
successful evaluation trials
6
functional tasks
2
dexterous hands
≤60 min
online HIL per task
98.3% mean success over 50 trials per task. The ≤60-minute budget covers online HIL; the full pipeline averages approximately 109 min/task.
Citation
@inproceedings{huang2026hifun,
title = {HiFun: a Hierarchical Framework for Efficient
Functional Dexterous Manipulation Learning},
author = {Huang, Linyi and Huai, Guowei and Liu, Weibin and
Jiang, Shulong and Tan, Ping and Zhang, Weixuan and
Zhang, Hui and Song, Jie},
booktitle = {Conference on Robot Learning (CoRL)},
year = {2026}
}