Real-to-Sim-to-Real
at the Last Millimeter

Tianchong Jiang^1,*Ruize Xu^2,*Yicheng Li²Xiangshan Tan¹Matthew Walter¹

¹ Toyota Technological Institute at Chicago ² University of Chicago

* Equal contribution

Figure 1. Contact, adjustment, and entry. The drawer makes limited forward progress while the commanded pose remains ahead. The arm then changes alignment and advances into the opening. The close-up pairs recorded joint targets and encoder readings with reconstructed forward hand displacement and wrist orientation. This qualitative sequence illustrates the contact-and-adjustment behavior; it is not a causal ablation of compliance. Shown at 1× speed.

Task and Setup

Close-up photograph of the drawer and cabinet gaps — Figure 2. Measured insertion clearances. The drawer has a 1.5 mm top gap and a 2 mm side gap. The 1.5 mm charger blade fits into a 2.2 mm slot, leaving 0.7 mm clearance.

Close-up photograph of the charger blade and outlet slots — Figure 2. Measured insertion clearances. The drawer has a 1.5 mm top gap and a 2 mm side gap. The 1.5 mm charger blade fits into a 2.2 mm slot, leaving 0.7 mm clearance.

A 6-DoF Seeed reBot B601-DM picks up a drawer and inserts it into a cabinet, or transfers a phone charger between adjacent outlets. The arm usually repeats a motion within 1 mm, but friction stops its joints up to 7° short of their targets. Our best rigid model still misplaces the wrist camera by a median 10 mm. Even within simulation, replaying the same action sequence on GPU instead of CPU, or with a smaller timestep, shifts the final drawer position by 1–2 cm. The policy must tolerate errors larger than the insertion clearances.

Drawer-position differences grow beyond insertion clearance when the same actions are replayed at different timesteps or on GPU versus CPU — Figure 3. (a) Drawer-position differences between simulator replays of the same action sequence at different timesteps or on GPU versus CPU; lines and bands show the median and interquartile range over 64 paired starts. (b) Measured wrist offsets across repeated motions, and wrist-camera model error. Model markers show medians and bars extend to maxima; “Model + play” includes joint play under gravity. Red bands mark insertion clearances.

Median wrist offset is 0.64 mm across repeats, 9.8 mm for the rigid model, and 2.5 mm when joint play is modeled — Figure 3. (a) Drawer-position differences between simulator replays of the same action sequence at different timesteps or on GPU versus CPU; lines and bands show the median and interquartile range over 64 paired starts. (b) Measured wrist offsets across repeated motions, and wrist-camera model error. Model markers show medians and bars extend to maxima; “Model + play” includes joint play under gravity. Red bands mark insertion clearances.

(a) Drawer insertion

(b) Charger insertion

Figure 4. Real-world execution of image-based policies trained in simulation on a roughly $1,000, 6-DoF arm. The drawer task requires picking, aligning, inserting, and pushing the drawer fully closed. The charger task requires unplugging, transferring, inserting, and releasing the charger. Successful examples are shown at 4× speed.

Abstract

We study drawer and charger insertion with 0.7–2 mm clearances on a roughly $1,000 arm through real-to-sim-to-real learning. Coding agents write scripted teachers that read the simulator state, and image-based student policies imitate them. Although our simulator cannot match the arm to within the clearance, the students succeed in 12/20 drawer and 18/20 charger trials on the real arm. We report three lessons. First, scripted teachers overfit the simulator less than reinforcement learning (RL) teachers, which learn behaviors that work only in simulation. Second, without human guidance, coding agents can edit scripted teachers to be much easier for students to learn. Third, without force sensors, the student senses contact from commanded and measured joint positions, so it need not line the part up perfectly before contact.

Training Pipeline

We fit the Isaac Sim drawer model to ten teleoperated demonstrations and build the charger model from one, measuring geometry in its images. Our teachers are Python programs written and revised by coding agents. They read simulator state and label actions for the student.

The student uses a fine-tuned DINOv2 visual encoder with a multilayer perceptron action head. It sees scene and wrist images, plus commanded and measured joint positions, and predicts short sequences of actions, called action chunks. We train it with online DAgger, in which the teacher labels the states the student visits. Training randomizes masses, friction, grip force, lighting, colors, backgrounds, and camera poses.

Real-World Results

Table 1. Successful real trials / total.
Task	Scripted-teacher student	RL-teacher student
Drawer	12/20 (60%)	6/20 (30%)
Charger	18/20 (90%)	0/20 (0%)

Drawer success includes the final push to close. Charger success includes insertion and release.

On the charger, the RL student succeeds as often as the scripted-teacher student in simulation: 85.1% versus 83.9%. On the real arm, their success rates are 0/20 and 18/20, respectively.

Three Lessons

1. Scripted teachers overfit the simulator less than RL teachers

The demonstrations used to fit a simulator do not reveal subtle contact details. Even under randomized physics, RL teachers can exploit errors in these details. One RL teacher grasped the drawer with only the fingertips of the gripper. This grasp was stable in simulation but unstable on the real arm, making insertion nearly impossible.

A scripted teacher does only what its code states, so it has less room to exploit these errors. Correcting the grasp through RL required task-specific reward shaping and retraining; a coding agent can fix a script in minutes. We spent roughly two weeks developing RL teachers and two days developing scripted teachers, with coding agents assisting both.

2. Successful scripts must also be learnable

A coding agent can make scripts easier for image-based students to imitate without human guidance. Our teachers’ privileged inputs are the objects’ true poses and, for the drawer, finger contact—all quantities the student can estimate. Even so, the early drawer teacher succeeded over 90% in simulation, while its student succeeded only 34% of the time.

The teacher’s gripper command flipped at a sharp pose threshold, giving nearly identical views opposite commands. The agent proposed and implemented smoothing these rules, raising student success to 61% while the teacher stayed above 90%.

Early, middle, and final drawer teachers succeed in 96, 94, and 92 percent of simulated trials; their students succeed in 34, 61, and 83 percent — Figure 5. Drawer teachers and their students across successive revisions, evaluated in simulation. Teacher success stays above 90%, while student success rises from 34% to 61% and then 83%.

A single-action student can stall where the teacher pauses: the unchanged state keeps producing the same command. One such student succeeded 90% in simulation but stalled on the real arm, so we adopted action chunking.

3. Use contact to resolve alignment uncertainty

The arm has no joint torque, wrist force/torque, or tactile sensors. Under joint proportional–derivative (PD) control, the difference between commanded joint positions, q_des, and measured joint positions, q, provides an indirect cue to contact resistance. The student receives both signals alongside images.

q_des − q

For both tasks, the student presses the part against the mating surface and slides it until it aligns with the opening and enters. The part therefore need not be aligned to within the insertion clearance before contact.

In a test using a stuck real drawer recording, we hold the images and measured joints fixed and shrink only the commanded–measured gap in the policy’s input. The student changes from stopping to commanding another push, showing that it uses the gap to sense when the drawer is blocked.

How Compliance Helps Insertion

Finite-gain joint PD control lets the arm yield when contact blocks its commanded motion. Maintaining pressure while changing alignment lets the surface guide the object toward the opening. Compliance and corrective motion work together: pressure alone cannot correct a lateral misalignment. Orange denotes the commanded pose; cyan denotes the actual or measured pose.

Figure 6. Contact in cross-section. A planar arm holds a block above a surface with a through-hole. Contact limits descent while a lower target maintains compliant pressure. A sideways correction brings the block over the opening, where descent becomes possible. The small comparison shows why downward pressure at a fixed misalignment remains blocked. Motion is illustrative; there is no back wall or bottom stop.

Figure 7. Simulated robot motion. Joint tracking during a scripted drawer insertion in simulation. Orange uses the joint target applied by the controller; cyan uses the actual joint positions at that simulation step. Both are projected through the simulator’s camera. Shown at 2× speed.

Domain Randomization

The student encounters variation in object placement, physical parameters, camera poses, appearance, backgrounds, and clutter. These 5 × 5 atlases show separate scripted-teacher rollouts across randomized scenes. Display exposure is brightened for visibility; the underlying trajectories and training randomization are unchanged.

Drawer insertion · 25 simulated scenes

Figure 8. Drawer randomization atlas. Twenty-five independent simulated episodes, shown at 4× speed. Each tile holds its last view when that episode ends. These qualitative examples show the range of scene variation used for camera-based learning.

Charger transfer and insertion · 25 simulated scenes

Figure 9. Charger randomization atlas. Twenty-five independent simulated episodes with the charger task’s object, physics, and appearance randomization, shown at 4× speed. Each tile holds its last view when that episode ends.

All videos include playback speed and step counters. Use the video controls to pause, seek, or view full screen.