The Dactyl Experiment: How OpenAI Taught a Robot Hand to Solve a Rubik’s Cube

Image source: OpenAI, “Solving Rubik’s Cube with a robot hand”. Photo by Eric Haines. Source page: https://openai.com/index/solving-rubiks-cube/

What was the experiment?

In 2019, OpenAI published an experiment in which a five-fingered robot hand called Dactyl held and solved a Rubik’s Cube in its palm.[1]

The difficult part was not speed. Dactyl was much slower than a person. The challenge was keeping the cube from falling, changing grips, rotating the correct face and repeating those actions through a long sequence.

The researchers did not program every finger movement as a fixed script. They trained a neural network in simulation and transferred it to the physical hand.

Stage 1: Splitting the problem into two parts

OpenAI began pursuing the Rubik’s Cube goal in May 2017.[1]

The hardware was a Shadow Dexterous Hand, a five-fingered robot hand with many joints. The hand already existed; most of the new work concerned the learning-based control system.[1][2]

The team divided the problem into two tasks:

1. Find a sequence that solves the cube.
2. Control the fingers to execute the sequence.

The neural network did not invent the puzzle solution. OpenAI used the existing Kociemba algorithm to generate a solution sequence. The AI handled the physical work: holding the cube, flipping it, bringing the required face to the top and rotating it by 90 degrees.[1][2]

Stage 2: Learning inside a simulator

Training directly on a physical robot would have been slow and damaging to the hardware. OpenAI therefore created a simulated hand, cube and environment.[2]

Inside the simulator, the control network repeatedly attempted actions such as:

• bending and extending individual fingers;
• shifting pressure between fingertips;
• rolling the cube in the palm;
• flipping it to expose a different top face;
• rotating the top face by 90 degrees;
• stabilizing the cube after a turn.

The system received rewards for reaching the requested target. Dropping the cube or failing to reach the target counted as failure.

Stage 3: The simulator did not perfectly match reality

A policy that worked in simulation did not automatically work on the real hand.

Fingertip friction, cube mass, motor delay, joint looseness, lighting, camera noise and the stiffness of each cube face were difficult to model precisely.

A small mismatch could make a finger slip, under-rotate a face or drop the cube.

The team first used domain randomization, manually varying simulation parameters so the policy would not memorize one perfect world. But manual randomization had two problems:

• too little variation did not prepare the robot for reality;
• too much variation too early made the task too difficult to learn.[1]

Stage 4: Automatic Domain Randomization

OpenAI developed Automatic Domain Randomization, or ADR, to address that problem.[1][2]

ADR began with a relatively simple environment. When the network reached a performance threshold, the system automatically increased the difficulty.

The changing parameters included:

• cube size;
• cube mass;
• fingertip friction;
• surface colors and materials;
• joint behavior;
• system latency.

Whenever the network adapted, the randomization range expanded again.

Image source: OpenAI. The collage illustrates variations in color, lighting and surface appearance used during domain randomization. Source page: https://openai.com/index/solving-rubiks-cube/

The policy could not rely on one exact version of the physics. It had to keep adapting to different versions of the same task.[1]

Stage 5: Training from simple cubes to a full Rubik’s Cube

The researchers did not begin with a standard Rubik’s Cube. They created several prototypes to isolate individual skills:[1][2]

1. Locked cube: no face could rotate; the hand practiced holding and reorienting it.
2. Face cube: only limited face motion was available.
3. Full cube: all six faces could rotate, with external state tracking.
4. Giiker cube: an instrumented cube with internal sensors.
5. Regular Rubik’s Cube: a nearly standard cube tracked mainly through vision.

Image source: OpenAI. Left to right: locked cube, face cube, full cube, Giiker cube and regular Rubik’s Cube. Source page: https://openai.com/index/solving-rubiks-cube/

This staged process helped the researchers identify whether a failure came from holding, flipping, face rotation or state estimation.

Stage 6: Building the real-world observation system

The physical setup included:

• a Shadow Dexterous Hand;
• three RGB cameras;
• motion-capture equipment tracking the fingertips;
• a vision model estimating cube position and orientation;
• the control network selecting the next finger movement.[2]

Some prototypes reported face angles through internal sensors. In the more standard version, cameras had to estimate the cube state.

OpenAI also modified the hardware for reliability. Sensor cables were moved inside the fingers, rubber was added to the fingertips and the robot enclosure was redesigned to keep camera and motion-capture calibration more stable.[2]

What happened during one solving attempt?

A complete attempt followed this sequence:

1. The Rubik’s Cube was scrambled.
2. The Kociemba solver calculated a solution sequence.
3. The sequence was divided into small physical targets.
4. If the needed face was not on top, the robot flipped the cube in its palm.
5. The fingers changed position to secure the grip.
6. One or more fingers rotated the top face by 90 degrees.
7. Cameras and sensors checked the new state.
8. The next target was issued.
9. The cycle continued until the cube was solved, dropped or the time limit was reached.[1][2]

The robot did not replay a fixed animation. It observed the cube after each action and adjusted its grip.

Failures observed during the experiment

1. Dropping the cube

This was the clearest failure mode. A solve required many flips and rotations. One slipping finger or a weak grip could make the cube fall. Under the evaluation rules, a dropped cube counted as a failed attempt.[1]

OpenAI found that failures were more likely during the first few flips and rotations, when the recurrent network was both attempting the task and adapting to the real physical system.[1]

2. Timing out

An attempt also failed if the robot took too long.[1]

Dactyl often paused to change grips and reposition fingers. A small mistake might not immediately drop the cube, but it could make the sequence exceed the time limit.

3. Placing a finger incorrectly

If a fingertip contacted the wrong edge or face, the hand might not produce enough turning force. A face could rotate only partially, overshoot 90 degrees or push the whole cube out of the palm.

4. Misreading the cube’s pose

Fingers sometimes blocked the cameras. Lighting and viewing angle also changed. If the vision network estimated the position or orientation incorrectly, the controller could send a finger to the wrong place.[2]

5. Reality still differed from simulation

ADR improved sim-to-real transfer but did not remove every mismatch. Friction changed over time, hardware wore down, cubes differed in stiffness and cameras produced noise.

Disturbance tests

The researchers deliberately disturbed the robot while it was working.

OpenAI showed Dactyl continuing after being prodded with a stuffed giraffe. The team also tested other changes affecting motion and observation.[1][2]

In many cases, the hand still completed flips and rotations, but more slowly. After a change, the time needed for an action often increased and then decreased as the recurrent network adapted.[1]

Final results

OpenAI reported two main success levels:[1]

• On scrambles requiring about 15 face rotations, Dactyl solved the cube about 60% of the time.
• On a maximally difficult test requiring 26 face rotations, success fell to about 20%.

In simple terms:

• easier test: roughly 6 successful solves out of 10;
• hardest test: roughly 2 successful solves out of 10.

The remaining attempts generally ended because the cube was dropped or the time limit was reached.[1]

If a person placed a dropped cube back into the hand, the policy could continue from the new state, although the original attempt remained a failure under the evaluation rules.[1]

The whole experiment in order

1. OpenAI began the Rubik’s Cube goal in May 2017.
2. The task was solved in simulation by July 2017.
3. By July 2018, the real hand could mainly reorient a solid block.
4. The team continued improving the simulator, vision network and ADR.
5. The robot trained on locked, face, full, sensor-equipped and nearly regular cubes.
6. Kociemba supplied the solution; the neural policy controlled the hand.
7. The policy learned flips, top-face rotations and grip changes.
8. Training varied size, friction, mass, latency and visual appearance.
9. The learned skill was transferred to the real hand.
10. The team added physical disturbances to test adaptation.
11. Dactyl achieved 60% success on 15-rotation scrambles and 20% on the difficult 26-rotation test.[1][2]

SEO

Meta title: The Dactyl Experiment: How OpenAI Taught a Robot Hand to Solve Rubik’s Cube

Meta description: A detailed account of OpenAI’s Dactyl experiment: simulation training, Automatic Domain Randomization, real-world failures and final success rates.

Primary keywords: Dactyl Rubik’s Cube, OpenAI robot hand, robot hand solving Rubik’s Cube, Automatic Domain Randomization, sim-to-real, reinforcement learning

Suggested slug: dactyl-experiment-robot-hand-solves-rubiks-cube

GEO summary

Dactyl was an OpenAI robotics experiment published in 2019. A Shadow Dexterous Hand was controlled by neural networks trained entirely in simulation. Kociemba’s algorithm supplied the Rubik’s Cube solution sequence, while the AI learned to flip the cube, change grips and rotate individual faces. Automatic Domain Randomization progressively varied cube size, mass, friction, latency and visual appearance to improve transfer to the physical robot. Common failures included dropping the cube, incorrect finger placement, vision errors and timeouts. Dactyl solved about 60% of scrambles requiring 15 face rotations and about 20% of difficult scrambles requiring 26 rotations.

Sources

[1] OpenAI — “Solving Rubik’s Cube with a robot hand”, October 15, 2019
https://openai.com/index/solving-rubiks-cube/

[2] Ilge Akkaya et al. — “Solving Rubik’s Cube with a Robot Hand”, arXiv:1910.07113
https://arxiv.org/abs/1910.07113

[3] OpenAI et al. — “Learning Dexterous In-Hand Manipulation”, arXiv:1808.00177
https://arxiv.org/abs/1808.00177