In the open-ended video game Minecraft, players extract blocks of virtual materials from a 3D environment to assemble objects of their own design, from trees to cathedrals. Researchers trained neural networks to generate these structures.

What’s new: Shyam Sudhakaran and researchers at University of Copenhagen, University of York, and Shanghai University used a neural cellular automaton algorithm to construct 3D objects. The work demonstrates the potential for such algorithms to generate structures in three dimensions, as typically they’re limited to two.

Key insight: A cellular automaton generates complex patterns on a 2D grid by changing each cell’s state iteratively based on simple rules that depend on the states of its neighbors. A neural cellular automaton updates cells depending on the output of a neural network and the states of neighboring cells. Using 3D convolutions enables a neural cellular automaton to generate patterns in 3D.

How it works: The authors trained several 3D convolutional neural networks to reproduce structures found on the community website Planet Minecraft. Each different structure required its own model. The structures comprised 50 block types mostly corresponding to materials (stone, glass, metals, and so on), including piston blocks that push or pull adjacent blocks to produce animated objects. The system spawned block types directly without needing to virtually mine them out of the virtual ground.

The authors initialized a single block in a 3D grid.
The network updated each cell in the grid depending on whether a neighboring cell was activated. The updates ran for a set number of steps, growing the structure at each step.
The loss function encouraged the generated structure to match the original in block type and placement.

Results: The authors reported few quantitative results. However, the trained models grew static structures like castles, temples, and apartments that appear to be accurate inside and out. One model learned to grow an animated caterpillar.

Why it matters: Cellular automata may have certain benefits. For instance, if part of the resulting structure is destroyed, the automaton can use what’s left to regenerate the missing part. This approach can produce resilient digital 3D structures with no human intervention after the first step.

We’re thinking: Machine learning engineers looking for an excuse to play Minecraft need look no further!

University of York

3D Object Factory

Subscribe to The Batch