R-RNN Doodle

Doodling with Robot & RNN


R-RNN Doodle

Doodling with Robots and RNN

Robotic-RNN Doodle is an exploration about the possibilities of integrating the RNN-Sketch Demo1 with a robotic arm to build a collaborative workflow between the user, RNN, and the robot. 

Sketch-RNN is trained on a data set of millions of doodles collected through Quick Draw project2. It leverages LSTM architecture to generate svg doodles based on a given svg stroke as the initial seed. Sketch-RNN comes with hundred of trained models, ranging from trivial doodles of animals and object to complex combination of them. For this assignment I used the Flamingo model to generate flamingos.


The motivation behind this project is to find a robot-human interaction scenario to open the discussion for further development in Robot-Art competition. UR robots are categorized as collaborative robots, which are safer for human users to engage in a collaborative scenario. This

project aims to develop a pipeline to connect user input, machine learning back-end, and a robotic control/simulation workflow.



The model consists of two main component, one is the grasshopper part that handles image processing process, toolpath generation, simulation, and communication with the robot. The other component is Sketch_RNN code that generates the doodle based on the user drawn seed.

R-RNN doodle diagram.png


1. User draws a simple stroke on a piece of paper;

2. Robot moves over the image and take a snapshot;

3. Image will be processed to find the boundary of the canvas and correct the perspective;

4. The stroke will be extracted as a openCV contour and will be passed to the Sketch_RNN;


5. Sketch-RNN will generate doodles and pass them back to the Grasshopper definition;


6. Using HAL plug-in for Grasshopper, the toolpath for the robot will be generated.



The codes for this project actually includes functions to control a GoPro camera in real time.

However, for the purpose of this demo, I used images already captured and stored on the disk.

The process of matching the drawn object and robot’s drawing coordination is still a challenge.


This project is developed as a part of Art and Machine Learning course at Carnegie Mellon University, School of Computer Science, under the supervision of Prof. Eunsu Kang and Prof. Barnabas Poczos.