Gmoji is an experimental openFrameworks application that translates your facial expressions into a giant emoji. It was created over a three week sprint and debuted at Google I/O 2016. We thought it might be fun for an attendee to turn their face into an emoji. So, we set out to figure out how to make that happen. This is the result.

The application was developed in two parts: the Android openFrameworks app that handles the face detection (written in C++), and the Web app to render the emoji (written primarily in JavaScript).

The app is running several algorithms to detect facial landmarks, and run sentimental analysis on these. The openFrameworks app delivers these landmark points and sentimental analysis results via JavaScript injection into a local WebView and/or over WebSockets to a remote browser, ready for HTML5 canvas animation.

Within a local WebView or remote browser a Paper.js app creates the persistent emoji vector objects for animation upon initialization. A lightweight JavaScript animation loop parses the incoming JSON facial landmarks and models the corresponding X/Y coordinates, viewable in debug mode. The landmarks and confidence scores are then translated into preset emoji models (such as kissy face, cheesy smile, etc.), augmented by live expressions like mouth shape or lip geometry.