Automated Lip Sync Animation for Any 3D Model

It is possible to set up an automated lip sync animation in Blender 2.8 for any character or a random object you find on a file repository website. Want to see a talking pipe? Yes, there is an automatically lip synced talking pipe animation on YouTube. Nearly every file on STL sharing websites can be animated so that it speaks automatically in sync with your speech.

Animation Tool in Blender 2.8

Blender3D is a sandbox. The more I use it, the more functionalities I discover. My most recent finds are animation rigs and armatures, tools that allow to create movement for any object you want, in a smooth and natural fashion. Additionally, you can change the colors of the objects in various transitions. There may be tools out there better suited for particular tasks, but I find it very convenient to have it all, designing, sculpting, etc., in a software that I am familiar with. And Blender 3D completely nailed it.

Lip Syncing

Lip syncing seems to be one of the most difficult tasks in character/object animation. There are dozens of different mouth positions, each covering a particular sound: E, A, O, W, etc. For a simple 2D animation, four mouth positions may be sufficient. However, for a 3D model face, for example, we would recommend using all positions described in the YouTube video by YOGYOG.

Pipe and Poses by YOGYOG — Pipe and Poses
(Image: YOGYOG)

Automatic Lip Syncing – Rhubarb Plugin

It is also possible to lip sync manually by simply adding your model’s mouth positions to “Object Mode – Object Data – Shape Keys”. Then start saving keyframes for every sound effect to create your desired speech (viewing a YouTube tutorial is highly recommended). This is a precise but very time consuming process. I would only use this method for a short speech animation, like greetings, etc. Alternatively, you can download the automatic lip-syncing plugin “Rhubarb”. It is hard to set it up properly, but when it is done, your mesh will speak any speech script you upload. And you can come back to this Blender animation setup time and time again.

Preparing the Mesh

When I uploaded an existing 3DPG face to Blender, I struggled to get the mouth positions to function. My mesh broke frequently. Only after combing through YouTube tutorials I understood that it is very important to correctly set up the mesh.

I went through a simple yet lengthy process and set up the lower face mesh how it should be. You can refer to the image below. In this case, the less vertexes and edges, the merrier.

Lip Sync - Before After - Mesh Editing in Blender 2.8 — Lip Sync – Before After – Mesh Editing in Blender 2.8
(Image: 3DWithUs)

Sound for The Automated Lip Sync Animation

Audacity is an old, trusty (and free) software that will help you to prepare your own voice. And if necessary, modify it further.
One place in particular where I got stuck when working with Rhubarb lip-syncing was the format. To work properly in Blender 2.8, the sound format should be “16 Bit – PCM” (and exported as WAV).

Audacity - 16 bit PCM WAV - The Best Rhubarb Lipsyncing Format — 16 bit PCM WAV – The Best Rhubarb Lipsyncing Format
(ScreenSHot: Audacity)

Amazon Polly – Text to Speech Service

The 3D Printing Guardian’s voice was achieved thanks to Amazon Polly (AWS) text-to-voice application. “25-45 years old Joey from the USA” helped us in his English and American accents. From now on 3DPG will present videos with voice and fun animation. Stay tuned. Please don’t forget to like and subscribe.

Off-Topic: Learning New 3D Designing Skills – in an Upside-down Way

Students used to take a few design courses in order to be able to create designs, and in most cases, their sketches and works were never realized. Today, 3D printer owners first print someone else’s object, then 3D design something simple, and later more advanced. In my case, I found myself animating my own 3D printing model that I created some time ago. I’m curious to see what comes next! See you soon and happy 3D designing and 3D printing.