Automated Lip Sync Animation for Any 3D Model – Blender Rhubarb Plugin


It is possible to set up an automated lip sync animation in Blender 2.8 for any character or a random object you find on a file repository website. Want to see a talking pipe? Yes, there is an automatically lip synced talking pipe animation on YouTube. Nearly every file on STL sharing websites can be animated so that it speaks automatically in sync with your speech. Click on picture to zoom

Lip Sync - Intimidating Animation Tab Screenshot - Blender 2.8

Lip Sync – Animation Tab in Blender 2.8

Animation Tool in Blender 2.8

Blender3D is a sandbox. The more I use it, the more functionalities I discover. My most recent finds are animation rigs and armatures, tools that allow to create movement for any object you want, in a smooth and natural fashion. Additionally, you can change the colors of the objects in various transitions. There may be tools out there better suited for particular tasks, but I find it very convenient to have it all, designing, sculpting, etc., in a software that I am familiar with. And Blender 3D completely nailed it.

Automatic Lip Syncing - 3DPG Mouth Positions - Shapekeys

Automatic Lip Syncing – 3DPG Mouth Positions – Shapekeys

Lip Syncing

Lip syncing seems to be one of the most difficult tasks in character/object animation. There are dozens of different mouth positions, each covering a particular sound: E, A, O, W, etc. For a simple 2D animation, four mouth positions may be sufficient. However, for a 3D model face, for example, we would recommend using all positions described in the YouTube video by YOGYOG.

Pipe and Poses by YOGYOG

Pipe and Poses by YOGYOG

Automatic Lip Syncing – Rhubarb Plugin

It is also possible to lip sync manually by simply adding your model’s mouth positions to “Object Mode – Object Data – Shape Keys”. Then start saving keyframes for every sound effect to create your desired speech (viewing a YouTube tutorial is highly recommended). This is a precise but very time consuming process. I would only use this method for a short speech animation, like greetings, etc. Alternatively, you can download the automatic lip-syncing plugin “Rhubarb”. It is hard to set it up properly, but when it is done, your mesh will speak any speech script you upload. And you can come back to this Blender animation setup time and time again.



Preparing the Mesh

When I uploaded an existing 3DPG face to Blender, I struggled to get the mouth positions to function. My mesh broke frequently. Only after combing through YouTube tutorials I understood that it is very important to correctly set up the mesh. I went through a simple yet lengthy process and set up the lower face mesh how it should be. You can refer to the image below. In this case, the less vertexes and edges, the merrier.

Lip Sync - Before After - Mesh Editing in Blender 2.8

Lip Sync – Before After – Mesh Editing in Blender 2.8

Sound for The Automated Lip Sync Animation

Audacity is an old, trusty (and free) software that will help you to prepare your own voice. And if necessary, modify it further.
One place in particular where I got stuck when working with Rhubarb lip-syncing was the format. To work properly in Blender 2.8, the sound format should be “16 Bit – PCM” (and exported as WAV).

Audacity - 16 bit PCM WAV - The Best Rhubarb Lipsyncing Format

Audacity – 16 bit PCM WAV – The Best Rhubarb Lipsyncing Format

Amazon Polly – Text to Speech Service

The 3D Printing Guardian’s voice was achieved thanks to Amazon Polly (AWS) text-to-voice application. “25-45 years old Joey from the USA” helped us in his English and American accents. From now on 3DPG will present videos with voice and fun animation. Stay tuned. Please don’t forget to like and subscribe.

Off-Topic: Learning New 3D Designing Skills – in an Upside-down Way

Students used to take a few design courses in order to be able to create designs, and in most cases, their sketches and works were never realized. Today, 3D printer owners first print someone else’s object, then 3D design something simple, and later more advanced. In my case, I found myself animating my own 3D printing model that I created some time ago. I’m curious to see what comes next!
See you soon and happy 3D designing and 3D printing.

Blender Character Rigging Course on Udemy


Read More: 3D Printing Software

There are several types of software that can be applied in 3D printing. Design software creates 3D printable objects. Scanning software and applications capture objects that can be further adjusted for 3D printing. Repair software fix mesh and make objects 3D printable. Slicing software prepares G-codes with the settings for the printer… Read More
3D Printing Software


Files & Links

Blender 2.7 plus Rhubarb Lipsync – Shapekeys by YOGYOG.
Rhubarb Plugin/Addon
Download Blender Software – Here
Blender Training Course by PTT (7 day Free Trial) – Here


Sign up for the new competitions, giveaways & new posts alert





About Max Funkner

Maker, Designer and Blogger on the subjects of 3D design and 3D printing. You can find my creations on all major file repositories. I also share my experience on 3dwithus.com. Find me on Twitter & Instagram
avatar
3000