Let me introduce an innovative technology developed by Tencent Games’ ZhiJi team – AniPortrait.
This technology can generate high-quality animated portraits from audio and reference portrait images. Whether it’s static images or videos, AniPortrait can achieve realistic facial reproduction, making the animation effects more vivid. ShanZha was particularly impressed with the facial reproduction feature, which can accurately capture the expression changes from the reference video and transfer them to a new portrait, making it very suitable for creating virtual hosts or digital characters.
Without further ado, let’s take a look at the effects:
System Requirements and Environment Setup
To use AniPortrait, you first need to configure a compliant computing environment. It is recommended to use Python version 3.10 and CUDA 11.7. The specific steps are as follows:
1. Install Required Python Libraries
Run the following command to install the necessary Python libraries:
pip install -r requirements.txt
2. Download Pre-trained Weight Files
You need to download the pre-trained weight files and place them in the ./pretrained_weights directory. These weights include multiple components such as the denoising network, reference image network, pose guider, action module, audio-to-mesh model, audio-to-pose model, and FILM network. The specific weight files and download paths are as follows:
-
StableDiffusion V1.5
-
sd-vae-ft-mse
-
image_encoder
-
After downloading and organizing the weight files, you can specify the paths of the weight files in the configuration file (e.g., ./config/prompts/animation.yaml).
Model Usage and Command Line Tools
AniPortrait provides various command line tools to support different animation generation needs.
1. Self-Driven Animation
Specify the animation configuration file and use the -acc parameter to accelerate video generation. The command is as follows:
python -m scripts.pose2vid --config ./configs/prompts/animation.yaml -W 512 -H 512 -acc
2. Facial Reproduction
Add source facial video and reference image, and generate facial animation using the following command:
python -m scripts.vid2vid --config ./configs/prompts/animation_facereenac.yaml -W 512 -H 512 -acc
3. Audio-Driven Animation
After adding audio and reference images in the configuration file, run the following command to generate audio-driven video animation:
python -m scripts.audio2vid --config ./configs/prompts/animation_audio.yaml -W 512 -H 512 -acc
Online Demo and Gradio Interface
Users can also experience the web demo of AniPortrait through the Gradio interface on HuggingFace Spaces. Run the following command to start the local demo:
python -m scripts.app
With AniPortrait, users can easily generate high-quality animations, enhancing not only the visual experience but also significantly improving interactivity and realism.
ShanZha’s Experience
During the use of AniPortrait, ShanZha was particularly satisfied with the facial reproduction feature. This feature can not only accurately capture the expression changes from the reference video but also transfer these expressions well to the new portrait. This is very valuable in the creation of virtual hosts or digital characters.
Trying to use AniPortrait to generate expression animations for a virtual host, the effect was quite realistic, almost indistinguishable from a real person.
Moreover, the audio-driven animation feature also amazed ShanZha. Just by providing audio and reference images, AniPortrait can generate realistic animations synchronized with the audio, making it very convenient to use.If you are also interested in this technology, hurry up and download to experience it!
Download Link: https://www.songshuhezi.com/index_open/aniportrait.html
I am an AI entrepreneur proficient in various AI tools. Through AI tools, I achieved a fan growth of over 200,000 in just three months. I represented my team in several innovative entrepreneurship competitions, where we won second place with our outstanding AI project in the innovation and entrepreneurship competition held jointly in Chengdu and Chongqing.
-
AI Tool Review – BrushNet: Local Repainting Tool
-
Surname Avatars: Zhang, Wang, Chen, Zhao, Huang, Yang, Peng, Zhu, Cheng, Zheng, He, Xiao, Pan, Tang, Du, Wu, Qian, Qin, Lin, Wei, Chang, Yue, Jin, Jiang, Tong, Lei, Jia, …
-
AI Tool Review – so-vits-svc: Achieving AI Song Cover Freedom
