Today’s Thoughts
Today’s Thoughts
I saw on X that Byte announced a new product related to AI, which allows a single image to be transformed into speaking, singing, and other actions and expressions through audio or video input. After seeing the examples on their official website, I was very impressed, so let me show you their new product.
Link: https://omnihuman-lab.github.io/
This is their official introduction to OmniHuman:
Translated, it says:
We propose an end-to-end multimodal conditional human video generation framework named OmniHuman, which can generate real human videos based on a single human image and motion signals (such as audio only, video only, or a combination of audio and video). In OmniHuman, we introduce a multimodal motion condition mixing training strategy that allows the model to benefit from mixed-condition data expansion. This addresses the issue faced by previous end-to-end methods due to the scarcity of high-quality data. OmniHuman significantly outperforms existing methods, capable of generating extremely realistic human videos based on weak signal inputs (especially audio). It supports image inputs of any aspect ratio, whether portrait, half-body, or full-body, and can produce more lifelike and high-quality results in various scenarios.
However, they have not yet released any services or model downloads for this product; they have only published a few examples and a rough flowchart of the model training process, without providing other details for now.
According to the official introduction, OmniHuman supports various visual and audio styles and can generate realistic videos in any aspect ratio and body proportion (portrait, half-body, full-body) while showing you their examples.
These videos give me the impression that: the visuals are very stable! Moreover, the expressions and actions of these animated characters are very natural. The official website also mentions that to generate these videos, all you need is any image and audio. Finally, the background of the generated video remains still, unlike other image-to-video generation methods where many areas are unstable.
Of course, this is just the basics; the official page also states that this product can generate rich gestures:
The videos are here:
Not only real people, but also cartoon animations are depicted very naturally:
After watching their animations, I feel that in the future, some animated films could also be made using AI, so we don’t have to keep making