(You need to download ComfyUI first, I recommend looking at the previous SVD article)
What! Are there still experts? (AI Drawing: Basic Installation and Use of ComfyUI)
Actually, during my previous learning of Stable Diffusion, I learned about AI video generation tools like Runway Gen2 and Pika, which, while useful, require payment.
A monthly subscription can easily cost dozens of dollars, and for someone like me who hesitates even to buy a game subscription, this would be a death sentence.
However, our old friend, the company that developed Stable Diffusion, launched Stable Video Diffusion (SVD) last year, a free and efficient AI video generation tool that has entered the public’s view.
Like the other tools mentioned, SVD can help users turn a static image or a piece of text into a series of smooth, flowing, and stylistically unique video frames.
Whether it’s animations, video materials, or the currently trending short skits on short video platforms, they can all be generated using SVD.
So let’s stop the chit-chat and start today’s lesson!
Downloading and Installing SVD
I previously had a brief article on SVD, but let’s consider this a fresh start.
Downloading SVD is very simple; just go to the Hugging Face official website:
https://huggingface.co/collections/stabilityai/video-65f87e5fc8f264ce4dae9bfa
Here you can see two models in the red boxes; these models are used to distinguish model parameters. The version without the “xt” suffix is trained on 14-frame videos, while the version with “xt” has been fine-tuned on an additional 25-frame video. I recommend downloading the latest xt-1-1 version model.
After verifying your email, select the model in the red box below for download.
This model should be placed in the CheckPoint model folder of ComfyUI:
If you haven’t downloaded ComfyUI yet, you can check out my previous article:
What! Are there still experts? (AI Drawing: Basic Installation and Use of ComfyUI)
Once installed, you can click on run_nividia_gpu here to run ComfyUI.
Of course, this SVD feature is also available in Stable Diffusion, as the developers of ControlNet have integrated it into Stable Diffusion:
https://github.com/lllyasviel/stable-diffusion-webui-forge
Whether through ComfyUI or WebUI, running the SVD model requires certain GPU performance.
According to tests conducted by the original video uploader, the VRAM requirement for rendering a 14-frame video is about 12GB, and for 25 frames, it is higher. While it can run with lower VRAM, you should be prepared for potential out-of-memory errors or slower performance.
Additionally, during the video generation process, other auxiliary means will be used to ensure the final result meets our expectations, which will require extra VRAM. Currently, if you have 16GB of VRAM, you can use SVD and similar video generation models without issues.
Of course, everyone can try it out first to see how it works; if it doesn’t work, I can recommend an online alternative.
Using SVD
Since the video tutorial uses ComfyUI, my notes here will also use ComfyUI for learning.
After clearing the default workflow, drag the SVD image-to-video workflow from the download link at the end of the article into the ComfyUI interface.
First, load the SVD model we downloaded earlier in the CheckPoint section at the top left:
Then, in the image loader below, place the image you want to animate:
Note: It is best for the initial size of this image to match the final generated video size.
Size adjustments can be made in the img2vid_Conditioning on the right:
Options like video height, width, frame count, and frame rate can all be adjusted here. The recommended video size is the default 1024*576 (16:9), as SVD is trained on videos of this size.
The number of frames in the video determines the total length, and it varies depending on the selected SVD model; the original version is 14 frames, while the XT version is 25 frames.
Here, I changed Video_frames from 14 to 15, leaving the others unchanged for now.
The Ksample on the right looks familiar; if you don’t know what it is, you can keep it at the default setting.
Next, click on the top of the right toolbar: Add Prompt Queue.
This will take some time, estimated at around 2-3 minutes, and then a video will be generated based on the placed image on the far right.
This video can also be found in the output folder of ComfyUI, and you can right-click to save it.
Of course, this is just the most basic operation process. Running the image with these default parameters can make it move, but the final effect might feel a bit strange. I’ll explain more detailed advanced parameters in the next article!
Now the public account has a new article comment and reply function. If anyone encounters any problems, feel free to leave a comment or send a private message.
See you in the next note!
Best quality, masterpiece, ultra high res, (photorealistic:1.4),
smile,a girl<lyco:xiaofu:0.7>(((lovely))),((Black hair, twin ponytails, sitting on a small stool))
Negative prompt: (easynegative:1.0),(bad-hands-5:1.0),(realisticvision-negative-embedding:1.0),paintings, sketches, (worst quality:2), EasyNegative, ng_deepnegative_v1_75t,
-
Steps: 40
-
Sampler: DPM++ 2M SDE Exponential
-
CFG scale: 7
-
Seed: 1783010975
-
Size: 512×512
-
Model hash: 7c819b6d13
-
Model: majicmixRealistic_v7
-
Denoising strength: 0.6
-
Clip skip: 2
-
Hires upscale: 2
-
Hires upscaler: R-ESRGAN 4x+
-
Lyco hashes: “xiaofu: b1011cfa133e”
-
Version: v1.8.0
Summary of Issues:
Some friends may encounter the following prompt after installing ComfyUI and dragging the workflow to the interface:
ComfyUI web interface:“When loading the graph, the following node types were not found:VHS_VideoCombineNodes that have failed to load will show as red on the graph.”
Solving this is also very simple; just follow my steps: first, close the entire ComfyUI, then find ComfyUI’s root directory:
Which is the place with the startup options; double-click the path above and enter CMD.
This interface will appear:
In this interface, enter the following two lines of code, first input this line:
python_embeded\python.exe -m pip uninstall -y opencv-python opencv-contrib-python opencv-python-headless
Then press enter, after it finishes, input the second line:
python_embeded\python.exe -m pip install opencv-python==4.7.0.72
Then press enter, and after a while, you will see this interface:
Then restart ComfyUI, and the problem will be resolved!
Some article materials sourced from
Original tutorial link (download link in the original video description):
https://www.bilibili.com/video/BV1iZ421t7Ax/?spm_id_from=333.337.search-card.all.click&vd_source=24f33eddf5587fb40e947798abce6715