Integrating SAM and Stable Diffusion: New AI Painting Techniques

Since the launch of the SAM “Segment Anything” model, a wave of creative works has begun, with ideas and actions emerging!

The developer from PaddlePaddle AI StudioThe Singing Alchemist has created an evolved version of SAM, integrating SAM and Stable Diffusion to achieve both “segmentation” and “generation” capabilities in one application, which is available for online use!

Free Online Application

Creator:The Singing AlchemistAuthor homepage:https://aistudio.baidu.com/aistudio/personalcenter/thirdview/338262

Application parameter interpretation:

Text: The segmentation target of the SAM image, if you input “car”, it will segment the car in the image.

Prompt:Controllable text generation for segmentation targets, input the corresponding Prompt

Advanced options:Advanced options

Images:Number of generated images

Image Resolution:Resolution of the generated images, generally used as 512*512

Control Strength:The weight of the control model’s conditions merged into the Stable Diffusion model, default is 1

Guess Model:To obtain more diverse results, generating results that do not strictly adhere to the image conditions

Segmentation Resolution:Image segmentation resolution

Steps:Inference steps

Guidance Scale:A method to increase adherence to guidance generation (in this case, text) and overall sample quality. It is also known as classifier-free guidance. Simply put, adjusting it can improve the use of images with better quality or more diversity. Values between 7 and 8.5 are usually good choices for Stable Diffusion. Default is set to 7.5.

If the value is too high, image quality may improve, but diversity will decrease

If the value is too low, image quality may worsen, but diversity will increase

Seed:Random seed number

eta (DDIM):DDIM sampling method

Added Prompt:Additional prompts added to the above Prompt to improve the generated images

Negative Prompt:Controls the SD generated images to filter out those related prompts

Technologies Used

Segment Anything:A large model for prompt segmentation

ControlNet Canny:Edge detection on segmented images to serve as control conditions for image generation

Stable Diffusion:A large model for text-to-image diffusion

With the powerful image segmentation capabilities of the Segment Anything Model, we can effectively obtain segmented pixel maps. Then, using ControlNet with Stable Diffusion, we can use the control maps for different semantic segmentation images and text (prompts) as control conditions to complete the Segment to Image task. Thus, the Segment Anything Model integrates with ControlNet and Stable Diffusion.

Run the Fork Project

Fork This Project

Start Entering the Online Development Environment

It is recommended to choose the V100 32GB development environment.

PaddlePaddle AI Studio offers 56 hours of free V100 32G computing power for running projects weekly

Install ppdiffusers Dependencies

1# Install paddlenlp and ppdiffusers dependencies
2!pip install "paddlenlp>=2.5.2" "ppdiffusers>=0.11.1" safetensors --user

1# Use Canny edge detection image as control condition
2!python gradio_canny2image.py

1# Use ADE20K segmentation protocol image as control condition
2!python gradio_seg2image_segmenter.py

Install ppsegSAM Dependencies

1# Install dependencies
2!pip install -r requirements.txt --user

1# Download sample images
2!mkdir ~/examples
3%cd ~/examples
4!wget https://paddleseg.bj.bcebos.com/dygraph/demo/cityscapes_demo.png

1# Download vocabulary
2%cd ~
3!wget https://bj.bcebos.com/paddleseg/dygraph/bpe_vocab_16e6/bpe_simple_vocab_16e6.txt.gz

1# Run the script
2!python scripts/text_to_sam_clip.py --model-type vit_h

AI technology is iterating daily, and we look forward to the wonderful creations of developers. We also invite developers to join AI Studio for open-source creation. For detailed creative incentive system, please click the bottomRead the Original Text.

END

#Previous Recommendations #

Artists Praise! Creating Chinese Ink Landscape Paintings Based on PaddlePaddle (qq.com)

AI Industry Veteran! AI Sun Yan Zi? AI Dubbing? Let’s Explore Voice Synthesis! (qq.com)

How to Use Large Models Prompt to Solve Industry Scene Problems? A Chinese Tutorial from Big Companies is Here! (qq.com)