What is Stable Diffusion?
Stable Diffusion is a type of deep learning, text-to-image model released in 2022. It is primarily used to generate detailed images conditioned on text descriptions, though it can also be applied to other tasks such as inpainting, outpainting, and generating image-to-image translations guided by a text prompt. It was developed by the start-up Stability AI in collaboration with a number of academic researchers and non-profit organizations.
Stable Diffusion is a latent diffusion model, a kind of deep generative neural network. Its code and model weights have been released publicly, and it can run on most consumer hardware equipped with a modest GPU with at least 8 GB VRAM. This marked a departure from previous proprietary text-to-image models such as DALL-E and Midjourney which were accessible only via cloud services.
How to start using Stable Diffusion
Before you can begin using Stable Diffusion there are a variety of dependencies you will need. Stable Diffusion relies primarily on Python and Git, however for it to be effective you will also need a model (we’ll get to that in just a second). You will also need at least 20-30Gb of disk space and as mentioned previously, 8Gb of virtual RAM for Stable Diffusion to work effectively. To begin, download Python 3.10+ by clicking here. Scroll to the bottom and click on the Windows installer. Depending on your system you will need either 32-bit or 64-bit. When you are ready to install be sure to select “Add Python 3.x to PATH” otherwise it will not work. Then click “Install now.” After that you can move on to the Git dependency.
To get Git-csm click here. Once you are on the home page select the Standalone Installer. You will need either 32-bit or 64-bit depending on your system. Open the installer and select Yes for administrator. Select Next and you’ll be able to select where you want to install Git. When you have selected where you want to install Git select Next and you’ll move to the components page. Make sure all of the boxes are checked before moving to the next page. Select Next until you arrive to the “Choosing the default editor used by Git” page. Select the drop down menu and choose “Use Notepad as Git’s default editor.” After that is done leave everything else as default and select Next until Git begins to install.
Now you will need to create a dedicated folder for Stable Diffusion. This can be anywhere on your computer. The folder could be given any name though I recommend naming it “SD” or “Stable Diffusion” to make it easily identifiable. Once the folder is created click on the navigation panel, select all of the text, delete them, and type “cmd.” This should open up the command line interface for the folder you created. If the command line interface does not pop up then Git was incorrectly installed. If the command line interface does pop up enter “git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui” without the quotations. This will install Stable Diffusion webui to your folder. Once installed navigate to the Stable Diffusion folder and run “webui-user.bat” for the first time. This will install necessary extensions. Congratulations you now have Stable Diffusion! Before you can use it to generate images effectively you may want a good model and LoRA.
What are models and what is a LoRA?
Models, also known as checkpoint files, are trained Stable Diffusion weights used for the purpose of generating general or a particular style of images. In layman’s terms it is a collection of images of a similar type or style used to produce a specific outcome. If you want a realistic generated image, then you’ll want a model based on realistic data. If you want a stylized image you will want a stylized model and so on.
A LoRA is a Low-Rank Adaptation. A LoRA is used to further specify the generation of the image. For example, if you want to ensure that a gray cat appears in the generated image you may consider using a LoRA trained on gray cat images. You can use more than one LoRA at a time. However too many LoRA’s in use may produce undesirable results. You can get models and LoRA’s at civitai.com.
In Civitai each image will be abled either “Checkpoint” or “Lora.” Brows around and download a model that you would like your generated images to be based on. Once you download your model and/or LoRA go to your Stable Diffusion folder. When opened navigate to the models folder. Models will go into the “Stable-diffusion” folder and LoRa’s go into the “lora” folder. Run Stable Diffusion for the second time and you will be able to select your model and/or LoRA. For a full tutorial on how to use Stable Diffusion webui be sure to stay tuned for an upcoming article.