So, I quickly implemented the ZipLoRA by 🤗🧨 (Some people have already noticed though)
code:
I hope it helps somehow and feel free to drop your comments and feedback~
Big thanks to the authors for their awesome work 🙌
With collaborators
@Google
we're announcing 💫 ZipLora 💫! Merging LoRAs has been a big thing in the community, but tuning can be an onerous process. ZipLora allows us to easily combine any subject LoRA with any style LoRA! Easy to reimplement 🥳
link:
In my experiment, SVDiff looks very promising!!
The file size is <1 MB, which is x3 smaller than LoRA and the result looks comparable as the paper.
left: LoRA, right: SVDiff
SVD 1.1 seems to have been released 👀
HF:
For those who can't wait for the official announcement, you can try it on this colab for free.
Colab:
1/3
#SVD
@StabilityAI
released a very powerful image-to-video model called "Stable Video Diffusion" 📽️
For those who are super excited to try it but don't have enough resources, I hacked it to work on Colab free plan 🤗 Check it out!
Colab:
#StableVideo
#SVD
I implemented SCEdit, especially SC-Tuner for now!
The experiments in the paper were done with SD 1.5/2.1 but you can apply it to any UNet-based diffusion models including SDXL.
code:
colab:
Any feedback is welcome!
Alibaba announces SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing
paper page:
Image diffusion models have been utilized in various tasks, such as text-to-image generation and controllable image synthesis. Recent
Super excited to announce the release of Japanese Stable Diffusion 🇯🇵🎉
Japanese Stable Diffusion is a Japanese-specific Stable Diffusion capable of generating Japanese-style images given any Japanese text.
#JapaneseStableDiffusion
#StableDiffusion
#AIart
SVDiff-pytorch was updated 🎉Now, you can get better results with less training steps!
Please see code for more details!
code:
"portrait of {gal gadot} woman wearing kimono"
In my experiment, SVDiff looks very promising!!
The file size is <1 MB, which is x3 smaller than LoRA and the result looks comparable as the paper.
left: LoRA, right: SVDiff
paper:
problem to solve:
Existing domain-tuning methods (eg. DreamBooth (DB)) can lead to overfitting due to finetuning the whole parameters on the small data. In addition, domain-tuning for multiple target concepts is difficult.
summary: 👇
1/4
@StabilityAI
released a very powerful image-to-video model called "Stable Video Diffusion" 📽️
For those who are super excited to try it but don't have enough resources, I hacked it to work on Colab free plan 🤗 Check it out!
Colab:
#StableVideo
#SVD
paper:
problem to solve:
Existing domain-tuning methods (eg. DreamBooth (DB)) can lead to overfitting due to finetuning the whole parameters on the small data. In addition, domain-tuning for multiple target concepts is difficult.
summary: 👇
1/4
In my experiment, SVDiff looks very promising!!
The file size is <1 MB, which is x3 smaller than LoRA and the result looks comparable as the paper.
left: LoRA, right: SVDiff
OneLLM: One Framework to Align All Modalities with Language
paper page:
Multimodal large language models (MLLMs) have gained significant attention due to their strong multimodal understanding capability. However, existing works rely heavily on
Würstchen v2, by
@pabloppp
and
@dome39931447
, is just awesome! Only 2 seconds of inference to generate 1024px images 🤩
Very excited about next week's release!
I am very excited to announce the first release of the Japanese LLM by
@StabilityAI_JP
🎉
We have released 2 7B models and both are in 1st and 2nd place in the evaluation leaderboard 🤩🔥
base:
instruction:
Now, the other important parameters are supported!
For example, the lower `motion_bucket_id` (default: 127) leads to less motion.
Tested with the Windows background.
(top: default 127, down: 31)
@StabilityAI
released a very powerful image-to-video model called "Stable Video Diffusion" 📽️
For those who are super excited to try it but don't have enough resources, I hacked it to work on Colab free plan 🤗 Check it out!
Colab:
#StableVideo
#SVD
Stable Cascade (Würstchen v3) by
@dome_271
and
@pabloppp
is out!
I've been a big fan of Würstchen since v1 but v3 is absolutely the best as their results are shown.
code:
Big congrats to both of you and
@StabilityAI
!
It finally happened. We are releasing Stable Cascade (Würstchen v3) together with
@StabilityAI
!
And guess what? It‘s the best open-source text-to-image model now!
You can find the blog post explaining everything here:
🧵 1/5
paper:
problem to solve:
The existing T2I domain-tuning methods require long training time and multiple concept images. Plus, fine-tuning methods such as Dreambooth need high storage requirements for several GB of ckpts.
summary:
1/5
Our latest work was finally released!
Model merging shows great performance w/o training. However, it relies on developers' experience and intuition a lot!
To address this issue, we introduced "Evolutionary Model Merge" to automate model merging based on specified metrics.
1/2
Introducing Evolutionary Model Merge: A new approach bringing us closer to automating foundation model development. We use evolution to find great ways of combining open-source models, building new powerful foundation models with user-specified abilities!
Finally, we released
#JSDXL
, Japanese-specific text-to-image model built on top of SDXL 🤗🎆🎊
#JSDXL
is capable of not only inputting prompts in Japanese but also generating Japanese-style images that understand and reflect Japanese culture and elements.
1/5
#JSDXL
was applied a PEFT method called OFT to tune into the JA domain while maintaining the versatility of the powerful SDXL.
Also, it's more efficient because
#JSDXL
does have only one text encoder.
For more details, please visit the HF page :)
4/5
Kandinsky 2.1 inherits best practicies from Dall-E 2 and Latent diffucion, while introducing some new ideas.
As text and image encoder it uses CLIP model and diffusion image prior (mapping) between latent spaces of CLIP modalities. This approach increases the visual performance
We just released "Japanese Stable VLM", which applied the LLaVA 1.5 technique but was trained from scratch on Japanese data 🎉🐶⚔️
Also, it's available for commercial use. Check it out 👍
Blog:
HF:
Colab:
@StabilityAI
released a very powerful image-to-video model called "Stable Video Diffusion" 📽️
For those who are super excited to try it but don't have enough resources, I hacked it to work on Colab free plan 🤗 Check it out!
Colab:
#StableVideo
#SVD
DiffFit was applied to DiTs but can be applied to
#stablediffusion
too as a PEFT.
Here's my WIP implementation of DiffFit.
code:
I'm still struggling with good results so I am happy to get feedback!
We have released another 2 kinds of powerful Japanese Stable LM (JSLM) under the commercial license 🎉🎉
JSLM 3B-4E1T:
(base)
(instruct)
JSLM Gamma 7B:
(base)
(instruct)
Japanese Stable Diffusion - SD trained on 100M japanese image-text pairs
The model works with prompts in the Japanese language. Requires it's own pipeline
Model:
Colab:
A single-step SDXL is out! SDXL Turbo was trained w/ a new distillation technology called Adversarial Diffusion Distillation.
You can quickly try it on
@clipdropapp
and here's my result😺
clipdrop:
paper:
HF:
Introducing SDXL Turbo: A real-time text-to-image generation model.
SDXL Turbo achieves state-of-the-art performance with a new distillation technology, enabling single-step image generation with unprecedented quality, reducing the required step count from 50 to just one.
The
paper:
problem to solve:
With existing DM-based image translation methods, there is a trade-off between maintaining the spatial layout of the input image and high-quality content.
summary:
1/4
Compared
#stablediffusion
sampling methods (ddim, plms, k_euler, k_euler_ancestral, k_heun, k_dpm_2, k_dpm_2_ancestral, k_lms)
code:
"A beautiful painting of a view of country town surrounded by mountains, ghibli"
#stablediffusion
#AIart
#AiArtwork
Introducing Sora, our text-to-video model.
Sora can create videos of up to 60 seconds featuring highly detailed scenes, complex camera motion, and multiple characters with vibrant emotions.
Prompt: “Beautiful, snowy
paper:
problem to solve:
Existing domain-tuning methods (eg. DreamBooth (DB)) can lead to overfitting due to finetuning the whole parameters on the small data. In addition, domain-tuning for multiple target concepts is difficult.
summary: 👇
1/4
Japanese InstructBLIP Alpha is built on top of our powerful JSLM model.
Japanese InstructBLIP Alpha enables not only to generate textual descriptions in Japanese for input images but also to generate answers to questions given images.
2/3
Plus, Japanese InstructBLIP Alpha is capable of recognising proper objects in Japan such as SkyTree🗼
Can't wait to see the applications using our Japanese InstructBLIP Alpha!
3/3
ImageReward, a t2i human reference reward model (RM), is out under Apache2.0.
"ImageReward outperforms existing text-image scoring methods, such as CLIP, Aesthetic, and BLIP, in terms of understanding human preference in text-to-image synthesis"
Compared to LoRA and DB in the same training step, LoRA leaded to underfitting and DB causes overfitting, while SVDiff provides good balanced results.
They also proposed a data augmentation technique called Cut-Mix-Unmix for domain-tuning for multiple concepts.
4/4
Today, I am very proud to release the new SOTA Japanese CLIP "Japanese Stable CLIP" 🥳 👏
This model significantly outperformed the rinna's CLOOB which was SOTA (and I trained too) 🎊 👏
Excited to see how the community uses it!
HF:
Colab: ↓↓
1/2
Although JSD is still not as versatile compared to SD, we hope to communicate to the research community the importance and potential of language-specific model development through JSD.
Stable Diffusion is a very powerful text-to-image model, not only in terms of quality but also in terms of computational cost.
So, why do we need Japanese-specific Stable Diffusion?
From researcher’s view, to make language-specific stable diffusion, fine-tuning is not efficient because the text tokenizer/encoder is English.
So, we had 2 stages overall, inspired by PITI.
3/5
They Proposed SVDiff, a method that performs SVD on the weights of the pre-trained diffusion model and train only the difference part of the singular values. This method is based on previous work applied to GAN (paper: ) and applied to diffusion models.
2/4
paper:
problem to solve:
One drawback of diffusion models is the slow sampling process but many improved methods have recently been proposed to generate images in fewer steps. However, these methods do not work well in the guidance setting.
summary:
1/3
paper:
problem to solve:
To further control t2i pre-trained models(PMs) with additional info (e.g. sketch), there are problems of the number of data and computational cost. Plus, fine-tuning could reduce the ability of powerful PMs.
summary:
1/5
This was released a month ago but I noticed that I haven't retweet it!
I was honored that I joined this project as the main AI researcher :)
The short anime is very touching, so please don't miss watching it~
Because we want a model to understand our culture, identity, and unique expressions.
For example, one of the famous Japanglish is "salary man" which means a businessman wearing a suit.
This is from
#stabledifussion
w/ "salary man, oil painting"
paper:
problem to solve:
Most SOTA vision-language models are very computationally expensive because large models are pre-trained on large multimodal dataset.
summary:
1/4
paper:
problem to solve:
Transformer-based multi-modal models such as MAGMA have become increasingly complex, and they lack explainability; Gradient-based interpretation methods consume high GPU memory due to the large number of parameters.
1/3
paper:
problem to solve:
Textual Inversion proceeds in the only one embedding space (P) and can be extended to the multiple embedding spaces (P+)
summary:
1/4
Our LLMs/VLM remarkably achieve SOTA performance! Ofc, all models are public on HF!
This is only our initial progress and I'm excited to expand its possibilities further!!
All details are written in our blog & paper! Check it out 🐠
2/2
Do you want to use `init_image` in
#stablediffusion
before they release?
Now I implemented k-diffusion samplers in
#stablediffusion
as glid-3-xl.
Here's the code 🎉
code:
prompt: "A ukiyoe of a countryside in Japan" by k_lms sampler
Implemented k-diffusion samplers in glid-3-xl because
#stablediffusion
uses `k_lms` sampler by default.
All k samplers (k_euler, k_euler_ancestral, k_heun, k_dpm_2, k_dpm_2_ancestral, k_lms) are supported.
"A ukiyoe of a countryside in Japan"