What is StoryDiffusion?
StoryDiffusion is an open-source image and video generation method from researchers at Nankai University and ByteDance. It uses Consistent Self-Attention to keep characters looking the same across a sequence of images, and a Semantic Motion Predictor to turn those images into smooth long videos. The code is MIT-licensed, with a demo on Hugging Face.
Standard diffusion models generate each image independently, so a character's face and clothing drift from panel to panel. This project adds an attention mechanism that shares identity information across a batch of images, making multi-panel comics and storyboards coherent. A second component predicts motion between frames, so a series of consistent images can become a longer video, including from images you supply.
Main Features
- Generates image sequences where characters keep the same identity and attire.
- Produces multi-panel comics in various art styles.
- Keeps several distinct characters consistent within the same sequence.
- Creates long videos by predicting motion between consecutive frames.
- Accepts user-supplied images as starting conditions for video.
- Publishes source code under the MIT license, alongside an arXiv paper.
- Offers an interactive demo on Hugging Face Spaces.
How to Get Started with StoryDiffusion
- Read the project page to see sample comics and videos.
- Try the Hugging Face Spaces demo with your own character prompts.
- For local use, clone the code repository and follow its setup instructions.
- Generate a consistent image sequence, then use the video component to animate it.
StoryDiffusion Pricing
It is a free research project: the code is MIT-licensed and the demo runs on Hugging Face. There is no paid plan, but running it yourself needs your own compute.
Who Should Use It?
- AI researchers working on consistent image and video generation
- Developers building comic or storyboard tools
- Comic artists experimenting with AI-generated panels
- Filmmakers sketching storyboards with recurring characters
- Hobbyists who are comfortable running open-source models
StoryDiffusion Alternatives
- Child Book AI: paid, turnkey children's book creator with printing
- Storyboard Hero AI: hosted AI storyboards for video projects
- Story Illustrator AI: illustrates stories through a hosted app
Frequently Asked Questions
Is StoryDiffusion free?
Yes. The source code is released under the MIT license and there is a free demo on Hugging Face Spaces. There is no paid plan, although running the model yourself requires your own compute.
Who made StoryDiffusion?
It was created by Yupeng Zhou, Daquan Zhou, Ming-Ming Cheng, Jiashi Feng and Qibin Hou, from VCIP at Nankai University and ByteDance. The method is described in a paper on arXiv, and the code is public.
Can it make videos as well as comics?
Yes. After generating a consistent image sequence, its Semantic Motion Predictor creates transitions between frames to produce longer videos. You can also start video generation from your own images rather than generated ones.