StoryDiffusion
VerifiedOpen-source research model for consistent characters across comics and long videos
Product preview
StoryDiffusion
What it is
StoryDiffusion is an open-source research project from Nankai University and ByteDance, presented as a NeurIPS 2024 spotlight paper. It tackles a common weakness of diffusion models: keeping the same character looking the same across many images. Its consistent self-attention technique lets a batch of generated images share identity and clothing details, which makes it suitable for comics and visual storytelling in different styles. A second component, an image semantic motion predictor, turns a sequence of consistent or user-supplied images into longer video clips with smooth transitions. Code is released on GitHub under the Apache 2.0 license, with demos available. Running it locally requires a capable GPU and some technical setup.
Key features
- Consistent self-attention for character identity
- Multi-panel comic generation
- Long video generation from image sequences
- Apache 2.0 licensed code
What we liked
- + Free and open source
- + Strong focus on character consistency
- + Backed by peer-reviewed research
Worth noting
- · Requires a GPU and technical setup to run locally
- · Research code with limited ongoing updates
Editor's notes
More context from AI BirdView
Anyone who has tried to make a comic with an image generator knows the frustration of a hero whose face and outfit change in every panel. StoryDiffusion is a research answer to that problem. Its consistent self-attention approach lets a set of images share character details as they are generated, and a companion motion predictor stitches those frames into longer, coherent video.
- Keeps faces and outfits consistent across a sequence of generated images
- Produces comics in a range of art styles from a set of prompts
- Generates longer videos by predicting motion between consistent frames
- Accepted as a spotlight paper at NeurIPS 2024
- Released on GitHub under the permissive Apache 2.0 license
StoryDiffusion is best for developers, researchers and technically comfortable artists who want to experiment with consistent character generation or build it into their own pipelines. A practical tip: try a hosted demo first to see whether the results suit your style before investing time in a local install, and when you do run it yourself, keep your character description identical across every prompt in a batch for the most stable results.
Reader reviews
From people who've used StoryDiffusion. Emails are never shown publicly.
No reviews yet. Be the first to share your experience.
Alternatives to StoryDiffusion
Other tools we like in this space.
Stable Diffusion
The open-weights model powering a whole ecosystem
FLUX
Frontier open image models from the SD original team
ComfyUI
The node graph that powers serious AI imaging