Back in 2024, AI image generators had a major flaw. They have been great for creating pictures of characters, but the moment you ask them to draw the same character again — in another pose, from another angle, or with another facial expression — everything starts to fall apart.
The jacket changes. The face changes. The colors shift… This problem is especially annoying if you want to use Images as keyframes to interpolate the images in-between in order to get ai animations.
And this was basically the main problem behind my master thesis.
Here I had a simple hypothesis: Since transformer models are great at understanding context and cross-references, can I leverage their capabilities for increased character consistency?
So the goal would be to have one character and create unlimited poses out of it, while keeping the same identity and style. Sounds simple. Was not…
The core issue is that image models don't have memory. They generate one image, then the next one starts from scratch again. Even if you use the same prompt and just change the generation seed by one number, a whole different character could appear.
For animation, this is a huge problem because consistency is everything. If every frame looks slightly different, the animation feels broken. You get morphing, flickering, weird transitions and that classic AI slop feeling.
The basic idea I had when I found this pickle tensor on civit AI that trained stable diffusion on character turnarounds: https://civitai.com/models/3036/charturner-character-turnaround-helper-for-15-and-21?modelVersionId=8387 The results of this pickle tensor have been verrrrry poor and you needed a lot of luck to get at least modest results.
So the question was now If this new type of image model called a Diffusion Transformer (DiT) could solve the issue.
Here I used FLUX.1-Fill-dev which is an in-painting model, which means it is usually for removing objects of images or adding small details. So my plan was to cover half of the image as inpainting area and use the other half just as reference and additional context.
The base model by itself was not really made for this, as it fails spectacularly and usually was just good at creating very tiny areas.
So I fine-tuned FLUX.1-Fill-dev with LoRA. LoRA is basically a lightweight way to teach a big AI model a new behavior without retraining the whole thing. This is useful because training a full model is extremely expensive, while LoRA can be trained with much less effort.
But still it wasn’t easy: FLUX.1 is a huge model with 21 billion parameters, which doesn't fit on a Consumer GPU easily. In addition, since it was brand new there wasn't any training framework for the model, so I needed to develop my own script which took me like a month!!! While I was testing all kind of alternatives in between.
Once I finally got the LoRa script working, It was clear to me to open source the source code so that others can benefit from my work. I remember how I stumbled on people that also seemed to achieve LoRa fine tuning of Flux Fill, but did not feel like releasing their code open source... I was so annoyed by them…
So here is the Repo Link: https://github.com/Sebastian-Zok/FLUX-Fill-LoRa-Training
Other problems I stumbled across was the creation of the training dataset. Here I tried to use other AI models to create synthetic datasets, but also this failed terribly. So I used 3D models, animated them and photographed them from all different poses.
There have been soooooo many problems during all of this but finally I got the LoRA working and the results have been amazing. Check out the GitHub Repo or my thesis about as it has some more images.
While the pose transfer was working great and proofed the capabilities of DiTs, my thesis was not done. Indeed, this was just the first part of my planned workflow for creating AI animations.
My workflow was roughly this:
- Start with one reference image of a character.
- Turn that character into a rough 3D model.
- Pose the 3D model.
- Use the rough pose as guidance for image generation.
- Let the AI start the diffusion form the guidance image including my LoRa.
- Use the resulting images as keyframes for animation.
The 3D modeling part was the easiest part as I used Stable-Diffusion-3D which was straight forward once it was running and can generate a 3D model out of a picture. But easy is relative since I had huge driver and compatibility issues, overall the project was very hard... so yeah...
The big advantage is now that the 3D model does not need to look good. It can be low-res, ugly and pixelated. It only has to provide pose and perspective information. The AI then tries to transfer the original character's style onto that pose from the unmasked image part.
In other words: the 3D model gives the body position, the reference image gives the identity. This would be a great replacement for ControlNets.
To animate the characters I used the easy path with Mixamo.
With the two character poses in place the last step was to do the keyframe interpolation. Here I planned to use a model called ToonCrafter (https://arxiv.org/abs/2405.17933) and while the paper looks amazing, the results I got have been rather poor.
So overall even when the video creation part didn't work reliable, this technique still has two major flaws: First, it needs a specialized fine-tuning dataset, which is hard to create and very time consuming. Second, after just one year after the release, all major AI image models (e.g. NanoBanana) already provide the same capabilities out of the box as they are multimodal. Which makes the technique basically obsolete.
So SPOILER ALERT — you will not find a big breakthrough, but if you want to read the full thesis, here it is ;) https://drive.google.com/file/d/16AT77IuW-jEoKjao9EzGkg67f5tCehxk/view?usp=sharing