The team behind the tool share more of what they’ve been up to.
A little while ago, I spotted a demo online of a new tool called EbSynth that brought live-action video into a painterly world, based on a single input keyframe. It was incredibly captivating.
Since then, the tool by Secret Weapons has been used for many other things, and now it may have a place for doing VFX re-touches.
befores & afters asked the 3-person Secret Weapons team about where EbSynth is up to, where deep learning might come in in the near future, and how it might become a production-ready tool soon.
b&a: Tell me how EbSynth went from a tool for rotoscopy to what it is today?
Secret Weapons: We created EbSynth as a tool for automatic rotoscopy. The motivation was to help Jakub, our lead artist, bring his concept art to life. But once we released it to the public, people started using it for all kinds of different purposes. They embraced the key idea, which is that EbSynth propagates changes made in one frame (or a set of keyframes) into the whole video.
Sometimes, the way people used it was surprising to us. Cartoon enhancement, for example. Artists started adding texture and shading to 2D animation and it looked great. 3D artists were using EbSynth to break the polished 3D look with painterly brushstrokes, which we thought was amazing. And then someone used it along with Faceapp to age themselves in a video. That was a lightbulb moment for us. Of course, you can use EbSynth for doing VFX retouches! Instead of painting over the video frame, you make a photorealistic edit. EbSynth doesn’t care, it propagates anything from the keyframe to the inbetweens.
Around that time, The Irishman came out on Netflix, and while watching the making-of documentary, we realized that EbSynth could be used for professional VFX production as well. We believe it has great potential to speed up a lot of work that would otherwise have to be tediously tracked. The results we see online are often really funny, and we enjoy watching all those vandalized movie scenes or colorized shots from black and white films.
b&a: People might think there’s an A.I. or deep learning side of EbSynth, but I don’t think that’s part of it, is it? Could A.I. form part of EbSynth in the future?
Secret Weapons: There is no AI involved in EbSynth. Our aim was to respect the keyframe content as much as possible and provide the artist with maximum control over the process. This can be hard to achieve with current deep learning techniques, so we decided to take a different approach instead. What EbSynth does under the hood is a non-parametric example-based synthesis. It works by dicing the input keyframe into many small pieces which are then reassembled to form the output frame. This way the input pixels are preserved 1:1.
Although EbSynth does not use AI at the moment, we are following the machine learning research very closely, and we are exploring opportunities to utilize it in EbSynth. Most recently, we have been looking into how we can improve our optical flow estimation with deep neural networks.
b&a: What do you see as some of the film, VFX or animation uses of EbSynth right now?
Secret Weapons: We believe EbSynth could be a powerful tool for VFX artists; they can think of it as free 2D tracking. Any edit or photomanipulation made in a single frame can now be easily propagated to the remaining frames. It could be used for removing spots, adding digital make-up, aging characters, relighting, beautifying, making local color changes, or coloring the whole black and white films.
Sometimes it even works for roto. You paint the mask in one frame, and EbSynth animates it for you. It can add texture and shading into a 2D cartoon, enhance 3D renders with brushstrokes, and the most significant use case in animation is the rotoscope.
You can shoot a home-made video with DIY props, use socks to elongate your ears, for example, then paint an alien over one frame, and EbSynth brings it to life. That’s where we see the creative potential of our tool. Even artists without much resources can create complex animated content.
EbSynth is currently in beta and can be downloaded here.
