Finally, Style Transfer For Smoke Simulations! 💨

Finally, Style Transfer For Smoke Simulations! 💨


Dear Fellow Scholars, this is Two Minute Papers
with Károly Zsolnai-Fehér. I can confidently say that this is the most
excited I’ve been for a smoke simulation paper since Wavelet Turbulence. Wavelet Turbulence is a magical algorithm
from 2008 that takes a low-quality fluid or smoke simulation and increases its quality
by filling in the remaining details. And here we are, 11 years later, the results
still hold up. Insanity. This is one of the best papers ever written
and has significantly contributed to my decision to pursue a research career. And, this new work performs style transfer
for smoke simulations. If your haven’t fallen out of your chair
yet, let me try to explain why this is amazing. Style transfer is a technique in machine learning
research where we have two input images, one for content, and one for style, and the output
is our content image reimagined with this new style. The cool part is that the content can be a
photo straight from our camera, and the style can be a painting, which leads to the super
fun results that you see here. An earlier paper had shown that the more sophisticated
ones can make even art curators think that they are real. However, doing this for smoke simulations
is a big departure from 2D style transfer, because that one takes an image, where this
works in 3D, and does not deal with images, but with density fields. A density field means a collection of numbers
that describe how dense a smoke plume is at a given spatial position. It is a physical description of a smoke plume,
if you will. So how could we possibly apply artistic style
from an image to a collection of densities? This doesn’t sound possible at all. Unfortunately, the problem gets even worse. Since we typically don’t just want to look
at a still image of a smoke plume, but enjoy a physically correct simulation, not only
the density fields, but the velocity fields and the forces that animate them over time
also have to be stylized. Hmm. Again, that’s either impossible, or almost
impossible to do. You see, if we run a proper smoke simulation,
we’ll see what would happen in reality, but that’s not stylized. However, if we stylize, we get something that
would not happen in mother nature. I have spent my master’s thesis trying to
solve a problem called fluid control, which would try to coerce a smoke plume or a piece
of fluid to take a given shape. Like a bunny, or a logo with letters. You can see some footage of what I came up
with here. Here, both the simulation and the controlling
force field is computed in real time on the graphics card and as you see, it can be combined
with Wavelet Turbulence. If you wish to hear more about this work,
make sure to leave a comment, but in any case, I had a wonderful time working on it, if anyone
wants to pick it up, the paper and the source code, and even a Blender addon version are
available in the video description. In any case, in a physics simulation, we’re
trying to simulate reality, and for style transfer, we’re trying to depart from reality. The two are fundamentally incompatible, and
we have to reconcile them in a way that is somehow still believable. Super challenging. However, back then when I wrote the fluid
control paper, learning-based algorithms were not nearly as developed, so it turns out,
they can help us perform style transfer for density fields, and also, animate them properly. Again, the problem definition is very easy,
in comes a smoke plume, we add an image for style, and the style of this image is somehow
applied to the density field to get these incredible effects. Just look at these marvelous results. Fire textures, starry night, you name it. It seems to be able to do anything! One of the key ideas is that even though style
transfer is challenging on highly detailed density fields, but it becomes much easier
if we first downsample the density field to a coarser version, perform the style transfer
there, and upsample this density field again with already existing techniques. Rinse and repeat. The paper also describes a smoothing technique
that ensures that the changes in the velocity fields that guide our density fields change
slowly over time to keep the animation believable. There are a lot more new ideas in the paper,
so make sure to have a look! It also takes a while, the computation time
is typically around 10 to 15 minutes per frame, but who cares! Today, with the ingenuity of research scientists
and the power of machine learning algorithms, even the impossible seems possible. If it takes 15 minutes per frame, so be it. What a time to be alive! Thanks for watching and for
your generous support, and I’ll see you next time!

65 thoughts on “Finally, Style Transfer For Smoke Simulations! 💨

  1. I hope more scholars follow your team's lead and start releasing Blender add-ons for their work 😍. That said, what do we have to do for a smoke sim, stylised as your chihuahua Lisa 😶

  2. I was going to run off to Blender developer forums to ask for Wavelet Turbulence algorithm within Blender and then you drop the amazing surprise that there is already an addon available for it. Its just fantastic how Open Source is becoming the norm, combined with the news that even Nvidia has become a Patron level supporter on the Blender Development Fund, What a time to be alive! 🙂

  3. Blender addon? Wow, beautiful. That's actually a lot more useful than most of these algorithms that only stays for the teoretical side with zero immidiate practical uses.

  4. This is truly amazing. I can't wait for it to be compiled in a way so we could use it in vfx software like Houdini. Sharing this video!

  5. 2019: It also takes a while, the computation time is typically around 15 minutes per frame, but who cares!

    2059: It also takes a while, the computation time is typically around 15 micro-seconds per frame, but who cares!

  6. That's very cool. I think the video (or rather the paper's videos) could use an example of where you'd use this, to really drive home WHY this is important. The volcano scene is along that lines, but I wasn't sure why it was really necessary. What problem does this solve?

  7. This will be useful for when we really need to make some spikey smoke? Maybe for the next Avengers movie.

  8. somebody really needs to do real time analog to high quality ai video conversion
    mainly focused on removing breakup

  9. 3:08 – Uh, personally, I believe that the Sato et al. 2018 paper has more realistic results. The biggest issue with this technique is that features appear and disappear from nowhere. I'd love to see a version of this approach that only applied (heavily constrained) forces on the simulated smoke instead of (and without) manipulating the densities. This is challenging, because you need to choose the forces such that there is no overall bias that makes the simulation give completely different results (But this is preferable still to smoke appearing from nowhere – and just looking at the videos, without reading the paper, it seems to stylize just a pre-baked simulation, instead of truly affecting the simulation to produce a style), and you need to be able to choose a set of forces that will create a stylized result _in the future_.

  10. So… What about sending this patch to blender? https://github.com/dfelinto/blender/blob/master/intern/smoke/extern/smoke_API.h it is not there https://developer.blender.org

  11. Could we use in the opposite direction too? If so, we might be able to apply data assimilation by changing the initial condition at various spatial scales.

  12. Tx for the review. Just when you say “even if it takes 15min per frame to compute, who cares”, well … the planet cares I guess. If we use coal power plants to compute 15min/frame style transfer simulation, we wont survive much longer on this warming ball. Resources are finite.

  13. Gonna unsubscribe. Tired of seeing all this stuff that is just gonna be locked away until it can be monetized, which to me might as well be gold bars moving shelves inside a vault in a different country. While I'm desperately super gluing my teeth and eating careful because access to dental care might as well be ownership of a private jet for all the distance they have from my grasp.

  14. Jeesh. The last few years, despite my reading more papers than in the previous 20….I feel more and more the reality that the bleeding edge is rapidly moving away from my grasp. It is both terrifying and exciting as a lover of learning to realize that the horizon of new things to learn continues to rapidly accelerate away faster than I can discover and engage it.

  15. Hmm, smoke filling defined shape looks a lot like ink diffusion in water… where the turbulence and vector is constrained more quickly by the fluid volume (as opposed to air)

Leave a Reply

Your email address will not be published. Required fields are marked *