Sample Video
Here, we show two applications of DeepFlow algorithm on video processing. With our parallelization, we can speedup those applications. For both applications, we use the following sample video. This video is a ~4 seconds clip of a YouTube video.
Video Stylization
We can use deep learning to apply the style of one picture to another. For example, by appling the style of Vincent van Gogh's paint to your selfie, you will get your portrait painted by him. In this example, we will use the paint The Starry Night below.
If we want to transfer the style of a video, we can simply process frame by frame. However, the resulting video will be very noisy due to the loss of temporal consistency. Incorporating temporal loss in the loss function will fix the noisiness and the temporal loss can be calculated using optical flow.
There's an existing implementation of video style transfer and this implementation uses DeepFlow to calculate the optical flow between frames. The stylization of videos is very slow and companies like deepart charges €99 (~US$123) for stylizing a 10-second video.
We present the stylization result of the sample video below.
You can stylize any video of your choice following our guide here.
Slow Motion Video (Advanced Feature)
With slow motion cameras, we can track objects with high speed (such as bullet). However, we might want the captured video to go even slower so we can see the details more clearly (for example, the sample video above).
An intuitive way to further slow down the video is simply reducing the FPS, which has negative impact on the video quality. Another way is to interpolate fake frames between each pair of original frames using optical flow, and the resulting video will look much more smooth.
We use DeepFlow to generate optical flow and the generated fake slow motion video is below (1 fake frame between each pair so it's 2x slow down).
There's only existing slow motion video software using default OpenCV optical flow algorithm. We wrote a tool to make the software able to use DeepFlow (so we listed it as an advanced feature). You can try your own video or reproduce our results by following this guide.