Style Transfer using Deep Neural Network and PyTorch

Nowadays everyone is excited about doing projects using machine learning or deep learning. Through this blog, I will give you a chance to be ‘ Picasso ‘ of deep learning as we are going to explore the method of style transfer using Deep Convolutional Neural Networks. we will use pre-trained network VGG19 for that.

Don’t worry, it just sounds tough but actually way easy. You just need to be familiar with Python, PyTorch and some Deep Learning stuff such as CNN and how to use pre-trained networks (as we will be using a pre-trained CNN for our style transfer). I will brush up your concepts about CNN.

A convolutional layer + activation function, followed by a pooling layer, and a linear layer (to create the desired output size) make up the basic layers of a CNN.

We can use either of VGG16 and VGG19 for feature extraction as they are performing very well as compared to others in case of style transfer. For example, here I have used VGG19.

Other than VGG, you can use SqueezeNet, it is faster but results are worst and in case of Inception, it performs well but you have to change striding/kernels, max pooling to average pooling, search over various layer combos. So VGG is best at the moment.

Style transfer relies on separating content and style of an image. Our target is to create a new image containing style of style image and content of content image( base image).

VGG19 is consists of two parts:

For style transfer we need only features portion ,so we will load in that and freeze the weights.

We will then calculate the gram matrix for output of each convolutional layer used for style feature extraction, to find a co-relation between any spatial information.

Gram matrix is calculated by multiplying a matrix by its transpose.

Individual layer style weight

Content and Style weight

Content loss

Style loss

Total Loss

It will be calculated by adding style and content loss after weighting them with alpha and beta.

Our aim here is to minimize the total loss by iterating and updating the values.

By reading this blog, you will get an overview about how style transfer happens and picture editing apps like Prisma works. Here we used gram matrix calculation but you can also improve your style transfer by using various other approaches such as encoder and decoder networks etc.

Moreover, the major drawback of this technique is we are paying in terms of time for better results, you can also search for real-time style transfer as an update on the existing one.

Related posts:

When talking about the Custom software product development process, little do we realize that a major part of the process is involved in NOT building the software! In fact, coding is only a small…

As heavy users of Azure we could choose among many different options in terms of compute, so six months ago we sat down to evaluate our possibilities and we agreed on one destination: Kubernetes. We…

Technology is in a state of continuous flux. While some businesses are adapting fast enough to brace this rapid evolution of technology, most businesses are reactive and unable to leverage…