Introduction
Art has always transcended eons of human existence. We can see their traces from prehistoric times such as Harappan art in the Indus Valley Civilization to contemporary art in modern times.. Above all, art has been a means to express one's creativity, views of how we perceive the world. As the legendary Leonardo Da Vinci has said,
"Painting is poetry that is seen more than it is felt".
What we sometimes forget is that most art follows a pattern. A pattern that we like and makes sense in our brain. The next time I see a painting, try to notice the brush strokes on it. You will see a pattern emerging from the paint. U.S, like human beings, we are experts in recognizing these patterns. Our neural mechanisms have evolved to be exceptionally cool over the years at recognizing patterns in nature..
Now you may ask why I'm ranting about art and patterns. This is because I will show you how to create art with the help of artificial brains!! In this article, we will build an artificial neural network that will extract style from one image and replicate it in the other. So are you ready?
Table of Contents
- What is neural art?
- Get acquainted with the crux
- Coding it
- Where to go from here?
- Additional Resources
What is neural art?
Let's try to understand this topic with an example.
Source [1]
The image above is the famous “The starry Night” the Vincent Van Gogh. Just look at the painting for a few minutes. Do you see? Do you notice the brush strokes? Do you see the curves and edges that define each and every object, what makes it so easy for you to recognize them?
Now let's do a quick task. Try to remember the patterns you see. Just cram your brain with every little detail. Done? Ok, now take a look at the following picture.
Source [2]
This is a photograph taken from a city called "Tubingen" located in Germany. For the next step of the task, just close your eyes and try to replicate the style of a starry night with this image. Ask yourself, if you are van gogh (Hypothetically, of course!) And they ask you to draw this photograph taking into account the styles you memorized before, how would you do it?
To think
.
.
.
.
You've done it? Excellent! You just made neural art!
.
.
.
Want to see what an artificial neural network can do?
Source [2]
You may ask how a machine accomplished such a task. It's simple once you understand the essence!
What the neural network does is try to extract the “Important points” of both images, namely, try to recognize what attributes define the image and learn from it. These learned attributes are an internal representation of the neural network, which can be seen below.
Source [2]
So you got to know the theoretical concepts involved in neural art, now let's know the practical aspects of implementing it.
Enter the brain of an artificial artist:
Neural art works as follows:
- We first define the loss functions necessary to generate our result, namely, loss of style, loss of content and total loss of variation.
- We define our optimization function, namely, backpropagation algorithm. Here we use L-BFGS because it is faster and more efficient for smaller data.
- Then we set our style and content attributes of our model.
- Then we pass an image to our model (preferably our base image) and we optimize it to minimize all the losses that we defined above.
We will know some of the important points that you should know before starting. While most of the fundamentals of neural networks are covered in this article, I will reiterate some of them and explain some additional things.
- What is a loss function? The loss function is a function that calculates the difference between the predicted values and the original values.. Basically, tells how much error has occurred in a calculation. In any machine learning algorithm, the loss function is used to estimate the performance of the model on the data. This is especially useful in the case of neural networks, where you iteratively try to make your model work better. When implementing neural art, must control three loss functions, namely:
- Loss of content, namely, the difference between the “contents” of the resulting image and the base image. This is done to ensure that your model does not deviate too much from the base image..
- Loss of style, namely, the difference between the “style” of the resulting image and the base image. To do this, you need to first calculate the gram matrix of both images and then find their difference. The Gram matrix is nothing more than finding the covariance of an image with itself. This is done to maintain the style in the resulting image.
- Total loss of validation, namely, the difference between a pixel in the resulting image and its neighboring pixel. This is done so that the image remains visually coherent..
- What is an optimization function? When we have calculated the loss function, we try to minimize our losses by changing the parameters. The optimization function helps us find out how much change is required to make our model better “optimized”. Here we implement an optimization method called the Broyden-Fletcher-Goldfarb-Shanno algorithm (BFGS). BFGS is a variant of the gradient descent algorithm in which a second order differentiation is done to find the local minimum. Read This article to get a mathematical perspective of the algorithm.
Now that we have understood what our flow will be to build a neural art, Let's go down and start hacking stuff!
Coding it!
This Diwali was interesting for me. I decided to do some research on neural art and how India lights up on Diwali day. I came across this picture “India on Diwali night”. And I thought of creating something similar along the same lines. To do that, we will combine the two images below with the help of neural art.
Source [3]
So first we will lay the foundations.
Paso 0: Install on pc Hard and its dependencies. For this, we will use a Theano backend. Change your backend by following the steps mentioned here. What's more, you should also set the proper order for the image. In the keras.json file, where has the backend changed, replace image_dim_ordering with 'tr'. So it should look like this,
"image_dim_ordering": "th"Paso 1: Then go to your working directory and configure your directory structure as shown below
|-- keras_NeuralStyle # this is your working directory | |-- base_image.jpg # this is your base image | |-- reference_image.jpg # this is your reference image
Paso 2: Start a jupyter notebook in your working directory by typing jupyter notebook and implement the following code. I will just provide you with a step by step overview of what each block does.
- First, you must import all the modules necessary to implement the code
- Then, set the paths of the images you want to project on.
- Define the required variables and give them values as shown below. Note that these values can be changed, but that can drastically change the output. What's more, make sure the value of the img_nrows variable is the same as img_ncols. This is necessary for the gram matrix to work.
- Then we define helper functions. These are responsible for handling image preprocessing.
- Create input placeholders to pass images to the model
- Load a pre-trained neural network model (if you don't know what pre-training is, see this discussion)
- Print the model summary to see which model is
- Store the names of all the layers of the neural network as a dictionary along with their outputs
- As defined above, we configure the loss functions
- Then we set the content and style attributes …
- And set gradients and final output function for neural art
- We define the functions to calculate losses and gradients
- Now we take the base image as input and iterate it to get our final image. On my local machine, it takes a minute to get the result in one iteration. According to your resources (and patience), would take at most 5 minutes get the result. You can also increase the number of iterations to further optimize the result.
- And after a long wait, we will get this beautiful image!
NOTE: The code file can be seen on github here.
Where to go from here?
We have seen a small demonstration of a significant discovery in the art world. Many modifications have been made to this method to make it aesthetically pleasing. For instance, I like very much this implementation in which they have taken different styles and applied them to different regions.
The first two images are the masks, that help establish which part should be styled. The next two images represent the styles to be used. The last image is the base image to be styled.
Below is the output generated by neural art.
It looks incredible, ¿no? I'm sure that, like me, you also love to try neural art. To help you get started with it, I have covered the basics of neural art and how you can create your first image. I'm sure you are eager to explore more and, Thus, I am adding some additional resources just for you.
Additional Resources
These are some of the best resources I have found on neural art. Go ahead and enter the fascinating world of neural art.
References
Image sources
[1] https://www.wikiart.org/en/vincent-van-gogh/the-starry-night-1889
[2] https://arxiv.org/abs/1508.06576
[3] Google
Final notes
I hope you found this article inspiring.. Now is the time for you to check it out and make art yourself!! An art is created, share with the community. If you have any question, I would love to interact with you in the comments.. And to get experience working in neural networks, don't forget to try our deep learning practice problem: Identify the digits.
