Since the last decade, we've seen the GPU come into the picture more often in fields like HPC (High performance computing) and the most popular field, namely, the games. GPUs have improved year over year and are now capable of doing incredibly cool things, but in recent years they have garnered even more attention due to deep learning.
How deep learning models spend a great deal of time training, even powerful CPUs weren't efficient enough to handle so many calculations at any given time and this is the area where GPUs simply outperformed CPUs due to their parallelism. But before we dive deep, let's first understand a few things about the GPU.
What is the GPU?
A GPU or 'Graphics Processing Unit’ it is a mini version of a full computer, but only dedicated to a specific task. It is different from a CPU that performs multiple tasks at the same time. The GPU comes with its own processor which is built into its own motherboard along with v-ram or video ram, and also a suitable thermal design for ventilation and cooling.
In the term 'Graphics Processing Unit', 'Graphics’ refers to rendering an image at specific coordinates in 2d or 3D space. A window or point of view is the perspective of a spectator to look at an object according to the type of projection used. Rasterization and ray tracing are some of the ways to render 3D scenes, both concepts are based on a type of projection called perspective projection. What is perspective projection?
In summary, is the way an image is formed on a view plane or canvas where parallel lines converge to a convergent point called ‘projection center’ also as the object moves away from the point of view it appears to be smaller , exactly how our eyes are portrayed in the real world and this also helps to understand the depth of an image, that's the reason why it produces realistic images.
What's more, GPUs also process complex geometry, vector, light sources or illuminations, textures, shapes, etc. As we now have a basic idea about the GPU, let's understand why it is used a lot for deep learning.
Why are GPUs better for deep learning?
One of the most admired features of a GPU is the ability to compute processes in parallel. This is the point where the concept of parallel computing go into action. A CPU generally completes its task sequentially. A CPU can be divided into cores and each core performs one task at a time. Suppose a CPU has 2 cores. Later, two different task processes can run on these two cores, thus achieving multitasking.
But still, these processes run serially.
This does not mean that the CPUs are not good enough. In fact, CPUs are really good at handling different tasks related to different operations like handling operating systems, deliver spreadsheets, play HD videos, extract large zip files, All at the same time. These are some of the things a GPU just can't do.
Where is the difference?
As mentioned earlier, a CPU is divided into multiple cores so they can multitask at the same time, while the GPU will have hundreds and thousands of cores, all of which are dedicated to a single task. These are simple calculations that are performed more frequently and are independent of each other.. And both store frequently required data in their respective cache, thus following the principle of ‘locality reference‘.
There are many programs and games that can take advantage of GPUs to run. The idea behind this is to make some parts of the task or application code parallel, but not all processes. This is because most of the task processes only need to run sequentially. For instance, logging into a system or application does not need to be parallel.
When there is part of the execution that can be done in parallel, just switch to GPU for processing, where at the same time the sequential task is executed on the CPU, then both parts of the task are combined again.
In the GPU market, there are two main players, namely, AMD and Nvidia. Nvidia GPUs are widely used for deep learning because they are widely supported in the forum software, Controllers, CUDA and cuDNN. Then, in terms of artificial intelligence and deep learning, Nvidia is a pioneer for a long time.
Neural networks are said to be embarrassingly parallel, which means that calculations in neural networks can be easily executed in parallel and are independent of each other.
Some calculations such as the calculation of weights and activation functions of each layer, backpropagation can be done in parallel. There are also many research articles available on this..
Nvidia GPUs come with specialized cores known as MIRACLES cores that help accelerate deep learning.
What is CUDA?
CUDA significa ‘Compute Unified Device Architecture’ which was launched in the year 2007, it is a way that you can achieve parallel computing and get the most out of your GPU power in an optimized way, resulting in much better performance when executing tasks.
The CUDA Toolkit is a complete package consisting of a development environment that is used to create applications that use GPUs.. This toolkit mainly contains the compiler, the debugger and c libraries / c ++. What's more, the CUDA runtime has its drivers so it can communicate with the GPU. CUDA is also a programming language that is specifically designed to instruct the GPU to perform a task.. Also known as GPU scheduling.
Below is a simple hello world program just to get an idea of what the CUDA code looks like.
/* hello world program in cuda * #include<stdio.h> #include<stdlib.h> #include<cuda.h>__global__ void demo() { printf("hello world!,my first cuda program"); }int main() { printf("From main!n"); demo<<<1,1>>>(); return 0; }
What is cuDNN?
cuDNN is a neural network library that is GPU optimized and can take full advantage of the Nvidia GPU. This library consists of the convolution implementation, forward and backward spread, activation and grouping functions. It is a must have library without which you cannot use GPU to train neural networks.
A big jump with the Tensor cores!!
In the year 2018, Nvidia launched a new line of its GPUs, namely, the Serie 2000. Also called RTX, These cards come with Tensor cores dedicated to deep learning and based on the Volta architecture.
Tensor nuclei are particular nuclei that perform matrix multiplication 4 x 4 FP16 and the sum with 4 x 4 FP16 or FP32 matrix in medium precision, the output will result in an array 4 x 4 FP16 or FP32 with full precision.
Note: ‘FP’ means floating point to understand more about floating point and precision, check this Blog.
As stated by Nvidia, new generation tensor cores based on volta architecture are much faster than CUDA cores based on Pascal architecture. This gave a huge boost to deep learning.
At the time of writing this blog, Nvidia announced the latest series 3000 of its GPU lineup that comes with Ampere architecture. In this, improved performance of tensor cores by 2x. It also brings new precision values like TF32 (tensor float 32), FP64 (floating point 64). The TF32 works the same as the FP32 but with up to 20x acceleration, as a result of all this, Nvidia claims model training or inference time will be reduced from weeks to hours.
AMD vs Nvidia
AMD GPUs are decent for gaming, but as soon as deep learning kicks in, simply Nvidia is way ahead. Doesn't mean AMD GPUs are bad. It is due to software optimization and drivers not being actively updated, on the Nvidia side they have better drivers with frequent updates and on top of that CUDA, cuDNN helps speed up the calculation.
Some libraries known as Tensorflow, PyTorch support for CUDA. It means entry-level GPUs from the GTX series can be used 1000. On the AMD side, has very little software support for its GPUs. On the hardware side, Nvidia has introduced dedicated tensioner cores. AMD has ROCm for acceleration, but not good as tensor nuclei, and many deep learning libraries are not compatible with ROCm. During the last years, no big leap was noticed in terms of performance.
Due to all these points, Nvidia simply excels at deep learning.
Summary
To conclude from all that we have learned, it is clear that as of now Nvidia is the market leader in terms of GPU, but I really hope that even AMD will catch up in the future or at least make some notable improvements to the next line of their GPUs.. since they are already doing a great job regarding their CPUs, namely, ryzen series.
The reach of GPUs in the coming years is huge as we make new innovations and advancements in deep learning, machine learning and HPC. GPU acceleration will always be helpful for many developers and students to enter this field, as their prices are also becoming more affordable. Also thanks to the broad community that also contributes to the development of AI and HPC.
Prathmesh Patil
Machine learning enthusiast, data science, python developer.
LinkedIn: https://www.linkedin.com/in/prathmesh
