the importance of GPUs. GPUs are a powerful tool for training deep nets, and nearly every

software library supports them. But when it comes to speeding up the training process,

there are several alternatives to GPUs that are worth considering. Let’s take a closer

look.

The CPU in your computer is capable of performing many different tasks across a wide variety

of domains. But this versatility comes at a cost – CPUs require sophisticated control

mechanisms in order to manage the flow of tasks. The CPU is also designed to perform

tasks serially – one after another – rather than in parallel. Parallelism can also be

achieved by building in a limited number of cores directly into the CPU.

These cores are also versatile, but they need to be created with general-purpose computing in mind. You

may have also noticed that CPU clock speeds haven’t improved much over the last few

years, even though there have been some minor improvements with CPU memory. Since training

a deep net requires so many computational resources, a CPU is impractical for large-scale

deep nets.

So if a lone CPU isn’t powerful enough for the job, what can we use to train a deep net

in a reasonable time window? There are a few tricks we can use, one of which is to implement

deep nets using vectors. Vector algebra – like addition, dot products, and transposes – are

all operations that can be performed in parallel. Take the dot product for example…Each multiplication

step can be performed in parallel, and the resulting products can then be added together.

Through the use of a parallel implementation, deep nets can be trained orders of magnitude

faster. Parallelism implemented at the hardware level is known as parallel processing, and

parallelism at the software level is parallel programming.

Parallel processing can be broken down into two general categories – shared memory,

and distributed computing. Let’s start by looking at a few shared memory options.

The first option is the GPU, a popular tool in the world of deep learning. Unlike a CPU,

where the number of built-in cores is typically in the single or double digits, GPUs implement

100s and sometimes even 1000s of cores. Each GPU core is versatile, and capable of general-purpose

parallel computing. Any task that can be implemented in parallel, can be performed on a GPU. With

regards to deep nets, the most popular application for GPUs is the training process. The Deep

Learning community provides great support for GPUs through libraries, implementations,

and a vibrant ecosystem fostered by nVidia. Despite all their advantages, GPUs do come

with one big drawback. Their versatility and general-purpose design leads to extremely

high power consumption. This becomes a significant issue for large scale deep nets, like the

ones that are used by the tech giants.

One alternative to the GPU is the “Field Programmable Gate Array”, or FPGA. FPGAs

are highly configurable, and they were originally used by electrical engineers to build mock-ups

of different chip designs…that way the engineers could test different solutions to a given

problem, without having to actually design a chip each time. FPGAs allow you to tweak

the chip’s function at the lowest level, which is the logic gate. So an FPGA can be

tailored specifically for a deep net application, allowing them to consume much less power than

a GPU. But there’s an additional benefit, since FPGAs can be used to run a deep net

model and generate predictions. This would come in handy if, for example, you needed

to run a large-scale convolutional net across 1000s of images per second. So FPGAs are a

great tool, but their big strength…that is, their configurability…can also be somewhat

of a weakness. To properly setup and configure an FPGA, an engineer would need highly-specialized

knowledge in digital and integrated circuit design.

Another option is an “Application Specific Integrated Circuit”, or ASIC. ASICs are

highly specialized with designs built in at the hardware and integrated circuit level.

Once built they will perform very well at the task they were designed for but are generally

unusable at any other tasks. Compared to GPUs and FGPAs, ASICs tend to have the lowest power

consumption requirements. There are several Deep Learning ASICs such as the Google Tensor

Processing Unit TPU, and the chip being built by Nervana Systems.

Aside from shared memory, parallelism can also be implemented using distributed computing.

Generally speaking, the three options for distributed computing are data parallelism,

model parallelism, and pipeline parallelism.

Data parallelism allows you to train different subsets of the data on different nodes in

a cluster for each training pass. This is followed by parameter averaging and replacement

across the cluster. We saw model parallelism with TensorFlow, where different portions

of the model are trained on different devices in parallel.

Pipeline Parallelism works like a production assembly line. Generally, there will be a

number of jobs to be completed, each of which can be broken up into independent tasks. Each

task for a given job will be dedicated to a worker, ensuring that each worker is relatively

well-utilized. When a worker finishes its task, it can move on to a task for another

job down the line, even if the other workers are still working on the current job. Here

is an example of a job involving 4 tasks, each of which is dedicated to a worker. When

worker 1 finishes task 1 for the first job, worker 1 can start working on a task for job

2. Worker 2 may still be working on task 2 for job 1, and when worker 2 finishes and

moves to job 2, worker 3 may still be working on task 3 for job 1, and so on. Even though

this is a bit simplified and processing times can be variable in practice, this example

should illustrate the concept of pipeline parallelism.

Computer scientists have been researching parallel programming for decades, and in that

time they’ve developed a set of advanced techniques. Most of these are beyond the scope

of this video, but the main idea is that designing algorithms with parallelism in mind will allow

you to take full advantage of the parallelism capabilities of the hardware. Let’s look

at three general ways to parallelize your code – note that this is an extensive area

of computer science, so we are not providing an exhaustive list.

The first method is to decompose your data model into chunks, where each chunk is needed

to perform an instance of a task. In this example, we see a data table where each row

represents a chunk of data that is independent from the others. By organizing your data in

this manner, each row can be used as an input in parallel.

The second method is to identify tasks that have dependencies, and place them into a single

group. By creating multiple groups that have no dependencies on one another, you can process

the final job in parallel by dividing up the groups.

The third method is to implement threads and processes that handle different tasks or task

groups. This method can be performed independently, but the performance benefits can be significant

when combined with the second method.

If you want to learn more about this topic, a great resource is the Open HPI Massive Open

Online Course on Parallel Programming.

Hopefully by now, you have a better understanding of the available options for training deep

nets in parallel. Next up, we’ll take a look at the use of deep neural networks for

Text Analytics.