HPC Opportunities in Deep Learning- Greg Diamos
-
Upload
nvidia -
Category
Technology
-
view
720 -
download
0
Transcript of HPC Opportunities in Deep Learning- Greg Diamos
![Page 1: HPC Opportunities in Deep Learning- Greg Diamos](https://reader036.fdocuments.us/reader036/viewer/2022081520/58f18e3f1a28ab17098b4629/html5/thumbnails/1.jpg)
HPC Opportunities In Deep LearningGreg Diamos, SC16
![Page 2: HPC Opportunities in Deep Learning- Greg Diamos](https://reader036.fdocuments.us/reader036/viewer/2022081520/58f18e3f1a28ab17098b4629/html5/thumbnails/2.jpg)
AN OVERVIEW …
1. Why is Deep Learning Important Now in HPC?2. Published Results with ImageNet, Google DeepMind,
Baidu AI Lab.3. Getting Started with Deep Learning in HPC.4. Future Direction and Opportunities for Growth with
HPC and Deep Learning.
Source: Greg Diamos SC16 Talk
![Page 3: HPC Opportunities in Deep Learning- Greg Diamos](https://reader036.fdocuments.us/reader036/viewer/2022081520/58f18e3f1a28ab17098b4629/html5/thumbnails/3.jpg)
Before, we had no idea how to train neural networks. The prevailing opinion, at the time, was that they were impossible to train.
Image Source: NVIDIA
Why is Deep Learning Important Now in HPC?
But now, we have powerful tools that can start applying to problem after problem and making progress on those that are really incredible inherently difficult.Content Source: Greg Diamos SC16 Talk
![Page 4: HPC Opportunities in Deep Learning- Greg Diamos](https://reader036.fdocuments.us/reader036/viewer/2022081520/58f18e3f1a28ab17098b4629/html5/thumbnails/4.jpg)
THE PUBLISHED EVIDENCE SPEAKS FOR ITSELF…
![Page 5: HPC Opportunities in Deep Learning- Greg Diamos](https://reader036.fdocuments.us/reader036/viewer/2022081520/58f18e3f1a28ab17098b4629/html5/thumbnails/5.jpg)
The ImageNet Challenge We first found success in the ImageNet challenge, in which ImageNet was given images and had to produce a corresponding label. The challenge encompassed over a very large dataset of images and then classified into a thousand different categories. We’ve approached human-level accuracy with deep learning algorithms for these systems. Content Source: Greg Diamos SC16 Talk Image Source: NVIDIA, Greg Diamos SC16
Talk
![Page 6: HPC Opportunities in Deep Learning- Greg Diamos](https://reader036.fdocuments.us/reader036/viewer/2022081520/58f18e3f1a28ab17098b4629/html5/thumbnails/6.jpg)
THIS PROGRESS ONLY CONTINUED GROWING EXPONENTIALLY…
![Page 7: HPC Opportunities in Deep Learning- Greg Diamos](https://reader036.fdocuments.us/reader036/viewer/2022081520/58f18e3f1a28ab17098b4629/html5/thumbnails/7.jpg)
DeepMind at Google Just last year, a deep neural network defeated one of the best players, the best human players in a game of ‘Go.’ This is a game with an absolutely enormous optimization space. There’s no way to search over all possible combinations. Content Source: Greg Diamos SC16 Talk
Image Source: Greg Diamos SC16 TalkImage Source
![Page 8: HPC Opportunities in Deep Learning- Greg Diamos](https://reader036.fdocuments.us/reader036/viewer/2022081520/58f18e3f1a28ab17098b4629/html5/thumbnails/8.jpg)
Baidu’s AI LabAt our lab, we can approach human-level accuracy on many test sets.For example, when you build a speech recognition system, you would hand-design all of these components. You would not have one neural network. You would have five or six components all hand-designed by linguists, speech, signal processing, and mathematicians.
Content Source: Greg Diamos SC16 Talk
Image Source
Image Source
We cut all that out.
![Page 9: HPC Opportunities in Deep Learning- Greg Diamos](https://reader036.fdocuments.us/reader036/viewer/2022081520/58f18e3f1a28ab17098b4629/html5/thumbnails/9.jpg)
Image Source
Content Source: Greg Diamos SC16 Talk
Baidu’s AI Lab Cont. We can now take a team of five people who don’t speak any Mandarin, and produce a speech recognition system that beats all of the existing systems that we have and actually does better than a human grader.
But these things are incredibly computationally intensive to train.
![Page 10: HPC Opportunities in Deep Learning- Greg Diamos](https://reader036.fdocuments.us/reader036/viewer/2022081520/58f18e3f1a28ab17098b4629/html5/thumbnails/10.jpg)
AND WE’RE NOW GETTING INTO THE RELATIONSHIP BETWEEN DEEP LEARNING AND HIGH PERFORMANCE COMPUTING SYSTEMS.
![Page 11: HPC Opportunities in Deep Learning- Greg Diamos](https://reader036.fdocuments.us/reader036/viewer/2022081520/58f18e3f1a28ab17098b4629/html5/thumbnails/11.jpg)
GETTING STARTED WITH DEEP LEARNING IN HPC
1.Big Model2.Big Data3.Big Computer
Source: Greg Diamos SC16 Talk
What do you need in order to get started solving a new problem that you want to apply deep learning to? There are three simple, but high-level factors:
![Page 12: HPC Opportunities in Deep Learning- Greg Diamos](https://reader036.fdocuments.us/reader036/viewer/2022081520/58f18e3f1a28ab17098b4629/html5/thumbnails/12.jpg)
1. Big Model
Image Source
First, you need a big model. Your model has to be able to approximate the function that you’re trying to represent. For example, the function that maps images to text is complicated. Many parameters are needed to actually represent it. The model must be big in order to capture a extremely intricate function. Content Source: Greg Diamos SC16 Talk
![Page 13: HPC Opportunities in Deep Learning- Greg Diamos](https://reader036.fdocuments.us/reader036/viewer/2022081520/58f18e3f1a28ab17098b4629/html5/thumbnails/13.jpg)
2. Big Data
Image Source
Deep Learning doesn’t perform very well with small datasets. This was another reason as to why people might not have thought Deep Learning was important before. Deep Learning on smaller datasets would easily be beaten out by simpler, more explicit methods. But, as the datasets get larger, they start to surpass other methods.”
Content Source: Greg Diamos SC16 Talk
![Page 14: HPC Opportunities in Deep Learning- Greg Diamos](https://reader036.fdocuments.us/reader036/viewer/2022081520/58f18e3f1a28ab17098b4629/html5/thumbnails/14.jpg)
3. Big Computer
Image Source
And when you have a big network and big data, you need a powerful supercomputer to run it. If you don’t have a fast enough computer, one can be stuck waiting years or decades for a result. So we come to this need for speed. And this is really the most important point in the talk.
Content Source: Greg Diamos SC16 Talk
![Page 15: HPC Opportunities in Deep Learning- Greg Diamos](https://reader036.fdocuments.us/reader036/viewer/2022081520/58f18e3f1a28ab17098b4629/html5/thumbnails/15.jpg)
WHAT ARE THE OPPORTUNITIES OF GROWTH FOR HPC IN DEEP LEARNING?
![Page 16: HPC Opportunities in Deep Learning- Greg Diamos](https://reader036.fdocuments.us/reader036/viewer/2022081520/58f18e3f1a28ab17098b4629/html5/thumbnails/16.jpg)
Opportunities For Growth
Image Source
First, we need to figure out a way of scaling up models. Currently, the biggest model that runs at a high percent of efficiency is about 100 processors, which is large from a machine learning perspective but it is small from an HPC perspective. Second, we are far away from the power limit in CMOS. Right now, we’re around ten teraflops per processor. I think we can get to 20 petaflops before we hit the power limit. You can make progress on speech, vision, and language problems by making faster computers. Content Source: Greg Diamos SC16 Talk
![Page 17: HPC Opportunities in Deep Learning- Greg Diamos](https://reader036.fdocuments.us/reader036/viewer/2022081520/58f18e3f1a28ab17098b4629/html5/thumbnails/17.jpg)
Image Source
Future Directions of HPC and Deep LearningThe two big directions that we see are around speech powered interfaces and self-driving cars. Speech powered interfaces are really three different components: recognition, human-level accuracy, and computer generation. Self-driving cars are also highly valuable as they leverage a lot of vision technology that’s already been developed.Both areas are significant directions going forward, but there are definitely even more application beyond these that are close to becoming possible using deep learning.Content Source: Greg Diamos SC16 Talk
Image Source
![Page 18: HPC Opportunities in Deep Learning- Greg Diamos](https://reader036.fdocuments.us/reader036/viewer/2022081520/58f18e3f1a28ab17098b4629/html5/thumbnails/18.jpg)
About the Speaker: Greg DiamosGreg Diamos is a senior researcher at Baidu’s Silicon Valley AI Lab (SVAIL). Previously, he was on the research team at NVIDIA. Greg holds a PhD from the Georgia Institute of Technology, where he contributed to the development of the GPU-Ocelot dynamic compiler, which targeted CPUs and GPUs from the same program representation.
FOR THE FULL RECORDING: WATCH HERE
![Page 19: HPC Opportunities in Deep Learning- Greg Diamos](https://reader036.fdocuments.us/reader036/viewer/2022081520/58f18e3f1a28ab17098b4629/html5/thumbnails/19.jpg)
LEARN MORE ABOUT THE INTERSECTION OF AI AND HPC
INSIDEBIGDATA GUIDE