machine learning, ai, cs · slides · talks & lectures

Deep Learning: Theory, History, State of the Art & Practical Tools

Slide 1 of 122: by Ilya Kuzovkin ilya.kuzovkin@gmail.com Machine Learning Estonia 2016 Deep Learning THEORY, HISTORY, STATE OF THE ART & PRACTICAL TOOLS http://neuro.cs.ut.eeSlide 2 of 122Slide 3 of 122: Where it has started How it evolved What is the state now How can you use it How it learnsSlide 4 of 122: Where it has startedSlide 5 of 122: Where it has started Artificial NeuronSlide 6 of 122: Where it has started Artificial Neuron McCulloch and Pitts “A Logical Calculus of the Ideas Immanent in Nervous Activity” 1943Slide 7 of 122: Where it has started Artificial Neuron McCulloch and Pitts “A Logical Calculus of the Ideas Immanent in Nervous Activity” 1943Slide 8 of 122: Where it has started Perceptron Frank Rosenblatt 1957Slide 9 of 122: Where it has started Perceptron Frank Rosenblatt 1957Slide 10 of 122: Where it has started Perceptron Frank Rosenblatt 1957Slide 11 of 122: Where it has started Perceptron Frank Rosenblatt 1957 “[The Perceptron is] the embryo of an electronic computer that [the Navy] expects will be able to walk, talk, see, write, reproduce itself and be conscious of its existence.” THE NEW YORK TIMESSlide 12 of 122: Where it has started Sigmoid & BackpropagationSlide 13 of 122: Where it has started Sigmoid & BackpropagationSlide 14 of 122: Where it has started Sigmoid & Backpropagation Measure how small changes in weights affect output Can apply NN to regressionSlide 15 of 122: Where it has started Sigmoid & Backpropagation (Werbos) Rumelhart, Hinton, Williams (1974) 1986 “Learning representations by back-propagating errors” (Nature) Measure how small changes in weights affect output Can apply NN to regressionSlide 16 of 122: Where it has started Sigmoid & Backpropagation (Werbos) Rumelhart, Hinton, Williams (1974) 1986 “Learning representations by back-propagating errors” (Nature) Measure how small changes in weights affect output Multilayer neural networks, etc. Can apply NN to regressionSlide 17 of 122: Where it has started Why DL revolution did not happen in 1986? FROM A TALK BY GEOFFREY HINTONSlide 18 of 122: Where it has started Why DL revolution did not happen in 1986? • Not enough data (datasets were 1000 times too small) ! • Computers were too slow (1,000,000 times) FROM A TALK BY GEOFFREY HINTONSlide 19 of 122: Where it has started Why DL revolution did not happen in 1986? • Not enough data (datasets were 1000 times too small) ! • Computers were too slow (1,000,000 times) ! • Not enough attention to network initialization ! • Wrong non-linearity FROM A TALK BY GEOFFREY HINTONSlide 20 of 122: How it learnsSlide 21 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example NET OUTSlide 22 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 NET OUTSlide 23 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Forward Pass — Calculating the total error NET OUTSlide 24 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Forward Pass — Calculating the total error NET OUTSlide 25 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Forward Pass — Calculating the total error NET OUTSlide 26 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Forward Pass — Calculating the total error NET OUTSlide 27 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Forward Pass — Calculating the total error NET OUT Repeat for h2 = 0.596, o1 = 0.751, o2 =…Slide 28 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Forward Pass — Calculating the total error NET OUT We have o1, o2Slide 29 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Forward Pass — Calculating the total error NET OUT We have o1, o2Slide 30 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Forward Pass — Calculating the total error NET OUT We have o1, o2Slide 31 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Forward Pass — Calculating the total error NET OUT We have o1, o2Slide 32 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Forward Pass — Calculating the total error NET OUT We have o1, o2Slide 33 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Backwards Pass — updating weights NET OUTSlide 34 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Backwards Pass — updating weights NET OUTSlide 35 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Backwards Pass — updating weights NET OUTSlide 36 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Backwards Pass — updating weights NET OUTSlide 37 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Backwards Pass — updating weights NET OUTSlide 38 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Backwards Pass — updating weights NET OUTSlide 39 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Backwards Pass — updating weights NET OUTSlide 40 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Backwards Pass — updating weights NET OUTSlide 41 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Backwards Pass — updating weights NET OUTSlide 42 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Backwards Pass — updating weights NET OUTSlide 43 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Backwards Pass — updating weights NET OUTSlide 44 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Backwards Pass — updating weights NET OUTSlide 45 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Backwards Pass — updating weights NET OUTSlide 46 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Backwards Pass — updating weights NET OUTSlide 47 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Backwards Pass — updating weights NET OUTSlide 48 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Backwards Pass — updating weights NET OUTSlide 49 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Backwards Pass — updating weights NET OUTSlide 50 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Backwards Pass — updating weights NET OUT Learning rateSlide 51 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 1. The Backwards Pass — updating weights NET OUT Gradient descent update rule Learning rateSlide 52 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 NET OUTSlide 53 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 NET OUT • Repeat for w6, w7, w8Slide 54 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 NET OUT • Repeat for w6, w7, w8 • In analogous way for w1, w2, w3, w4Slide 55 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 NET OUT • Repeat for w6, w7, w8 • In analogous way for w1, w2, w3, w4 • Calculate the total error…Slide 56 of 122: How it learns Backpropagation http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example Given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99 NET OUT • Repeat for w6, w7, w8 • In analogous way for w1, w2, w3, w4 • Calculate the total error…Slide 57 of 122: How it learns Optimization methods Alec Radford “Introduction to Deep Learning with Python”Slide 58 of 122: How it learns Optimization methods Alec Radford “Introduction to Deep Learning with Python”Slide 59 of 122: How it evolvedSlide 60 of 122: How it evolved 1-layer NN INPUT OUTPUT Alec Radford “Introduction to Deep Learning with Python”Slide 61 of 122: How it evolved 1-layer NN 92.5% on the MNIST test set INPUT OUTPUT Alec Radford “Introduction to Deep Learning with Python”Slide 62 of 122: How it evolved 1-layer NN 92.5% on the MNIST test set INPUT OUTPUT Alec Radford “Introduction to Deep Learning with Python”Slide 63 of 122: How it evolved One hidden layer Alec Radford “Introduction to Deep Learning with Python”Slide 64 of 122: How it evolved One hidden layer 98.2% on the MNIST test set Alec Radford “Introduction to Deep Learning with Python”Slide 65 of 122: How it evolved One hidden layer 98.2% on the MNIST test set Activity of a 100 hidden neurons (out of 625) Alec Radford “Introduction to Deep Learning with Python”Slide 66 of 122: How it evolved OverfittingSlide 67 of 122: How it evolved Dropout Srivastava, Hinton, Krizhevsky, Sutskever, Salakhutdinov, “Dropout: A Simple Way to Prevent Neural Networks from Overfitting”, 2014Slide 68 of 122: How it evolved Dropout Srivastava, Hinton, Krizhevsky, Sutskever, Salakhutdinov, “Dropout: A Simple Way to Prevent Neural Networks from Overfitting”, 2014Slide 69 of 122: How it evolved Dropout Srivastava, Hinton, Krizhevsky, Sutskever, Salakhutdinov, “Dropout: A Simple Way to Prevent Neural Networks from Overfitting”, 2014Slide 70 of 122: How it evolved Dropout Srivastava, Hinton, Krizhevsky, Sutskever, Salakhutdinov, “Dropout: A Simple Way to Prevent Neural Networks from Overfitting”, 2014Slide 71 of 122: How it evolved ReLU X. Glorot, A. Bordes, Y. Bengio, “Deep Sparse Rectifier Neural Networks”, 2011Slide 72 of 122: How it evolved ReLU X. Glorot, A. Bordes, Y. Bengio, “Deep Sparse Rectifier Neural Networks”, 2011Slide 73 of 122: How it evolved “Modern” ANN • Several hidden layers • ReLU activation units • DropoutSlide 74 of 122: How it evolved “Modern” ANN • Several hidden layers • ReLU activation units • DropoutSlide 75 of 122: How it evolved “Modern” ANN • Several hidden layers • ReLU activation units • Dropout 99.0% on the MNIST test setSlide 76 of 122: How it evolved ConvolutionSlide 77 of 122: How it evolved Convolution +1 +1 +1 0 0 0 -1 -1 -1 Prewitt edge detectorSlide 78 of 122: How it evolved Convolution +1 +1 +1 0 0 0 -1 -1 -1 Prewitt edge detectorSlide 79 of 122: How it evolved Convolution +1 +1 +1 0 0 0 -1 -1 -1 Prewitt edge detectorSlide 80 of 122: How it evolved Convolution +1 +1 +1 0 0 0 -1 -1 -1 Prewitt edge detector 40 40 40 40 40 40 40 40 40 10 10 10 10 10 10 10 10 10Slide 81 of 122: How it evolved Convolution +1 +1 +1 0 0 0 -1 -1 -1 Prewitt edge detector +1 +1 +1 0 0 0 -1 -1 -1 40 40 40 40 40 40 40 40 40 10 10 10 10 10 10 10 10 10Slide 82 of 122: How it evolved Convolution +1 +1 +1 0 0 0 -1 -1 -1 Prewitt edge detector +40 +40 +40 0 0 0 -40 -40 -40 10 10 10 10 10 10 10 10 10Slide 83 of 122: How it evolved Convolution +1 +1 +1 0 0 0 -1 -1 -1 Prewitt edge detector 0 +40 +40 +40 0 0 0 -40 -40 -40 10 10 10 10 10 10 10 10 10Slide 84 of 122: How it evolved Convolution +1 +1 +1 0 0 0 -1 -1 -1 Prewitt edge detector 0 90 90 0 40 40 40 40 40 40 40 40 40 10 10 10 10 10 10 10 10 10Slide 85 of 122: How it evolved Convolution +1 +1 +1 0 0 0 -1 -1 -1 Prewitt edge detector 0 90 90 0 40 40 40 40 40 40 40 40 40 10 10 10 10 10 10 10 10 10Slide 86 of 122: How it evolved Convolution +1 +1 +1 0 0 0 -1 -1 -1 Prewitt edge detector 0 90 90 0 40 40 40 40 40 40 40 40 40 10 10 10 10 10 10 10 10 10Slide 87 of 122: How it evolved Convolution +1 +1 +1 0 0 0 -1 -1 -1 Prewitt edge detector 0 90 90 0 40 40 40 40 40 40 40 40 40 10 10 10 10 10 10 10 10 10 Edge detector is a handcrafted feature detector.Slide 88 of 122: How it evolved Convolution The idea of a convolutional layer is to learn feature detectors instead of using handcrafted onesSlide 89 of 122: How it evolved Convolution The idea of a convolutional layer is to learn feature detectors instead of using handcrafted ones http://yann.lecun.com/exdb/mnist/Slide 90 of 122: How it evolved Convolution The idea of a convolutional layer is to learn feature detectors instead of using handcrafted ones 99.50% on the MNIST test set CURRENT BEST: 99.77% by committee of 35 conv. nets http://yann.lecun.com/exdb/mnist/Slide 91 of 122: How it evolved More layersSlide 92 of 122: How it evolved More layers C. Szegedy, et al., “Going Deeper with Convolutions”, 2014Slide 93 of 122: How it evolved More layers C. Szegedy, et al., “Going Deeper with Convolutions”, 2014 ILSVRC 2015 winner — 152 (!) layers K. He et al., “Deep Residual Learning for Image Recognition”, 2015Slide 94 of 122: How it evolved Hyperparameters • Network: • architecture • number of layers • number of units (in each layer) • type of the activation function • weight initialization • Convolutional layers: • size • stride • number of filters • Optimization method: • learning rate • other…Slide 95 of 122: How it evolved Hyperparameters • Network: • architecture • number of layers • number of units (in each layer) • type of the activation function • weight initialization • Convolutional layers: • size • stride • number of filters • Optimization method: • learning rate • other…Slide 96 of 122: How it evolved Hyperparameters • Network: • architecture • number of layers • number of units (in each layer) • type of the activation function • weight initialization • Convolutional layers: • size • stride • number of filters • Optimization method: • learning rate • other…Slide 97 of 122: How it evolved Hyperparameters • Network: • architecture • number of layers • number of units (in each layer) • type of the activation function • weight initialization • Convolutional layers: • size • stride • number of filters • Optimization method: • learning rate • other…Slide 98 of 122: How it evolved Hyperparameters • Network: • architecture • number of layers • number of units (in each layer) • type of the activation function • weight initialization • Convolutional layers: • size • stride • number of filters • Optimization method: • learning rate • other…Slide 99 of 122: How it evolved Major Types of ANNs feedforward convolutionalSlide 100 of 122: How it evolved Major Types of ANNs recurrent feedforward convolutionalSlide 101 of 122: How it evolved Major Types of ANNs recurrent autoencoder feedforward convolutionalSlide 102 of 122: What is the state nowSlide 103 of 122: What is the state now Computer vision http://cs.stanford.edu/people/karpathy/deepimagesent/ Kaiming He, et al. “Deep Residual Learning for Image Recognition” 2015Slide 104 of 122: What is the state now Natural Language Processing speech recognition + translation Facebook bAbi dataset: question answering http://smerity.com/articles/2015/keras_qa.html http://karpathy.github.io/2015/05/21/rnn-effectiveness/Slide 105 of 122: What is the state now AI DeepMind’s DQNSlide 106 of 122: What is the state now AI Sukhbaatar et al. “MazeBase: A Sandbox for Learning from Games”, 2015 DeepMind’s DQNSlide 107 of 122: What is the state now Neuroscience Güclü and Gerven, “Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Ventral Stream”, 2015Slide 108 of 122: What is the state now Neuroscience Güclü and Gerven, “Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Ventral Stream”, 2015Slide 109 of 122: How can you use itSlide 110 of 122: How can you use it Pre-trained modelsSlide 111 of 122: How can you use it Pre-trained models • Go to https://github.com/BVLC/caffe/wiki/Model-Zoo, pick a model ! ! ! ! • … and use it in your applicationSlide 112 of 122: How can you use it Pre-trained models • Go to https://github.com/BVLC/caffe/wiki/Model-Zoo, pick a model ! ! ! ! • … and use it in your application • Or … ! ! !Slide 113 of 122: How can you use it Pre-trained models • Go to https://github.com/BVLC/caffe/wiki/Model-Zoo, pick a model ! ! ! ! • … and use it in your application • Or … ! ! ! ! • … use part of it as the starting point for your model . . .Slide 114 of 122: How can you use it Zoo of Frameworks Low-level High-level & WrappersSlide 115 of 122: How can you use it Keras http://keras.ioSlide 116 of 122: How can you use it Keras http://keras.ioSlide 117 of 122: How can you use it Keras http://keras.ioSlide 118 of 122: How can you use it Keras http://keras.ioSlide 119 of 122: How can you use it Keras http://keras.ioSlide 120 of 122: How can you use it Keras http://keras.ioSlide 121 of 122: How can you use it Keras http://keras.ioSlide 122 of 122: • A Step by Step Backpropagation Example http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example • Online book by By Michael Nielsen http://neuralnetworksanddeeplearning.com • CS231n: Convolutional Neural Networks for Visual Recognition http://cs231n.stanford.edu/
1 / 122

Deep Learning: Theory, History, State of the Art & Practical Tools

An introductory talk about deep learning given at the Machine Learning Estonia meetup.

Deep Learning from AS Proekspert on Vimeo.

Download PDF