machine learning, ai, cs · slides

Paper overview | Unsupervised Learning of Visual Structure Using Predictive Generative Networks

Slide 1 of 45: Article overview by Ilya Kuzovkin William Lotter, Gabriel Kreiman & David Cox Computational Neuroscience Seminar University of Tartu 2015 Harvard University, Cambridge, USA Unsupervised Learning of Visual Structure Using Predictive Generative NetworksSlide 2 of 45: The idea of predictive coding in neuroscienceSlide 3 of 45: “state-of-the-art deep learning models rely on millions of labeled training examples to learn”Slide 4 of 45: “state-of-the-art deep learning models rely on millions of labeled training examples to learn” “in contrast to biological systems, where learning is largely unsupervised”Slide 5 of 45: “state-of-the-art deep learning models rely on millions of labeled training examples to learn” “in contrast to biological systems, where learning is largely unsupervised” “we explore the idea that prediction is not only a useful end-goal, but may also serve as a powerful…Slide 6 of 45: PART I THE IDEA OF PREDICTIVE ENCODER "prediction may also serve as a powerful unsupervised learning signal"Slide 7 of 45: PREDICTIVE GENERATIVE NETWORK (a.k.a “Predictive Encoder” Palm 2012) vs.Slide 8 of 45: input output “bottleneck” AUTOENCODERSlide 9 of 45: input output “bottleneck” AUTOENCODERSlide 10 of 45: input output “bottleneck” AUTOENCODERSlide 11 of 45: input output “bottleneck” Reconstruction AUTOENCODERSlide 12 of 45: input output “bottleneck” Reconstruction AUTOENCODERSlide 13 of 45: input output “bottleneck” Can we do prediction? Reconstruction AUTOENCODERSlide 14 of 45: PREDICTIVE GENERATIVE NETWORK (a.k.a “Predictive Encoder” Palm 2012) vs.Slide 15 of 45: RECURRENT NEURAL NETWORKSlide 16 of 45: RECURRENT NEURAL NETWORKSlide 17 of 45: RECURRENT NEURAL NETWORKSlide 18 of 45: RECURRENT NEURAL NETWORKSlide 19 of 45: PREDICTIVE GENERATIVE NETWORK (a.k.a “Predictive Encoder” Palm 2012) vs.Slide 20 of 45: PREDICTIVE GENERATIVE NETWORK (a.k.a “Predictive Encoder” Palm 2012) vs. Convolution ReLu Max-pooling 2x {Slide 21 of 45: PREDICTIVE GENERATIVE NETWORK (a.k.a “Predictive Encoder” Palm 2012) vs. Long Short-Term Memory (LSTM) 5 - 15 steps 1024 units Convolution ReLu Max-pooling 2x {Slide 22 of 45: PREDICTIVE GENERATIVE NETWORK (a.k.a “Predictive Encoder” Palm 2012) vs. Long Short-Term Memory (LSTM) 5 - 15 steps 1024 units Convolution ReLu Max-pooling 2x { 2 layers NN upsampling ReLu ConvolutionSlide 23 of 45: PREDICTIVE GENERATIVE NETWORK (a.k.a “Predictive Encoder” Palm 2012) vs. Long Short-Term Memory (LSTM) 5 - 15 steps 1024 units Convolution ReLu Max-pooling 2x { MSE loss RMSProp optimizer LR 0.001 2 layers NN upsampling ReLu ConvolutionSlide 24 of 45: PREDICTIVE GENERATIVE NETWORK (a.k.a “Predictive Encoder” Palm 2012) vs. Long Short-Term Memory (LSTM) 5 - 15 steps 1024 units http://keras.io 2 layers NN upsampling Convolution ReLu ReLu Convolution Max-pooling 2x { MSE loss RMSProp optimizer LR 0.001Slide 25 of 45Slide 26 of 45: PART II ADVERSARIAL LOSS "the generator is trained to maximally confuse the adversarial discriminator"Slide 27 of 45Slide 28 of 45: vs. Long Short-Term Memory (LSTM) 5 - 15 steps 1568 units Fully connected layer 2 layers NN upsampling Convolution ReLu ReLu Convolution Max-pooling 2x { MSE loss RMSProp optimizer LR 0.001Slide 29 of 45: vs. Long Short-Term Memory (LSTM) 5 - 15 steps 1568 units Fully connected layer 2 layers NN upsampling Convolution ReLu ReLu Convolution Max-pooling 2x { MSE loss RMSProp optimizer LR 0.001Slide 30 of 45: MSE lossSlide 31 of 45: MSE lossSlide 32 of 45: MSE lossSlide 33 of 45: 3 FC layers (relu, relu, softmax) MSE lossSlide 34 of 45: 3 FC layers (relu, relu, softmax) "trained to maximize the probability that a proposed frame came from the ground truth data and minimize it when it is produced by the generator" MSE lossSlide 35 of 45: 3 FC layers (relu, relu, softmax) "trained to maximize the probability that a proposed frame came from the ground truth data and minimize it when it is produced by the generator" AL loss to train PGN MSE loss AL lossSlide 36 of 45: 3 FC layers (relu, relu, softmax) "trained to maximize the probability that a proposed frame came from the ground truth data and minimize it when it is produced by the generator" AL loss to train PGN MSE loss AL lossSlide 37 of 45: “with adversarial loss alone the generator easily found solutions that fooled the discriminator, but did not look anything like the correct samples” MSE model is fairly faithful to the identities of the faces, but produces blurred versions combined AL/MSE model tends to underfit…Slide 38 of 45: PART III INTERNAL REPRESENTATIONS AND LATENT VARIABLES "we are interested in understanding the representations learned by the models"Slide 39 of 45: PGN model LSTM activities L2 regression Value of a latent variableSlide 40 of 45: PGN model LSTM activities L2 regression Value of a latent variableSlide 41 of 45: “ An MDS algorithm aims to place each object in N-dimensional space such that the between-object distances are preserved as well as possible. ” MULTIDIMENSIONAL SCALINGSlide 42 of 45: PART IV USEFULNESS OF PREDICTIVE LEARNING "representations trained with a predictive loss outperform other models of comparable complexity in a supervised classification problem "Slide 43 of 45: THE TASK: 50 randomly generated faces (12 angles per each) Generative models: Internal representation SVM Identify classSlide 44 of 45: THE TASK: 50 randomly generated faces (12 angles per each) Generative models: Internal representation SVM Identify class • Encoder-LSTM-Decoder to predict next frame (PGN) • Encoder-LSTM-Decoder to predict last frame (AE LSTM dynamic) • Encoder-LSTM-Decoder on frames made into…Slide 45 of 45
1 / 45

Article overview: Unsupervised Learning of Visual Structure Using Predictive Generative Networks

This set of slides goes over the recent article that tries to tie together the idea of predictive coding and deep learning. The main point of the article is that a generative system trained on sequential data to predict future samples learns a more “useful” representation than a usual autoencoder. The result resonates with the idea that our brain probably uses predictive mechanisms.

Download PDF