Notes on NIPS 2016
The Neural Information Processing Systems (NIPS) conference is a place where computational neuroscience meets machine learning. Due to the rise of deep learning (DL) in recent years, the conference is now flooded with presentations on deep anything, while the brain part is left somewhat behind, although some presentations here and there draw the analogies and several workshops try to reviveThanks! the tradition.

Notes are organized into five topics coinciding with the major trends represented at the conference:
- General Deep Learning
- Reinforcement Learning
- Neuroscience and ML
- Learning to Learn
- Non-Deep Learning
Let’s begin!
General Deep Learning
tutorial Tutorial: Building Applications using Deep Learning by Andrew Ng
A presentation about practical tips on applying deep learning and ways to think about the ML experimental pipeline, evaluation measures, and the concepts of bias and variance in the era of human-level performance. Content-wise the same as https://www.youtube.com/watch?v=F1ka6a13S9I.
tutorial Tutorial: Generative Adversarial Networks by Ian Goodfellow
Introduction to Generative Adversarial Networks (GAN) and DCGAN Architecture. Demonstration of semantic algebra operations in image space (picture on the right). Noise Contrastive Estimation (NCE) is a similar idea
from the past. On regularization: weight decay affects training accuracy of a network, label smoothing works better for regularization with GANs, batch normalization introduces correlation within a batch, reference batch normalization and virtual batch normalization to avoid that problem. Minibatch GAN generates more diverse samples. Extra discriminator outputs: instead of the usual binary “real” / “fake”, a discriminator is trained to distinguish “cat” / “dog” / “fake” — enters the domain of semi-supervised learning. GANs can be used for imitation learning. Plug and Play Generative Networks are the state of the art in terms of the quality of generated images (see picture on the left).
keynote Keynote: Predictive Learning (by Yann LeCun)
A general talk stating results from last year(s). Cool-looking SharpMask code for image segmentation.

talk Award: Value Iteration Networks
Value iteration is an old algorithm to estimate V*. The idea of the article is to add a learnable value iteration module into a neural network, giving it the ability to learn planning without having a model of the world and thus effectively being a model-free method, but with planning. Trained a trivial agent in a grid world that has to reach the end location from the start location. The architecture presumably learns the concept of planning instead of memorizing the path from start location to end location.
talk Session: Deep Learning
| Ba, Hinton, Mnih Using Fast Weights to Attend to the Recent Past Using two weights for each connection in RNNs: slow for long-term information, fast for rapid learning. Fast associative weights act as a Hopfield network by attracting a state most similar to the current input. Performs 4% better than other sequential models (including LSTM) on a recursive vision task (worse than usual convnet though) of face emotion classification. | Sequential Neural Models with Stochastic Layers code Stochastic transitions allow modeling uncertainty. Combining advantages of RNNs and SSMs. | Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences Extremely long sequences (10k time steps). In many applications that involve long sequences, we would like to exploit all of that data, but since sequences are so long we subsample/average/etc. They add a “kronos” k gate to the LSTM unit that controls when the cell value can be updated. Outperforms usual LSTMs in high-resolution sampling and asynchronous sampling conditions (synthetic data). |
talk Weight normalization
Represent each weight vector as a vector and a scalar as follows: and use the usual stochastic gradient descent over the new parameters and . The results show that this approach speeds up training of a neural network in various architectures, does not introduce noise like batch normalization does, and thus can be used with LSTMs and in the reinforcement learning domain (where extra noise can cause trouble).
talk Differentiable Neural Computer by Alex Graves
Presentation about the quite famous Nature article by DeepMind. It is a descendant from Neural Turing Machines and continues on the idea of adding memory to a neural network. Unlike LSTMs, this memory is an actual read-write device, and the neural network learns from data how to use it, what to write and where to read. Personally, I think this is fascinating work, however with no practical applications at its current level of development. DeepMind has promised to publish the code in April 2017.
talk Credit Assignment Beyond Backpropagation by Yoshua Bengio
There are several algorithms for credit assignment: Boltzmann machines — high variance; REINFORCE algorithm — very high variance, does not scale, susceptible to noise; Actor-Critic models — lower variance, but potentially high bias; Backpropagation — wins over all previous algorithms as it considers only one direction of update and does not waste time exploring other possible directions. However, backprop has limitations as well: with increasing depth of deep neural networks it becomes harder and harder to train them (with the exception of ResNet which is a nice trick), small changes in the output do not always reach the place that has to be changed to correct for them. In this talk he proposes a novel concept of equilibrium propagation: given a real physical system (the brain, for example) that we can observe, we perform a pull on the output away from the “correct” energy minimum and observe which changes in the system it leads to. We will notice the places that were most responsive to the pull we made, and this way we will know which places (parameters) of the system must be changed to influence that particular output. The very nice property is that if we can monitor the system via sufficient statistics, then we can alter it even without knowing how exactly the system works. The talk outlines the mathematical framework which allows reasoning about this method. Limited by a requirement that there is an energy function (so that we can pull the output away from energy minima to introduce perturbations).
Reinforcement Learning and Robotics
Recent successes of deep reinforcement learning (DRL), such as Atari, AlphaGo, etc., created a wave of interest towards that field, an explosion in the number of proposed deep RL architectures and the strive to solve the ultimatein my opinion goal of RL — control of a real agent in a real environment, which is an open problem due to the high sample complexity of deep RL algorithms.
tutorial Tutorial: Deep Reinforcement Learning through Policy Optimization
- One of the main obstacles for applying RL in the real world is sample complexity.
- An interesting difference between supervised learning and RL: in supervised learning the training data distribution is fixed, while in RL a change in policy leads to sampling from another distribution, and too big a step in the wrong direction might break everything.
talk Learning to poke by poking
A robotic hand is trained with a reinforcement learning algorithm to poke objects in such a way that eventually they end up in a required position. Trained in the physical world. The novel idea is to simultaneously optimize the forward prediction generative model (given a state and an action, what will be the next state?) and the inverse model (given the current state s and the next state s’ what is the action to be taken to get from s to s’?). Learning such a joint model improves performance and decreases sample complexity.
concept Guided Policy Search
Explored the possibility of demonstrating a correct trajectory that is given by a human operator or an expert algorithm to a reinforcement learning agent. Starting the training process from such a demonstration helps the agent to find good local minima and drastically decreases sample complexity needed to learn the task.
talk Using a slow RL algorithm to learn a fast RL algorithm using recurrent neural networks by Ilya Sutskever
All notable RL success stories are relying on huge sample complexity. To move RL forward we need to overcome this hunger. The core of the agent is a recurrent neural network that gives the agent the ability to incorporate previous states and experiences in its current state. During the training an agent is subject to training examples from different MDPs (different games) and its objective is averaged across all possible MDPs. One can argue that a resulting agent learns how to create a reinforcement learning algorithm; however, what is actually learned by an agent is a matter of philosophical debate. In order to get familiar with the topic, read:
- “Learning to Navigate in Complex Environments” by P. Mirowski et al.,
- “Learning to reinforcement learn” by J. X Wang et al.,
- “RL2: Fast Reinforcement Learning via Slow Reinforcement Learning” by Y. Duan et al., and
- “Learning to Learn for Global Optimization of Black Box Functions” by Y. Chen et al.
All these look into the problem of complex environments and/or multi-environment agents.
concept OpenAI Universe
Is a platform where you can train your RL agents in computer-based environments (games, browser, word processors, anything!). Provides the interface and has a large number of built-in games and browser-like tasks. The hopes are that Universe can serve as the next evaluation benchmark for RL research, and once it is “solved” we will have on our hands agents that can do whatever you and I can do via usual keyboard + mouse + screen interface. Which, in our current world, is a lot! In the light of strong research towards multi-environment agents and transfer learning it seems like a very useful benchmark indeed.
talk Representation learning by SGD in cross-validation by Rich Sutton
An idea presented by the author of the textbook on reinforcement learning is about a better way to learn representations in deep nets. The proposed method is an alternative to backpropagation and is called crossprop: SGD is used to find the gradient for outgoing weights wrt incoming weights of each node. On synthetically designed data, that is generated from known representations, this approach makes the nodes of the network better coincide with those representations than if trained via backprop.
talk The Nuts and Bolts of Deep RL Research by John Schulman
A set of practical advice on how to do and think about RL.
Approaching new problems: be able to run experiments quickly, visualize a lot, make the problem easier and easier until you see the first signs of life from the learning algorithm, at the early stages shape the reward to be favorable for learning.
POMDP design: it is hard to describe a whole new formalization of the environment, states and rewards, make random agents and visualize their policies to see how the environment is behaving, use human control to explore the environment, plot histograms of useful stats, expect to need more samples than you initially would expect.
Tuning phase: once your algorithm works, check for sensitivity to parameters — a good one should be robust, use multiple random seeds and do not trust articles with learning curves based on a single seed, run a set of benchmarks (other environments and tasks) to make sure that the algorithm still works and nothing is broken by recent changes, data whitening can be more useful than smarter algorithms, automate your experiments as much as possible — it pays off, estimate the required frame and action frequency by human control (meaning try to play it yourself), track mean / max / min / stdev of episode reward.
talk Learning to Navigate in Complex Environments by Raia Hadsell
The backbone of the system consists of the following blocks: Convolutional network for feature learning -> Stacked LSTM for action planning -> Additional inputs -> A3C RL algorithm -> Auxiliary target such as depth prediction. Such an agent is trained on a maze navigation task. The core observation is that having an auxiliary task that is relevant to understanding the environment increases total reward more than two times. Presumably due to better internal representations, the learning of which was enforced by the auxiliary task.
talk Learning to Experiment
An agent has to give answers about physical properties of an object, such as weight. This property cannot be learned from static visual input and in order to collect the information an agent has to interact with the world.
talk Combining Policy Gradient with Q-Learning
The task is to combine the power of an off-policy RL method (Q-Learning) with the on-policy Policy Gradient approach. Achieved and demonstrated better data efficiency.
talk Sample Efficient and Stable Deep Reinforcement Learning by Sergey Levine
As mentioned multiple times before, the main problem for real-world RL is data efficiency. Proposed a new method called Q-Prop that combines policy gradients with off-policy methods. Policy gradients are unbiased and stable; however, they have high variance, are sample inefficient and tend to forget previously learned things. Off-policy methods have low variance and are sample efficient, but can be biased and are less stable. Seems only natural to combine the two. Q-Prop has on-policy critic, off-policy critic with replay, combines Policy Gradient and Actor-Critic model with TRPO-GAE method. Demonstrated sample efficiency in MuJoCo with smaller batch size and is more stable on harder problems. In the real world it is trained on a 7-DOF robotic arm to open a door, and within 2.5 hours it was able to learn toclankily do that.
talk Large-Scale Self-Supervised Robotic Learning by Chelsea Finn
The application is robotic vision + control. The problem with human supervision is that it does not scale to more complex or more numerous tasks. Proposed a concept of self-supervision via reinforcement learning. One of the important aspects is a convolutional LSTM that predicts the next visual frame. The task of grasping different objects is learnt from 50K sequences (1M+ frames). Current limitations are the speed of execution and problems with more complex tasks. The dataset is available for download.
concept Course on Deep Reinforcement Learning: http://rll.berkeley.edu/deeprlcourse
talk Multi-Agent and Multi-Robot Coordination with Uncertainty and Limited Communication by Chris Amato
Nice picture to illustrate the difference between MDP and POMDP. The core idea of the presentation is that given complex robotic units and environment we might not want to waste RL efforts on exploring and learning how to operate each particular tool out of a toolbox a robot has. It makes more sense in practice to use hardcoded control logic for basic operations and apply RL on a larger scale — to learn how to achieve the desired goal by using the tools and existing primitives. In “Planning with Macro-Actions in Decentralized POMDPs” by Amato et al., 2014 they introduce the MacDec-POMDP approach, which uses subcontrollers for subtasks while the RL algorithm learns the general strategy, not the whole state space, which makes learning much more sample efficient.
talk Deep Robotic Learning by Sergey Levine
End-to-end in robotics. Current robotic applications consist of specialized blocks for different stages of operation: one for vision and feature extraction, one for planning, another for servo control. From examples of computer vision and speech recognition we’ve seen how such block-based pipelines were replaced by end-to-end deep learning. Does it mean that end-to-end is the way to go for robotics as well? Sergey thinks so, and in his talk he outlines his steps towards end-to-end robotics. First motivating example demonstrates an example of a motor taskwas it door opening or something else? does not matter where using the block-based pipeline the robot does not learn to perform the task at all, while using end-to-end learning it achieves 96.3% accuracy in performance. The explanation for such a drastic difference is that if you learn piece by piece, it does not guarantee that what you’ve learnt will assemble into a functioning pipeline, while when learning end-to-end the performance on the end task is exactly what you are optimizing for, and therefore if learning is successful, the robot will perform the task successfully.
Data efficiency. The first challenge and difference between end-to-end supervised learning and end-to-end reinforcement learning isyes, again the amount of data. In his experiments they took several robotic hands and put them to collect grasping data: with multiple hands the data acquisition process is faster. DNN behind the learning algorithm consisted of 16 convolutional layers. Deep Deterministic Policy Gradient algorithm (DDPG) and Normalized Advantage Function algorithm (NAF) can achieve training times that are suitable for real robotic systems (see “Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates“).
Safety is an important aspect in real-world robotics. There are articles that show how to incorporate model uncertainty into the planning in such a way that a robot will operate in safe conditions (low speed for example) if uncertain.
Reality gap occurs when you want to take your agent that is trained in a simulator into the real world. One approach for bridging the gap is proposed in “Real Single-Image Flight Without a Single Real Image” where a drone was trained to fly in a simulator and later tested in a real environment: it was able to fly very successfully until it encountered a glass door. Another work in the same direction is “EPOpt: Learning Robust Neural Network Policies Using Model Ensembles” where by use of policy ensembles and adversarial training the agent that was trained in a simulated environment is robust towards and generalizes to the real world including the effects that were not modeled by the simulator.
talk Deep Learning for Robotics by Pieter Abbeel
We might need something other than wheels or tracks to locomote in unknown environments like other planets and such. NASA has built the tensegrity Super Ball Botsee picture on the right that is able to move on a larger number of surfaces, is much more resilient and can be deployed without a parachute from greater heights. Since no one has a clue how this thing should use its motors to move, RL is the obvious choiceand it works!. The learning force is again RL2 with TRPO as the base algorithm.
concept RL on simple species
It would be cool to see an RL algorithm that is trained to replicate the behavior of simple species like C. elegans.
talk Sim-to-Real using Progressive Neural Networks by Raia Hadsell
One NN that is trained to perform one task is called a “column”reference to cortical columns?.
The idea is to train a new column for each new task. Each new column is connected to all previous columns so that features that were learnt for the previous tasks can be reused in the new tasks. Once a new column is added, the weights in the previous ones are frozen. Advantages: no forgetting of previous tasks, feature transfer, new capacity for each new task. Disadvantages: a separate mechanism is needed to know when to switch between the columns, number of parameters grows with the number of tasks (whole new network for each task). The approach demonstrates the usefulness of relevant intermediate tasks (a bit like curriculum learning). Even better generalization and robustness can be achieved via data augmentation: changing shapes, size, color, lighting of the objects in the simulator leads to a better final agent.
Speeding up the training in a physical environment can be achieved using this technology: the simulator is the first column, each new task in the real world is an additional column. Imitation learning is also a good way to beat sample complexity. Training in the simulator happens with A3C and is fast and parallelizable. Comparing training for the same task: 24h in the simulator with subsequent PNN for bridging the gap, 55d in a real physical system.
An alternative approach is fine-tuning to the real world after training in a simulator, but the PNN approach beats that by a large margin.
talk Bayesian Reasoning and Deep Learning in Agent-based Systems by Shakir Mohamed
A very nice talk that puts Bayesian ML and Deep Learning together and demonstrates how they fit together. Mention of the Bayesian framework being naturally related to memory, which seems to be an important topic at the moment in the context of Memory Networks and Neural Turing Machines.
panel Panel: Lawrence, Goodfellow, Mohamed, Welling, Adams
Is ensembling a practical replacement for Bayesian methods? Estimating confidence intervals and measuring the uncertainty were important points of discussion. How to specify good priors, why don’t we do that? Actually we do — the whole hierarchical structure of deep convolutional neural networks is a prior we imposed and it happened to be a useful one. A new important trend is to build generative models into algorithms and agents and use their predictions about the world in training and deployment.
concept CoRL 2017: 1st Annual Conference on Robot Learning
Neuroscience and Machine Learning
Comparison between the actual brain and DNNs seems to be a very popular theme. This is something I am doing myself, and it evoked mixed feelings to see 3 posters doing almost the same thing I am doing. On the other hand, it means that this direction in research seemed important to several groups, therefore it could have merit. Or just an effect of the hype, who knows…
keynote Keynote: Engineering Principles From Stable and Developing Brains
Synapse pruning happens with age. Why could that be useful? Initially everything is connected; with time active connections persist, useless ones disappear. They experimentally tested pruning approach (start with fully connected and remove) vs. growth approach (start with no connections and add as needed) — pruning is better in terms of “efficiency” and “robustness” measures. The next thing to explore is what the optimal rate of pruning is. Conducted experiments on mouse whiskers (and somatosensory columns) — sliced the brain, applied automatic synapse detection to count the number of synapses. The experiment a) confirmed the synaptic decrease over time, b) with the optimal pruning rateexpressed with a curve the resulting network is 20% more efficient.
Another topic they’ve explored is the brain mechanism for nearest neighbor search. Locality-sensitive hashing (LSH) is a common way to assign similar things close to each other in the hash space (same bin). Done via the random projection method. How does a fly do that? For its odor mechanism it maps a representation vector (molecule, 54-D) to an extended space (2000-D), applies a sparsity constraint (5% survives) on the output vector of 2000-D space. The resulting code is a “hash” of the input. Compared those two approaches on MNIST: nearly identical mean average precision; while on nearest neighbor specific tasks (SIFT10k, GIST10k datasets) the “Fly algorithm” gives considerably higher average precision. The same sparse locality-preserving algorithms seem to appear in other brain areas as well and might be an important underlying mechanism for some of the brain processes.
keynote Keynote: Learning About the Brain: Neuroimaging and Beyond
- Mental state recognition using ML: take data from the brain and try to predict something. For neuroscience it is important to focus on interpretable ML. Regarding automatic feature extraction — manual feature engineering is still important in brain sciences. Transform the channels x time matrix of EEG signal into an image and apply convolutional networks and LSTMs. Using deconvolution gave a bit of that desired interpretability.
- Another question they addressed is whether cheap wearable devices measure anything related to mental states. Using MUSE: 75% accuracy in classifying whether a video a subject is watching is a cat video or a Khan Academy video.
- Paths towards computational psychiatry: analyze patient interviews, build story graphs — showed that the complexity of the graphs is different for schizophrenia / control / mania cases.
talk Showing versus Doing: Teaching by Demonstration
When you (a human) demonstrate behavior, you tend to exaggerate to highlight behavior patterns that are important. If your task is just to perform, not to demonstrate, you do it quite differently and focus on optimality. I expected the next logical step, which would be to demonstrate how an agent trained with “showing” performs better than the one trained with “doing”, but that did not happen in the talk.
talk Predictive Coding for Unsupervised Learning by David Cox
Analogy between visual cortex and CNNs. New larger visual DNNs match the ventral stream better and better. The talk is about temporal coding and predictive coding in the context of deep learning. Next frame prediction works well on synthetic datasets, natural road images, comma.ai steering wheel dataset. code
panel Panel: Demis Hassabis, Terrence Sejnowski, Yoshua Bengio, Jakob Macke, Andreas Tolias
Synergy of neuroscience and machine learning
It would be nice to have more in vivo neuroscience experiments that produce more data for neuroinspired algorithms to borrow ideas from. Brain-like constraints (speed, size, memory efficiency) could be added to algorithmic approaches and that could lead to more brain-like algorithms. On the other hand, if the goal is not to understand the neural code, but to build intelligence, we might not need to limit ourselves by mere biological constraints. Maybe it is more beneficial to focus on understanding the learning process vs. trying to understand particular units of computation. Imagination-based planning, that is, model-based learning and generative learning, seems important at the moment.
Neuroscientists criticize the simplicity of ANNs
ML people are having a hard time trying to understand what exactly ANNs are doing, despite the fact that we have access to each particular neuron — imagine how hard it is for neuroscience to figure out what the neurons are doing without such a level of access. Neuroscience could record more area-wide data as opposed to single unit recordings. Dense networks (not hierarchical) can outperformthere was a paper on DenseNets hierarchical ones with a smaller number of neurons. A real synapse can be seen as having a state and this is a promising area to explore.
What does neuroscience have to do to be useful for ML?
Test the algorithms (especially biologically plausible ones) proposed by ML in in vivo experiments.
concept Hawkes process
Processes where the intensity of an event depends on the previous events are called Hawkes processes. There are examples of such in our world, for example spiking of neurons. There are models designed to fit such processes.
talk Approximating Human Vision with Deep NNs
Rapid image categorization is done well by humans: conducted a set of experiments using Amazon Mechanical Turk. The same set of images was fed to VGG-16 and compared how “difficult”not sure how they defined it, probably the number of errors per category it is for a human and for the net to classify the same image. Until the 5th layer of the CNN, difficulty correlates; after that the correlation drops, indicating that the DNN is doing something different from what a human is doing. The simpler AlexNet correlates better than more complex VGG. Also performed attention analysis: CNNs focus on different locations/features from humans.
Learning to Learn
Meta learning, or learning to learn, was a concept that reemerged here and there as a yet-uncertain-but-conceptually-promising direction. There are various ideas under that umbrella, one of which is to formulate the design of an optimization algorithm as a learning problem. That way, what is learnt are not only the best parameters that minimize the loss function, but also the parameters of an optimizer that learns those parameters, alleviating the need for manual design of the optimization algorithm. To know more about this and related concepts read about Learning to learn by gradient descent by gradient descent code . Another way to look at meta learning is proposed by Differentiable Neural Computer by Google DeepMind.
concept Sentient Technologies
Is a company that uses massive distributed computational resources to evaluate learning algorithms using genetic algorithms. Just curious to know that such thing exists.
talk Symposium: Recurrent Neural Networks and Other Machines that Learn Algorithms
Presentations without “deep” in the title
keynote Keynote: Reproducible Research for Multidomain Data: the Case of the Human Microbiome
Lots of data in bioinformatics, lots of experiments, needs to be reproducible. They work on that by fixing pipelines, open-sourcing data and code, checking results of others. “Give me any study I can always find you a significant p-value”. Transparency in analysis allows publishing non-perfect results and still move science forward.
talk Award: Matrix Completion has No Spurious Local Minimum
As the title says: they show that for a Netflix-like matrix completion objective function all local minima are global.
talk Talk: Examples are not enough, learn to criticize! Criticism for Interpretability.
Given a dataset, you pick common samples (prototypes) and most uncommon examples (criticisms) from each category (using Maximum Mean Discrepancy). Those are useful to explore your data, applicable when the dataset is huge and the best thing you can normally do is to randomly sample to familiarize yourself with the data (meaning that you’ll never see the weirdos in the data as random sampling will not show them to you). Examples from ImageNet: prototypes are, for example, usual pictures of dogs, criticisms are the pictures of dogs from behind, from weird angles etc.
talk Program Synthesis and Machine Learning
Code completion formulated as an ML task: the input is a piece of code and output is the next line. Unlike other similar tools, it makes better use of the context to propose the completion.
concept Random notes from Neurorobotics workshop
- Papers on human-guided learning in robotics: “Strategies for Human-in-the-Loop Robotic Grasping“, “Humanoid Robot Posture-Control Learning in Real-Time Based on Human Sensorimotor Learning Ability“, “Adaptive Control of Exoskeleton Robots for Periodic Assistive Behaviours Based on EMG Feedback Minimisation“
- During the panel old-school roboticists were questioning whether all this deep learning end-to-end robotics is just hype or could really replace the old ways. The dominating mood was that it is a promising direction indeed and many are eager to try it out. However, it must be seen as a tool as it does not provide much insight into understanding of control.
Posters & Demos
Having demos was a nice initiative — they show the gap between all-is-so-well you can observe in the papers and the reality. Nothing special to report here: a few applications in robotics, a few real-time applications of CNNs, everything almost works, but does not amaze.
There was a huge amount of posters, here are a few of personal interest.

