- The one and only newsletter about the latest research, current trends, and upcoming events in Machine Learning, flavored with graphs.
- The ability to store and manipulate information is a hallmark of computational systems. Whereas computers are carefully engineered to represent and perform mathematical operations on structured data, neurobiological systems adapt to perform analogous functions without needing to be explicitly engineered. Recent efforts have made progress in modelling the representation and recall of information in neural systems. However, precisely how neural systems learn to modify these representations remains far from understood. Here, we demonstrate that a recurrent neural network (RNN) can learn to modify its representation of complex information using only examples, and we explain the associated learning mechanism with new theory. Specifically, we drive an RNN with examples of translated, linearly transformed or pre-bifurcated time series from a chaotic Lorenz system, alongside an additional control signal that changes value for each example. By training the network to replicate the Lorenz inputs, it learns to autonomously evolve about a Lorenz-shaped manifold. Additionally, it learns to continuously interpolate and extrapolate the translation, transformation and bifurcation of this representation far beyond the training data by changing the control signal. Furthermore, we demonstrate that RNNs can infer the bifurcation structure of normal forms and period doubling routes to chaos, and extrapolate non-dynamical, kinematic trajectories. Finally, we provide a mechanism for how these computations are learned, and replicate our main results using a Wilson–Cowan reservoir. Together, our results provide a simple but powerful mechanism by which an RNN can learn to manipulate internal representations of complex information, enabling the principled study and precise design of RNNs. Recurrent neural networks (RNNs) can learn to process temporal information, such as speech or movement. New work makes such approaches more powerful and flexible by describing theory and experiments demonstrating that RNNs can learn from a few examples to generalize and predict complex dynamics including chaotic behaviour.
- Any fundamental discovery involves a significant degree of risk. If an idea is guaranteed to work then it moves from the realm of research to engineering. Unfortunately, this also means that most…
- Chris G. Willcocks, Durham University
- - Sep. 28 – Oct. 2, 2020 - Lihong Li (Google Brain; chair), Marc G. Bellemare (Google Brain) - The success of deep neural networks in modeling complicated functions has recently been applied by the reinforcement learning community, resulting in algorithms that are able to learn in environments previously thought to be much too large. Successful applications span domains from robotics to health care. However, the success is not well understood from a theoretical perspective. What are the modeling choices necessary for good performance, and how does the flexibility of deep neural nets help learning? This workshop will connect practitioners to theoreticians with the goal of understanding the most impactful modeling decisions and the properties of deep neural networks that make them so successful. Specifically, we will study the ability of deep neural nets to approximate in the context of reinforcement learning.
- The program focused on the following four themes: - Optimization: How and why can deep models be fit to observed (training) data? - Generalization: Why do these trained models work well on similar but unobserved (test) data? - Robustness: How can we analyze and improve the performance of these models when applied outside their intended conditions? - Generative methods: How can deep learning be used to model probability distributions?
- TL;DR: Have you even wondered what is so special about convolution? In this post, I derive the convolution from first principles and show that it naturally emerges from translational symmetry. During…
- How to stay updated with the latest and coolest machine learning advancements

- (
*2021*)*cite arxiv:2105.04026Comment: This review paper will appear as a book chapter in the book "Theory of Deep Learning" by Cambridge University Press.* - (
*2018*)*cite arxiv:1801.05894.* *IEEE Transactions on Multimedia*(*2021*)- (
*2019*)*cite arxiv:1905.09550Comment: 12 pages, 5 figures, 2 tables.* - (
*2020*)*cite arxiv:2006.05582Comment: ICML 2020.* - (
*2021*)*cite arxiv:2104.14554.* *Neural Computation**9 (8): 1735--1780*(*November 1997*)- (
*2015*)*cite arxiv:1503.04069Comment: 12 pages, 6 figures.* - (
*2021*)*cite arxiv:2104.07012.* - (
*2021*)*cite arxiv:2101.10353Comment: Accepted by AAAI 2021.* - (
*2021*)*cite arxiv:2103.16775Comment: 66 pages, 24 figures.* - (
*2021*)*cite arxiv:2103.12608.* - (
*2021*)*cite arxiv:2103.03404.* - (
*2018*)*cite arxiv:1811.12560.* - (
*2018*)*cite arxiv:1805.07440Comment: To appear in the Thirty-Fourth AAAI conference on Artificial Intelligence (AAAI-2020).* *Nature**529 (7587): 484--489*(*2016*)- (
*2020*) - (
*2019*)*cite arxiv:1910.03560Comment: ECCV 2020 camera ready. This is an updated version of "Boosting Supervision with Self-Supervision for Few-shot Learning" arXiv:1906.07079.* - (
*2020*)*cite arxiv:2007.15484Comment: 17 pages, 10 figures.* - (
*2020*)*cite arxiv:2005.14165Comment: 40+32 pages.*