RESEARCH

Physical Atari brings the richness of the Arcade Learning Environment benchmark to the physical world. It consists of the Robotroller, which actuates the Atari CX40+ controller; the Atari Devbox, which runs the Arcade Learning Environment asynchronously; a camera; and a reinforcement learning agent. Together, these components enable the study of challenges associated with learning in the physical world, such as time delays and non-stationarities, which are ignored in simulation benchmarks.

Presented at Upper Bound 2025, this talk explores an alternative path toward artificial general intelligence centered on reinforcement learning, online adaptation, and learning from streams of experience rather than static datasets.

Topics include Physical Atari, continual learning, catastrophic forgetting, transfer learning, latency, embodiment, exploration, function approximation, and the challenges of building agents that learn efficiently over long time horizons.

This work studies how neural networks gradually lose their ability to adapt during continual learning and explores methods for preserving long-term learning capacity.

The project became widely discussed within the reinforcement learning and continual learning communities due to its implications for scalable lifelong learning systems.

Introduces a new class of reinforcement learning algorithms that discover predictive structure directly from experience, avoiding the need for hand-crafted input representations or domain-specific architectures.

Demonstrates linear scaling on combinatorial reinforcement learning problems while outperforming conventional deep reinforcement learning approaches on unstructured observation spaces.

Characterizes the loss of plasticity observed when reinforcement learning agents train across changing environments, identifying progressively sparser activations and diminishing gradients as key failure mechanisms.

Demonstrates a simple architectural change can improve plasticity with the Concatenated ReLU (CReLU).

Combines prioritized replay, distributional reinforcement learning, double Q-learning, noisy networks, dueling architectures, and multi-step returns into a unified learning system that dramatically improved Atari performance.

Rainbow became an influential algorithm in deep reinforcement learning and remains as a common touchstone for modern Atari research.

Introduces multi-timescale nexting, enabling autonomous agents to learn rich predictive models of future sensorimotor experience while interacting continuously with the environment.

The work helped establish predictive knowledge representations as a scalable foundation for continual learning and autonomous robotics.

Introduces Horde, an architecture composed of thousands of independent reinforcement learning ‘demons’ that simultaneously learn predictive and control knowledge from a shared stream of experience.

Horde established a practical framework for continual off-policy learning and influenced later work on predictive representations, auxiliary learning, and scalable reinforcement learning systems.

A popular view in the AI community is that a sufficiently large neural network, once trained on enough data, would be able to do everything we want our AI systems to do. This idea makes sense if the world we live in has a finite number of things that are useful to learn. It does not make sense if the world is filled with infinitely many things that could be learned, and achieving different goals requires learning different things. The view that the world is big and filled with infinitely many subtle things that could be learned for achieving goals is the big world hypothesis (Javed & Sutton, 2024).

SwiftTD is a TD learning algorithm that can learn to assign credit to signals that generalize well by learning per-feature step-size parameters. Additionally, it uses a bound on the size of the update to prevent the update from overshooting the target. An algorithm like SwiftTD will be a key ingredient for robust real-time learning from rich and noisy sensorimotor streams.

Step-size optimization learns the step-size parameters for each feature by minimizing the prediction error on the next step. This is a more principled way to learn the step-size parameters than step-size normalization, which does not optimize a clear objective. This work shows, in a simple synthetic experiment, that step-size optimization is fundamentally different from step-size normalization.

Introduces scalable variants of Real-Time Recurrent Learning (RTRL) by decomposing recurrent networks into modular or staged architectures that scale linearly with model size.

This work introduces OML, a meta-learning objective that explicitly trains representations for continual adaptation, enabling rapid online learning while minimizing interference with previously acquired knowledge.