ERC Starting Grant
May 2025 - May 2030
In this project, I explore what role can be played by Reinforcement Learning in the development and understanding of Continual Learning agents.
This project is funded by the European Research Council
and by the Cluster of Excellence ML for Science of the University of Tübingen
Machine Learning systems, while promising, lack the autonomy needed for many real-world applications beyond mere proof-of-concept stages. The key challenge lies in enabling AI to continuously adapt to shifting data distributions and proactively seek information under high uncertainty.
Fields such as drug discovery and micro-chemistry are expecting breakthroughs from AI, given the vast and intricate search spaces they deal with, coupled with expensive data acquisition. It is vital for algorithms to steer this search, assimilate new data, and strategically explore promising zones.
Reinforcement Learning (RL) offers tools and methods for agents to autonomously learn from their actions, but its efficacy has been largely confined to stationary, single-task settings.
ConSequentIAL's vision is a Continual and Sequential Learning AI that marries supervised and unsupervised learning with advanced data gathering and RL-driven discovery mechanisms.
To achieve these goals, I propose to bridge the theory of constrained and non-stationary RL to build a sound and useful mathematical formulation of the problem. On these new solid grounds, I develop novel algorithmic principles that allow the agent to detect and respond to external shifts, while remaining aware of her own impact on the system she interacts with. I address the memory-versus-stability trade-off central to continual learning by enabling agents to actively plan their skill acquisition in accordance with their long-term goals.
The ambition of this project is to position AI to tackle the consequential scientific challenges ahead.
Emmy Noether -- AI initiative
January 2023 - January 2029
In this project, I propose to further explore the theory of RL in non-stationary but structured environments. In particular, I study how we can caracterize and quantify meta-learning in RL, and how this affects exploration strategies.
This project is funded by the Deutsche Forschung Gemeinshaft (DFG)
and by the Cluster of Excellence ML for Science of the University of Tübingen
A major, challenging problem that arises in Artificial Intelligence (AI) is that of allowing machines to automatically and efficiently reuse past experience.
In the Machine Learning parlance, Meta-Learning, is the ability of an agent to acquire skills across tasks, and relies on a long-term definition of the loss function which averages the learner’s risk over tasks and entice them to learn high-level, generalizable concepts. This goal is multifaceted, especially in a online and interactive learning setting such as Reinforcement Learning.
I identify three key challenges to resolve:
• Handling the non-stationarity and impermanence of the world;
• Learning and using appropriately a correct prior over the task-generating environment;
• Building a hierarchical approach to problems to abstract low-level complexities when facing long-term goals.
I use the term Lifelong Reinforcement Learning to describe a learning problem that requires overcoming all three challenges. It combines the challenges of reinforcement learning with those of unsupervised learning and non-stationary online learning, which are my three areas of expertise in machine learning.
Students:
Nicolas Nguyen (January 2023 - now)
Onno Eberhard (June 2023 - now)
Ziyad Sheebaelhamd (September 2024 - now)