(Maths-IA & IOP) Bilevel optimization for machine learning

Bilevel problems are optimization problems characterized by a hierarchical structure. In these problems, one seeks to minimize an outer function subject to the constraint that certain variables minimize an inner function. These problems are gaining popularity in the machine learning community due to their wide range of applications, such as hyperparameter optimization and data reweighting.

In this talk, we introduce bilevel optimization and demonstrate how various machine learning problems can be formulated within this framework. We then focus on the algorithmic aspects of solving bilevel problems. Specifically, we present a general algorithmic framework that enables the adaptation of first-order stochastic solvers—originally designed for single-level problems—to the bilevel setting. We provide theoretical guarantees for specific instances of this framework.