MPOL

Mehrphasige probabilistische Optimierer für tiefe Lernprobleme

01/07/2021 to 30/06/2024

This proposal to SPP 2298/1 proposes to investigate a novel paradigm for the training of deep neural networks. The peculiarities of deep models, in particular strong stochasticity (SNR<1), preclude the use of classic optimization algorithms. And contemporary alternatives, of which there are already many, are wasteful with resources. Rather than add yet another optimization rule to the long and growing list of such methods, this project aims to make substantial conceptual progress by investigating two key ideas: First, leveraging the entire probability distribution of gradients across the dataset (empirically estimated at runtime) to identify algorithmic parameters that the user would otherwise have to manually tune. And second, splitting the optimization process into at least three distinct phases with differing mathematical objectives. The goal is to develop (concretely, as a software library) an optimizer that requires no manual tuning, and achieves good generalization performance without repeated re-starts.