RAT

Resource-efficient Autonomous Training Algorithms for Deep Learning

01/01/2022 to 31/12/2024

This project aims to drastically reduce the resource footprint of deep learning, by eliminating the need for complicated parameter tuning. Stochastic gradient descent (SGD) and its variants have become the standard for training deep neural. However, by exposing crucial hyperparameters – most notably the learning rate – these optimization methods still rely to a large extent on human intervention. Successful training is often impossible without either carefully crafted hyperparameters by experts or extensive and costly parameter searches. Instead, we argue for a new type of tuning-free training algorithm, that works out-of-the-box, making training deep neural networks more efficient and sustainable. Replacing the static update rule of many current methods with an active agent-like behavior, allows the optimizer to dynamically adapt the learning rate within a single training run. Our training algorithm will carefully consider the different learning phases and their distinct objectives. Augmenting the optimization method with additional information from the entire distribution of each batch will ensure that an informed decision is possible. Each aspect will help address one of the two major challenges for automatic deep learning training: The stochasticity of the learning signal and the greedy myopic decision of current methods.