ProjectRAT – Resource-efficient Autonomous Training Algorithms for Deep Learning

Basic data

Acronym:
RAT
Title:
Resource-efficient Autonomous Training Algorithms for Deep Learning
Duration:
1/1/2022 to 12/31/2024
Abstract / short description:
This project aims to drastically reduce the resource footprint of deep learning, by eliminating the need for complicated parameter tuning. Stochastic gradient descent (SGD) and its variants have become the standard for training deep neural. However, by exposing crucial hyperparameters – most notably the learning rate – these optimization methods still rely to a large extent on human intervention. Successful training is often impossible without either carefully crafted hyperparameters by experts or extensive and costly parameter searches. Instead, we argue for a new type of tuning-free training algorithm, that works out-of-the-box, making training deep neural networks more efficient and sustainable. Replacing the static update rule of many current methods with an active agent-like behavior, allows the optimizer to dynamically adapt the learning rate within a single training run. Our training algorithm will carefully consider the different learning phases and their distinct objectives. Augmenting the optimization method with additional information from the entire distribution of each batch will ensure that an informed decision is possible. Each aspect will help address one of the two major challenges for automatic deep learning training: The stochasticity of the learning signal and the greedy myopic decision of current methods.
Keywords:
optimising
Optimierung
machine learning
Maschinelles Lernen
algorithms
Algorithmen
Deep Learning

Involved staff

Managers

Wilhelm Schickard Institute of Computer Science (WSI)
Department of Informatics, Faculty of Science

Other staff

Department of Informatics
Faculty of Science
Wilhelm Schickard Institute of Computer Science (WSI)
Department of Informatics, Faculty of Science

Local organizational units

Department of Informatics
Faculty of Science
University of Tübingen

Funders

Tübingen, Baden-Württemberg, Germany
Help

will be deleted permanently. This cannot be undone.