Artificial Intelligence and Machine Learning

Introduction

How should an AI model’s training change when it is given more data? Researchers at Forschungszentrum Jülich used JUWELS Booster to investigate how training settings and neural network building blocks affect performance at scale. The BeSL (Better models with Scaling Laws) project established practical relationships between data budgets, learning rates, and batch sizes. It developed a stable way to train networks with learnable activation functions. The results offer guidance for making better use of supercomputing resources when developing AI models.

Training settings must change with the data budget

Training a large language model involves choices that can strongly affect its performance. The learning rate controls how far the model’s parameters move at each update. The batch size determines how much training data contributes to that update. Finding a good combination usually requires many costly trial runs.

BeSL examined these choices across Transformer models ranging from approximately 32 million to 5 billion parameters. The team used maximal update parameterisation, or µP, a framework for transferring training settings between model sizes, and investigated how those settings should change with the total number of training tokens, the pieces of text processed by a language model.

The critical batch size, a characteristic scale governing the relationship between batch size and the best learning rate, grew approximately in proportion to the token budget. The batch size giving the best validation performance also increased. Keeping the batch size fixed therefore became suboptimal as the data budget grew, even when the learning rate was adjusted [1].

A complementary study found a useful indicator called norm transfer. Across models with up to 1.3 billion parameters trained on up to 138 billion tokens, the best learning rate and batch size combinations produced nearly the same operator norm of the output layer, a measure of how strongly that layer can amplify a signal. This pattern held for both Adam and Scion optimizers. Matching the norm alone did not guarantee the best result, but it offered a guide for tuning across scales [2].

Learning the activation functions

The project also revisited activation functions, the nonlinearities that shape signals as they pass through a neural network. Standard models usually use a fixed function such as ReLU or GELU... BeSL explored functions whose shape can instead be learned during training.

The researchers studied three families: Hermite polynomial functions, Fourier functions built from sines and cosines, and Tropical functions formed from maxima of linear expressions. Polynomial and trigonometric activations can be difficult to train because signals and gradients may grow uncontrollably or fade as they pass through many layers.

The team developed a variance-preserving initialisation method that balances the scale of signals and gradients at the start of training. This enabled the functions to be used in deep networks without additional clamping mechanisms. Tests used GPT-2 for language modelling on OpenWebText and ConvNeXt for image classification on ImageNet-1K. The report describes performance comparable to, and in some tested configurations better than, standard activations. The work was published at ICLR 2026 [3].