Towards a validation-less approach for small data: training with neural velocity

Tuning hyperparameters such as learning rate decay and stop conditions is typically done by assessing loss on a held-out validation set. This work introduces neural velocity (NeVe), the rate of change in neuron transfer functions, as a novel indicator of model convergence. We leverage NeVe to create a dynamic training method where learning rates and stop conditions are adjusted based on neural velocity computed from an auxiliary dataset generated by sampling noise. Our experiments across multiple tasks and architectures show that our approach performs on par with traditional validation-based methods, outperforming them in some tasks. Our method does not require withholding data for validation, making it especially advantageous in data-limited scenarios. This work highlights the potential of neural velocity as a key metric for optimizing neural network training.