Periodic Signal Recovery with Regularized Sine Neural Networks

We consider the problem of learning a periodic one-dimensional signal with neural networks, and designing models that are able to extrapolate the signal well beyond the training window. First, we show that multi-layer perceptrons with ReLU activations are provably unable to perform this extrapolation task, and lead to poor performance in practice even close to the training window. Then, we propose a modified training procedure for two-layer architectures with sine activations with a more diverse feature initialization and well-chosen non-convex regularization, that is able to extrapolate the signal with low error well beyond the training window. This procedure yields results several orders of magnitude better than its competitors for distant extrapolation (beyond 100 periods of the signal), while being able to accurately recover the frequency spectrum of the signal in a multi-tone setting.