Quantitative bounds for concentration-of-measure inequalities and empirical regression: The independent case

This paper is devoted to the study of the deviation of the (random) average L²−error associated to the least-squares regressor over a family of functions F_n (with controlled complexity) obtained from n independent, but not necessarily identically distributed, samples of explanatory and response variables, from the minimal (deterministic) average L²−error associated to this family of functions, and to some of the corresponding consequences for the problem of consistency. In the i.i.d. case, this specializes as classical questions on least-squares regression problems, but in more general cases, this setting permits a precise investigation in the direction of the study of nonasymptotic errors for least-squares regression schemes in nonstationary settings, which we motivate providing background and examples. More precisely, we prove first two nonasymptotic deviation inequalities that generalize and refine corresponding known results in the i.i.d. case. We then explore some consequences for nonasymptotic bounds of the error both in the weak and the strong senses. Finally, we exploit these estimates to shed new light into questions of consistency for least-squares regression schemes in the distribution-free, nonparametric setting. As an application to the classical theory, we provide in particular a result that generalizes the link between the problem of consistency and the Glivenko-Cantelli property, which applied to regression in the i.i.d. setting over non-decreasing families (F_n)_n of functions permits to create a scheme which is strongly consistent in L² under the sole (necessary) assumption of the existence of functions in ∪_nF_n which are arbitrarily close in L² to the corresponding regressor.