Overcoming the curse of dimensionality with Laplacian regularization in semi-supervised learning

Vivien Cabannes; Loucas Pillaud-Vivien; Francis Bach; Alessandro Rudi

Overcoming the curse of dimensionality with Laplacian regularization in semi-supervised learning

Vivien Cabannes
, Loucas Pillaud-Vivien
, Francis Bach
, Alessandro Rudi

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

As annotations of data can be scarce in large-scale practical problems, leveraging unlabelled examples is one of the most important aspects of machine learning. This is the aim of semi-supervised learning. To benefit from the access to unlabelled data, it is natural to diffuse smoothly knowledge of labelled data to unlabelled one. This induces to the use of Laplacian regularization. Yet, current implementations of Laplacian regularization suffer from several drawbacks, notably the well-known curse of dimensionality. In this paper, we provide a statistical analysis to overcome those issues, and unveil a large body of spectral filtering methods that exhibit desirable behaviors. They are implemented through (reproducing) kernel methods, for which we provide realistic computational guidelines in order to make our method usable with large amounts of data.

Original language	English
Title of host publication	Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021
Editors	Marc'Aurelio Ranzato, Alina Beygelzimer, Yann Dauphin, Percy S. Liang, Jenn Wortman Vaughan
Publisher	Neural information processing systems foundation
Pages	30439-30451
Number of pages	13
ISBN (Electronic)	9781713845393
Publication status	Published - 1 Jan 2021
Externally published	Yes
Event	35th Conference on Neural Information Processing Systems, NeurIPS 2021 - Virtual, Online Duration: 6 Dec 2021 → 14 Dec 2021

Publication series

Name	Advances in Neural Information Processing Systems
Volume	36
ISSN (Print)	1049-5258

Conference

Conference	35th Conference on Neural Information Processing Systems, NeurIPS 2021
City	Virtual, Online
Period	6/12/21 → 14/12/21

Cite this

Cabannes, V., Pillaud-Vivien, L., Bach, F., & Rudi, A. (2021). Overcoming the curse of dimensionality with Laplacian regularization in semi-supervised learning. In MA. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, & J. Wortman Vaughan (Eds.), Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021 (pp. 30439-30451). (Advances in Neural Information Processing Systems; Vol. 36). Neural information processing systems foundation.

Cabannes, Vivien ; Pillaud-Vivien, Loucas ; Bach, Francis et al. / Overcoming the curse of dimensionality with Laplacian regularization in semi-supervised learning. Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. editor / Marc'Aurelio Ranzato ; Alina Beygelzimer ; Yann Dauphin ; Percy S. Liang ; Jenn Wortman Vaughan. Neural information processing systems foundation, 2021. pp. 30439-30451 (Advances in Neural Information Processing Systems).

@inproceedings{dd54ee3b184a4823a5346f7c0bb73491,

title = "Overcoming the curse of dimensionality with Laplacian regularization in semi-supervised learning",

abstract = "As annotations of data can be scarce in large-scale practical problems, leveraging unlabelled examples is one of the most important aspects of machine learning. This is the aim of semi-supervised learning. To benefit from the access to unlabelled data, it is natural to diffuse smoothly knowledge of labelled data to unlabelled one. This induces to the use of Laplacian regularization. Yet, current implementations of Laplacian regularization suffer from several drawbacks, notably the well-known curse of dimensionality. In this paper, we provide a statistical analysis to overcome those issues, and unveil a large body of spectral filtering methods that exhibit desirable behaviors. They are implemented through (reproducing) kernel methods, for which we provide realistic computational guidelines in order to make our method usable with large amounts of data.",

author = "Vivien Cabannes and Loucas Pillaud-Vivien and Francis Bach and Alessandro Rudi",

note = "Publisher Copyright: {\textcopyright} 2021 Neural information processing systems foundation. All rights reserved.; 35th Conference on Neural Information Processing Systems, NeurIPS 2021 ; Conference date: 06-12-2021 Through 14-12-2021",

year = "2021",

month = jan,

day = "1",

language = "English",

series = "Advances in Neural Information Processing Systems",

publisher = "Neural information processing systems foundation",

pages = "30439--30451",

editor = "Marc'Aurelio Ranzato and Alina Beygelzimer and Yann Dauphin and Liang, \{Percy S.\} and \{Wortman Vaughan\}, Jenn",

booktitle = "Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021",

}

Cabannes, V, Pillaud-Vivien, L, Bach, F & Rudi, A 2021, Overcoming the curse of dimensionality with Laplacian regularization in semi-supervised learning. in MA Ranzato, A Beygelzimer, Y Dauphin, PS Liang & J Wortman Vaughan (eds), Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. Advances in Neural Information Processing Systems, vol. 36, Neural information processing systems foundation, pp. 30439-30451, 35th Conference on Neural Information Processing Systems, NeurIPS 2021, Virtual, Online, 6/12/21.

Overcoming the curse of dimensionality with Laplacian regularization in semi-supervised learning. / Cabannes, Vivien; Pillaud-Vivien, Loucas; Bach, Francis et al.
Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. ed. / Marc'Aurelio Ranzato; Alina Beygelzimer; Yann Dauphin; Percy S. Liang; Jenn Wortman Vaughan. Neural information processing systems foundation, 2021. p. 30439-30451 (Advances in Neural Information Processing Systems; Vol. 36).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Overcoming the curse of dimensionality with Laplacian regularization in semi-supervised learning

AU - Cabannes, Vivien

AU - Pillaud-Vivien, Loucas

AU - Bach, Francis

AU - Rudi, Alessandro

PY - 2021/1/1

Y1 - 2021/1/1

N2 - As annotations of data can be scarce in large-scale practical problems, leveraging unlabelled examples is one of the most important aspects of machine learning. This is the aim of semi-supervised learning. To benefit from the access to unlabelled data, it is natural to diffuse smoothly knowledge of labelled data to unlabelled one. This induces to the use of Laplacian regularization. Yet, current implementations of Laplacian regularization suffer from several drawbacks, notably the well-known curse of dimensionality. In this paper, we provide a statistical analysis to overcome those issues, and unveil a large body of spectral filtering methods that exhibit desirable behaviors. They are implemented through (reproducing) kernel methods, for which we provide realistic computational guidelines in order to make our method usable with large amounts of data.

AB - As annotations of data can be scarce in large-scale practical problems, leveraging unlabelled examples is one of the most important aspects of machine learning. This is the aim of semi-supervised learning. To benefit from the access to unlabelled data, it is natural to diffuse smoothly knowledge of labelled data to unlabelled one. This induces to the use of Laplacian regularization. Yet, current implementations of Laplacian regularization suffer from several drawbacks, notably the well-known curse of dimensionality. In this paper, we provide a statistical analysis to overcome those issues, and unveil a large body of spectral filtering methods that exhibit desirable behaviors. They are implemented through (reproducing) kernel methods, for which we provide realistic computational guidelines in order to make our method usable with large amounts of data.

UR - https://www.scopus.com/pages/publications/85130807181

M3 - Conference contribution

AN - SCOPUS:85130807181

T3 - Advances in Neural Information Processing Systems

SP - 30439

EP - 30451

BT - Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021

A2 - Ranzato, Marc'Aurelio

A2 - Beygelzimer, Alina

A2 - Dauphin, Yann

A2 - Liang, Percy S.

A2 - Wortman Vaughan, Jenn

PB - Neural information processing systems foundation

T2 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021

Y2 - 6 December 2021 through 14 December 2021

ER -

Cabannes V, Pillaud-Vivien L, Bach F, Rudi A. Overcoming the curse of dimensionality with Laplacian regularization in semi-supervised learning. In Ranzato MA, Beygelzimer A, Dauphin Y, Liang PS, Wortman Vaughan J, editors, Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. Neural information processing systems foundation. 2021. p. 30439-30451. (Advances in Neural Information Processing Systems).

Overcoming the curse of dimensionality with Laplacian regularization in semi-supervised learning

Abstract

Publication series

Conference

Other files and links

Fingerprint

Cite this