LEGALBENCH: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models

Neel Guha; Julian Nyarko; Daniel E. Ho; Christopher Ré; Adam Chilton; Aditya Narayana; Alex Chohlas-Wood; Austin Peters; Brandon Waldon; Daniel N. Rockmore; Diego Zambrano; Dmitry Talisman; Enam Hoque; Faiz Surani; Frank Fagan; Galit Sarfaty; Gregory M. Dickinson; Haggai Porat; Jason Hegland; Jessica Wu; Joe Nudell; Joel Niklaus; John Nay; Jonathan H. Choi; Kevin Tobia; Margaret Hagan; Megan Ma; Michael Livermore; Nikon Rasumov-Rahe; Nils Holzenberger; Noam Kolt; Peter Henderson; Sean Rehaag; Sharad Goel; Shang Gao; Spencer Williams; Sunny Gandhi; Tom Zur; Varun Iyer; Zehua Li

LEGALBENCH: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models

Neel Guha
, Julian Nyarko
, Daniel E. Ho
, Christopher Ré
, Adam Chilton
, Aditya Narayana
, Alex Chohlas-Wood
, Austin Peters
, Brandon Waldon
, Daniel N. Rockmore
, Diego Zambrano
, Dmitry Talisman
, Enam Hoque
, Faiz Surani
, Frank Fagan
, Galit Sarfaty
, Gregory M. Dickinson
, Haggai Porat
, Jason Hegland
, Jessica Wu

Joe Nudell, Joel Niklaus, John Nay, Jonathan H. Choi, Kevin Tobia, Margaret Hagan, Megan Ma, Michael Livermore, Nikon Rasumov-Rahe, Nils Holzenberger, Noam Kolt, Peter Henderson, Sean Rehaag, Sharad Goel, Shang Gao, Spencer Williams, Sunny Gandhi, Tom Zur, Varun Iyer, Zehua Li

Stanford University
University of Chicago
Maxime Tools
Dartmouth College
LawBeta
South Texas College of Law
University of Toronto
St. Thomas University Benjamin L. Crump College of Law
Harvard Law School
Stanford Center for Legal Informatics - CodeX
University of Southern California
Georgetown University Law Center
University of Virginia
Osgoode Hall Law School
Harvard University
Casetext
Golden Gate University
Luddy School of Informatics - Indiana University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

The advent of large language models (LLMs) and their adoption by the legal community has given rise to the question: what types of legal reasoning can LLMs perform? To enable greater study of this question, we present LEGALBENCH: a collaboratively constructed legal reasoning benchmark consisting of 162 tasks covering six different types of legal reasoning. LEGALBENCH was built through an interdisciplinary process, in which we collected tasks designed and hand-crafted by legal professionals. Because these subject matter experts took a leading role in construction, tasks either measure legal reasoning capabilities that are practically useful, or measure reasoning skills that lawyers find interesting. To enable cross-disciplinary conversations about LLMs in the law, we additionally show how popular legal frameworks for describing legal reasoning—which distinguish between its many forms—correspond to LEGALBENCH tasks, thus giving lawyers and LLM developers a common vocabulary. This paper describes LEGALBENCH, presents an empirical evaluation of 20 open-source and commercial LLMs, and illustrates the types of research explorations LEGALBENCH enables.

Original language	English
Title of host publication	Advances in Neural Information Processing Systems 36 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023
Editors	A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, S. Levine
Publisher	Neural information processing systems foundation
ISBN (Electronic)	9781713899921
Publication status	Published - 1 Jan 2023
Event	37th Conference on Neural Information Processing Systems, NeurIPS 2023 - New Orleans, United States Duration: 10 Dec 2023 → 16 Dec 2023

Publication series

Name	Advances in Neural Information Processing Systems
Volume	36
ISSN (Print)	1049-5258

Conference

Conference	37th Conference on Neural Information Processing Systems, NeurIPS 2023
Country/Territory	United States
City	New Orleans
Period	10/12/23 → 16/12/23

Cite this

Guha, N., Nyarko, J., Ho, D. E., Ré, C., Chilton, A., Narayana, A., Chohlas-Wood, A., Peters, A., Waldon, B., Rockmore, D. N., Zambrano, D., Talisman, D., Hoque, E., Surani, F., Fagan, F., Sarfaty, G., Dickinson, G. M., Porat, H., Hegland, J., ... Li, Z. (2023). LEGALBENCH: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, & S. Levine (Eds.), Advances in Neural Information Processing Systems 36 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023 (Advances in Neural Information Processing Systems; Vol. 36). Neural information processing systems foundation.

Guha, Neel ; Nyarko, Julian ; Ho, Daniel E. et al. / LEGALBENCH : A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models. Advances in Neural Information Processing Systems 36 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023. editor / A. Oh ; T. Neumann ; A. Globerson ; K. Saenko ; M. Hardt ; S. Levine. Neural information processing systems foundation, 2023. (Advances in Neural Information Processing Systems).

@inproceedings{99bca375b51144cfb0f6af1b8f2806e5,

title = "LEGALBENCH: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models",

abstract = "The advent of large language models (LLMs) and their adoption by the legal community has given rise to the question: what types of legal reasoning can LLMs perform? To enable greater study of this question, we present LEGALBENCH: a collaboratively constructed legal reasoning benchmark consisting of 162 tasks covering six different types of legal reasoning. LEGALBENCH was built through an interdisciplinary process, in which we collected tasks designed and hand-crafted by legal professionals. Because these subject matter experts took a leading role in construction, tasks either measure legal reasoning capabilities that are practically useful, or measure reasoning skills that lawyers find interesting. To enable cross-disciplinary conversations about LLMs in the law, we additionally show how popular legal frameworks for describing legal reasoning—which distinguish between its many forms—correspond to LEGALBENCH tasks, thus giving lawyers and LLM developers a common vocabulary. This paper describes LEGALBENCH, presents an empirical evaluation of 20 open-source and commercial LLMs, and illustrates the types of research explorations LEGALBENCH enables.",

author = "Neel Guha and Julian Nyarko and Ho, \{Daniel E.\} and Christopher R{\'e} and Adam Chilton and Aditya Narayana and Alex Chohlas-Wood and Austin Peters and Brandon Waldon and Rockmore, \{Daniel N.\} and Diego Zambrano and Dmitry Talisman and Enam Hoque and Faiz Surani and Frank Fagan and Galit Sarfaty and Dickinson, \{Gregory M.\} and Haggai Porat and Jason Hegland and Jessica Wu and Joe Nudell and Joel Niklaus and John Nay and Choi, \{Jonathan H.\} and Kevin Tobia and Margaret Hagan and Megan Ma and Michael Livermore and Nikon Rasumov-Rahe and Nils Holzenberger and Noam Kolt and Peter Henderson and Sean Rehaag and Sharad Goel and Shang Gao and Spencer Williams and Sunny Gandhi and Tom Zur and Varun Iyer and Zehua Li",

note = "Publisher Copyright: {\textcopyright} 2023 Neural information processing systems foundation. All rights reserved.; 37th Conference on Neural Information Processing Systems, NeurIPS 2023 ; Conference date: 10-12-2023 Through 16-12-2023",

year = "2023",

month = jan,

day = "1",

language = "English",

series = "Advances in Neural Information Processing Systems",

publisher = "Neural information processing systems foundation",

editor = "A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine",

booktitle = "Advances in Neural Information Processing Systems 36 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023",

}

Guha, N, Nyarko, J, Ho, DE, Ré, C, Chilton, A, Narayana, A, Chohlas-Wood, A, Peters, A, Waldon, B, Rockmore, DN, Zambrano, D, Talisman, D, Hoque, E, Surani, F, Fagan, F, Sarfaty, G, Dickinson, GM, Porat, H, Hegland, J, Wu, J, Nudell, J, Niklaus, J, Nay, J, Choi, JH, Tobia, K, Hagan, M, Ma, M, Livermore, M, Rasumov-Rahe, N, Holzenberger, N, Kolt, N, Henderson, P, Rehaag, S, Goel, S, Gao, S, Williams, S, Gandhi, S, Zur, T, Iyer, V & Li, Z 2023, LEGALBENCH: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models. in A Oh, T Neumann, A Globerson, K Saenko, M Hardt & S Levine (eds), Advances in Neural Information Processing Systems 36 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023. Advances in Neural Information Processing Systems, vol. 36, Neural information processing systems foundation, 37th Conference on Neural Information Processing Systems, NeurIPS 2023, New Orleans, United States, 10/12/23.

LEGALBENCH: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models. / Guha, Neel; Nyarko, Julian; Ho, Daniel E. et al.
Advances in Neural Information Processing Systems 36 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023. ed. / A. Oh; T. Neumann; A. Globerson; K. Saenko; M. Hardt; S. Levine. Neural information processing systems foundation, 2023. (Advances in Neural Information Processing Systems; Vol. 36).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - LEGALBENCH

T2 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023

AU - Guha, Neel

AU - Nyarko, Julian

AU - Ho, Daniel E.

AU - Ré, Christopher

AU - Chilton, Adam

AU - Narayana, Aditya

AU - Chohlas-Wood, Alex

AU - Peters, Austin

AU - Waldon, Brandon

AU - Rockmore, Daniel N.

AU - Zambrano, Diego

AU - Talisman, Dmitry

AU - Hoque, Enam

AU - Surani, Faiz

AU - Fagan, Frank

AU - Sarfaty, Galit

AU - Dickinson, Gregory M.

AU - Porat, Haggai

AU - Hegland, Jason

AU - Wu, Jessica

AU - Nudell, Joe

AU - Niklaus, Joel

AU - Nay, John

AU - Choi, Jonathan H.

AU - Tobia, Kevin

AU - Hagan, Margaret

AU - Ma, Megan

AU - Livermore, Michael

AU - Rasumov-Rahe, Nikon

AU - Holzenberger, Nils

AU - Kolt, Noam

AU - Henderson, Peter

AU - Rehaag, Sean

AU - Goel, Sharad

AU - Gao, Shang

AU - Williams, Spencer

AU - Gandhi, Sunny

AU - Zur, Tom

AU - Iyer, Varun

AU - Li, Zehua

PY - 2023/1/1

Y1 - 2023/1/1

N2 - The advent of large language models (LLMs) and their adoption by the legal community has given rise to the question: what types of legal reasoning can LLMs perform? To enable greater study of this question, we present LEGALBENCH: a collaboratively constructed legal reasoning benchmark consisting of 162 tasks covering six different types of legal reasoning. LEGALBENCH was built through an interdisciplinary process, in which we collected tasks designed and hand-crafted by legal professionals. Because these subject matter experts took a leading role in construction, tasks either measure legal reasoning capabilities that are practically useful, or measure reasoning skills that lawyers find interesting. To enable cross-disciplinary conversations about LLMs in the law, we additionally show how popular legal frameworks for describing legal reasoning—which distinguish between its many forms—correspond to LEGALBENCH tasks, thus giving lawyers and LLM developers a common vocabulary. This paper describes LEGALBENCH, presents an empirical evaluation of 20 open-source and commercial LLMs, and illustrates the types of research explorations LEGALBENCH enables.

AB - The advent of large language models (LLMs) and their adoption by the legal community has given rise to the question: what types of legal reasoning can LLMs perform? To enable greater study of this question, we present LEGALBENCH: a collaboratively constructed legal reasoning benchmark consisting of 162 tasks covering six different types of legal reasoning. LEGALBENCH was built through an interdisciplinary process, in which we collected tasks designed and hand-crafted by legal professionals. Because these subject matter experts took a leading role in construction, tasks either measure legal reasoning capabilities that are practically useful, or measure reasoning skills that lawyers find interesting. To enable cross-disciplinary conversations about LLMs in the law, we additionally show how popular legal frameworks for describing legal reasoning—which distinguish between its many forms—correspond to LEGALBENCH tasks, thus giving lawyers and LLM developers a common vocabulary. This paper describes LEGALBENCH, presents an empirical evaluation of 20 open-source and commercial LLMs, and illustrates the types of research explorations LEGALBENCH enables.

UR - https://www.scopus.com/pages/publications/85204658613

M3 - Conference contribution

AN - SCOPUS:85204658613

T3 - Advances in Neural Information Processing Systems

BT - Advances in Neural Information Processing Systems 36 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023

A2 - Oh, A.

A2 - Neumann, T.

A2 - Globerson, A.

A2 - Saenko, K.

A2 - Hardt, M.

A2 - Levine, S.

PB - Neural information processing systems foundation

Y2 - 10 December 2023 through 16 December 2023

ER -

Guha N, Nyarko J, Ho DE, Ré C, Chilton A, Narayana A et al. LEGALBENCH: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models. In Oh A, Neumann T, Globerson A, Saenko K, Hardt M, Levine S, editors, Advances in Neural Information Processing Systems 36 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023. Neural information processing systems foundation. 2023. (Advances in Neural Information Processing Systems).

LEGALBENCH: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models

Abstract

Publication series

Conference

Other files and links

Fingerprint

Cite this