Forecasting the Supreme Court: A Comparative Analysis of Machine Learning Algorithms on Petitioner vs. Appellee Outcomes

Abstract

Since its inception, Supreme Court decisions have impacted American laws and life. The ability to predict the high court’s decisions, known as quantitative legal prediction, would be of interest to those in the legal profession and the general public. While much research has been conducted on quantitative legal prediction, for various foreign high courts, the few experiments that have specifically addressed United States Supreme Court cases are now outdated, have been prone to overfitting, or were based on limited datasets. Our work and experimentation attempt to predict case outcomes while addressing the shortcomings of past research. In this work, we deployed several machine learning algorithms to predict whether the petitioner or appellee will win a Supreme Court case and compared the algorithms based on their prediction accuracy. Finally, we embarked on identifying which case features have the greatest predictive impact on the winner of a case. Using four machine learning algorithms (Random Forest, XGBoost, LightGBM, and Multilayer Perceptron) we trained, evaluated, and tested the predictive accuracy on the Washington University School of Law dataset of over 8,000 Supreme Court cases that were litigated between 1946 to 2016. Success was measured via a model’s accuracy, AUROC, and the associated weighted F1 score. Three of the four algorithms achieved accuracy, AUROC, and weighted F1 score in the mid-0.70s with LightGBM being the most accurate. The three case features that most influence LightGBM’s performance are the reason the Supreme Court granted a petition for certiorari, the category of the appellee, and the category of the petitioner. High performing algorithms and models such as the ones we have deployed could provide some predictive insight to individuals, lawyers, and policymakers that may be affected by Supreme Court decisions. Future research directions may include training the algorithms using semantically meaningful textual data or additional case variables.