Researchers Develop Methodology to Assess the Quality of Legal Representation in Criminal Proceedings

© iStock

Having a good defence attorney in criminal proceedings can largely determine whether a defendant retains their freedom, health and good name. Researchers at HSE University propose a method for predicting an attorney’s performance based on the outcomes of their previous cases. The methodology takes into account the severity of the charges, the complexity of the cases, and the most likely outcome, drawing on judicial statistics.

Anton Kazun, Director of the Institute for Industrial and Market Studies at the HSE Faculty of Economic Sciences, Vadim Devyatnikov, Research Assistant at the Institute, and Mikhail Belov, a PhD student at the University of California, have proposed a methodology for assessing the quality of legal representation in criminal proceedings. They present their approach in Information Asymmetry in the Legal Services Market: Creating a Ranking Using Machine Learning Methods, published in Theoretical Economics.

Anton Kazun

The authors highlight the importance of choosing the right legal representation in criminal proceedings: a skilled defence attorney can help secure a case dismissal, an acquittal or a non-custodial sentence, while ineffective defence can result in a prison sentence for the client. International studies have shown that clients represented by the highest-performing defence attorneys are 14 percentage points less likely to be incarcerated than those represented by attorneys in the lowest performance percentile.

In this study, the researchers developed a methodology for measuring the quality of legal representation by comparing machine-learning predictions based on objective case characteristics with actual case outcomes. Reviews of machine-learning methods for predicting court decisions have confirmed their high accuracy and reliability, suggesting that such models can also be used to assess attorneys’ performance. The difference between the predicted and actual outcomes was interpreted as the attorney’s contribution, with a pattern of favourable outcomes resulting in a higher rating.

Vadim Devyatnikov

The authors examined more than 7 million published decisions in criminal cases issued between 2010 and 2025. They automatically extracted information from the texts on the severity of the crime; the defendant’s characteristics; mitigating and aggravating circumstances; the court and region; the socio-demographic characteristics of the locality; the date of the decision; procedural features; and information about the defence attorney.

Case outcomes were predicted in stages, taking into account the key procedural decision points in each case, which ensured that the predicted and actual outcomes could be compared accurately.

At the first stage, the model predicts the probability of a criminal conviction, including a suspended sentence. Non-conviction outcomes include termination of the case, the return of the case to the prosecutor, reconciliation between the parties, a court-imposed fine, an acquittal, and other outcomes. The models at this stage are trained on the full dataset of criminal cases.

The next stage focuses on a key factor in assessing the severity of the outcome for the defendant: the punishment imposed by the court, such as imprisonment, a suspended sentence, a fine, or compulsory labour. The probability of a custodial sentence is then compared with the probabilities of alternative non-custodial punishments.

Finally, for cases resulting in a custodial sentence, the model predicts the length of the sentence in months, allowing the researchers to assess the attorney’s ability to reduce the severity of the punishment. These models are trained on cases that resulted in a custodial sentence.

The calculation of an attorney’s effectiveness rating begins with determining the ‘raw’ delta for each case—the difference between the actual and predicted outcomes. A positive delta indicates a favourable outcome for the defendant: for example, the case was dismissed even though a conviction was predicted, or the actual prison sentence was shorter than expected. If a favourable outcome is not achieved at the first stage despite its high predicted probability, the attorney’s bonus for securing a more lenient sentence at the next stage is reduced.

To calculate an attorney’s final score, the delta values are weighted by the number of cases at each stage. Based on these final scores, each attorney’s position in the overall distribution is calculated, and their rating is determined by the decile in which they fall.

The authors acknowledge several limitations of their approach related to data availability and methodological assumptions. At present, it is impossible to assess an attorney’s performance at the pre-trial stage, which may lead to underestimating the performance of attorneys who are particularly effective at this stage. Not all court decisions are publicly available, and personal data, including attorneys’ full names, have been removed from some texts. Finally, the involvement of highly skilled attorneys in complex cases may lower their ratings because the methodology does not fully account for case complexity.

The authors emphasise that their ranking system cannot and should not be used as the sole criterion for choosing a defence attorney. They point out extending the analysis to the pre-trial stage and improving the accuracy of outcome predictions as promising directions for further development of the methodology. Another task for future research is to address cases involving multiple defendants.

The proposed methodology forms the basis of hodat.ai, an online service that helps users find a defence attorney based on objective measures of their professional performance.