Legal Analytics

Definition

Natural language processing (NLP) is the study of how computers handle language. Legal analytics applies artificial intelligence and NLP to legal documents such as laws, court decisions, contracts, and privacy policies (Lippi et al., 2019; Zhong et al., 2020). It finds the parts of a document, the arguments of a court, the unfair terms of a contract, and the likely outcome of a case. These documents are long and hard to read. Legal analytics tools can support consumers, lawyers, judges, and public agencies.

Core concepts

Structure

Legal documents are long and lack explicit markup of their parts. Systems first segment a raw document into its relevant parts, such as the facts, the requests of a party, or the arguments of a lawyer. Each sentence can also receive a rhetorical role, the function it plays in the document.

FACTSREQUESTARGUMENTS

Arguments

Argument mining retrieves the chains of premises and conclusions behind a decision, which show the reasoning of a court. Legal argument mining uses domain-specific component types, such as factual and legal premises. It also labels each legal premise with its argument scheme, the pattern of reasoning it follows, such as reasoning from a rule or a precedent.

ConclusionPremisefactualPremiselegal · rulesupports

Outcome

Outcome prediction predicts the decision of a court from the text of a case. For instance, a system predicts whether the court upholds or rejects the request of a party.

RequestOUTCOMEUpheldRejected

Compliance

Compliance analysis checks whether a document respects the law. Systems flag terms of service clauses that may be unfair to consumers, and check privacy policies against the General Data Protection Regulation (GDPR), the EU data protection law.

GDPR CHECK✓Required information✓Lawful processing✗Clear languagemetmetnot met

What is legal annotation?

Most legal analytics methods start from annotated documents. Legal experts mark the parts of a document that matter for a legal question and give each part a label. An annotation schema fixes these parts, their labels, and their links. Annotation guidelines tell the annotators how to apply the schema. The agreement between annotators measures how reliable the labels are. Models trained on such annotations learn to produce the same structure for new documents.

The examples below follow the four core concepts. For structure, clauses of privacy policies are annotated with their parts, and rules derived from the GDPR rate how informative each clause is. For arguments, court decisions are annotated with chains of premises and conclusions. For outcome, decisions are annotated with the requests of the parties, their claims and arguments, and the final ruling. For compliance, terms of service are annotated with the category and the fairness level of each clause. The last example follows one annotated clause across two languages.

Under the EU General Data Protection Regulation (GDPR), a privacy policy must say which categories of personal data it processes. Experts mark each clause about data with its category of data, the specification that introduces a list, and the subcategories in that list. Each mark is open when its terms are vague or its list is open-ended, and closed when they are precise or exhaustive. Annotation rules combine these marks to rate each clause as sufficiently or insufficiently informative.

TikTok privacy policy

C1Category · openProfile Information. C2We collect Category · closedinformation that you provide when you set up an account, Specification · opensuch as Subcategory · closedyour date of birth, Subcategory · closedusername, Subcategory · closedemail address and/or Subcategory · closedtelephone number, and Subcategory · closedpassword. C3You can add Category · openother information to your profile, Specification · opensuch as Subcategory · opena bio or Subcategory · closeda profile photo.

Merchants, Payment and Transaction Fulfillment Providers. C4We receive Category · openinformation about you from merchants as well as payment and transaction fulfillment providers, Specification · opensuch as Subcategory · closedpayment confirmation details, and Subcategory · closedinformation about the delivery of products you have purchased through our shopping features.

CategorySpecificationSubcategory OpenClosed

2 Clauses and rules

ClauseCategorySpecificationSubcategories
C1L2 · R1opennonenone
C2L1 · R2closedopenclosed5 closed
C3L2 · R2openopenopen1 open · 1 closed
C4L2 · R2openopenclosed2 closed
Rules
L1 · R1closednonenone
L1 · R2closedanyany
L1 · R3openclosedclosed
L2 · R1opennonenone
L2 · R2openopenany
L2 · R3openclosedopen

L1sufficiently informativeL2insufficiently informative

4 clauses about personal data: 1 sufficiently informative3 insufficiently informative

Clauses from the TikTok privacy policy of 19 November 2023, with the expert annotation used in Grundler et al. (2025). The rules are those of Table 2 of the paper.

In court decisions, legal premises state rules, precedents, or principles, and factual premises describe the case. Each legal premise is also labelled with the argument scheme it follows.

1 The text

[...] FP1 Factual premiseHowever, it must be pointed out that that argument is based on a misreading of that judgment. [...] LP1 Legal premise · Precedent, classificationIt is apparent from that judgment that the fact of coming from a compulsory levy is, on the contrary, sufficient to identify State resources (see, to that effect, [...]). [...] FP2 Factual premiseThe first part of the fourth ground of appeal must therefore be rejected as unfounded.

2 Argument graph

Legal premise LP1 Precedent, classification
Factual premise FP1
Factual premise FP2
  • Legal premise LP1 "It is apparent from that judgment that the fact of coming from a compulsory levy is, on the contrary, sufficient to identify State resources" supports Factual premise FP1 "However, it must be pointed out that that argument is based on a misreading of that judgment."
  • Factual premise FP1 "However, it must be pointed out that that argument is based on a misreading of that judgment." supports Factual premise FP2 "The first part of the fourth ground of appeal must therefore be rejected as unfounded."
Paragraph 46 of Case C-850/19 P of the Court of Justice of the European Union, shortened, as annotated by Santin et al. (2023).

Tax decisions are annotated with the request of each party, the claims that ground it, the arguments that support the claims, and the decision of the court on the request. Outcome prediction uses these annotated parts to predict whether the court upholds or rejects a request.

1 The text

[...] R1 RequestThe first-instance decision was appealed by the tax office before this Regional Tax Commission, C1 Claimwhich insisted that its assessment was correct A1 Argumentbecause the analysis of the company revealed undeclared revenues, assessed inductively, and the disallowance of non-deductible costs. [...] For these reasons, [the Commission] D1 Decision · Upheldupholds the appeal of the tax office and reverses the first-instance decision.

2 Argument graph

Decision D1 Upheld
Request R1
Claim C1
Argument A1
  • Decision D1 "upholds the appeal of the tax office and reverses the first-instance decision." upholds Request R1 "The first-instance decision was appealed by the tax office before this Regional Tax Commission,"
  • Claim C1 "which insisted that its assessment was correct" supports Request R1 "The first-instance decision was appealed by the tax office before this Regional Tax Commission,"
  • Argument A1 "because the analysis of the company revealed undeclared revenues, assessed inductively, and the disallowance of non-deductible costs." supports Claim C1 "which insisted that its assessment was correct"
Decision 362/2018 of an Italian Regional Tax Commission from the italianVAT corpus, shortened, as annotated by Galli et al. (2022). The English text is our translation of the Italian original.

Online terms of service often contain clauses that may be unfair to consumers under EU law. Annotators mark each clause that falls into a category of potential unfairness, such as unilateral termination, and grade it as clearly fair, potentially unfair, or clearly unfair.

Academia.edu terms of service

Modification

Unilateral change · Potentially unfairAcademia.edu reserves the right, at its sole discretion, to modify the Site, Services and these Terms, at any time and without prior notice. If we modify these Terms we will post the modification on the Site or provide you with notice of the modification. We will also update the “Last Updated Date” at the top of these Terms. Contract by using · Potentially unfairBy continuing to access or use the Site or Services after we have posted a modification on the Site or have provided you with notice of a modification, you are indicating that you agree to be bound by the modified Terms. If the modified Terms are not acceptable to you, your only recourse is to cease using the Site and Services.

Termination and Account Cancellation

Unilateral termination · Clearly unfairAcademia.edu reserves the right, at its sole discretion, to discontinue or terminate the Site and Services and to terminate these Terms, at any time and without prior notice. Unilateral termination · Clearly unfairIf you breach any of these Terms, Academia.edu will have the right to suspend or disable your Account or terminate these Terms, at its sole discretion and without prior notice to you. Unilateral termination · Clearly unfairAcademia.edu reserves the right to revoke your access to and use of the Site, Services and Collective Content at any time, with or without cause. You may cancel your Account at any time by visiting your Account Settings page and clicking on “Remove” or by sending an email to feedback@academia.edu.

Two sections of the Academia.edu terms of service, as annotated by legal experts in Lippi et al. (2019). Detect shows these expert labels, which the CLAUDETTE classifiers learn to reproduce.

5 of 9 sentences flagged as unfair: 3 clearly unfair2 potentially unfair

Annotation projection copies the labels of an English contract to the same contract in another language, so that a classifier can be trained there without new annotation. The example shows why sentence-level projection is hard: one English sentence matches two German ones.

1 The texts

English

AC1 Annotated clause · content removal, unilateral terminationWe reserve the right to delete or disable Content alleged to violate copyright laws or these Terms and reserve the right to terminate the account(s) of violators.

German

MC1 Matching clause · content removalWir behalten uns das Recht vor, Inhalte zu löschen oder zu deaktivieren, die vorgeblich gegen Urheberrechtsgesetze oder diese Bedingungen verstoßen. MC2 Matching clause · unilateral terminationZusätzlich behalten wir uns das Recht vor, den/die Account(s) des Beschuldigten zu sperren.

2 Clause alignment

Annotated clause AC1 content removal, unilateral termination
Matching clause MC1
Matching clause MC2
  • Annotated clause AC1 "We reserve the right to delete or disable Content alleged to violate copyright laws or these Terms and reserve the right to terminate the account(s) of violators." content removal Matching clause MC1 "Wir behalten uns das Recht vor, Inhalte zu löschen oder zu deaktivieren, die vorgeblich gegen Urheberrechtsgesetze oder diese Bedingungen verstoßen."
  • Annotated clause AC1 "We reserve the right to delete or disable Content alleged to violate copyright laws or these Terms and reserve the right to terminate the account(s) of violators." unilateral termination Matching clause MC2 "Zusätzlich behalten wir uns das Recht vor, den/die Account(s) des Beschuldigten zu sperren."
Example 4.1 of Galassi et al. (2020), from the Box.com terms of service. Both versions show the annotation of legal experts.

Research areas

Legal documents are long, lack explicit markup of their parts, and use legal jargon. Systems label their sentences, names, topics, and key statements so that later tasks can build on them.

Rhetorical role prediction
Given a court decision split into sentences, the task is to label each sentence with its rhetorical role, such as preamble, facts, or arguments. A rhetorical role is the function that a sentence plays in the document.
Legal named entity recognition
Given a legal document, the task is to find named entities and label each with its type. Named entities are mentions of specific entities, such as the petitioner, the respondent, the court, or a statute.
Legal topic classification
Given a law or another legal document, the task is to assign one or more topic labels from a fixed list of categories.
Extraction of legal principles
Given a court decision, the task is to find the passages where the court states a principle of law or a general interpretation.

Outcome prediction links the text of a case to the final decision of the court. Studies rely either on features that describe the case or on the text of the decisions.

Judgment outcome prediction
Given the description of a case or an appeal, the task is to predict whether the court upholds or rejects it.
Request outcome prediction
Given a request by a party, with its claims and arguments, the task is to predict whether the court upholds or rejects that request.
Violation prediction
Given the facts of a case before the European Court of Human Rights, the task is to predict whether the court finds a violation. A violation is a breach of an article of the European Convention on Human Rights.
Unanimity prediction
Given the description of a case, the task is to predict whether the judges reached the decision unanimously.

Consumers rarely read terms of service, the contracts that govern online services. Systems flag clauses that may be unfair and check privacy policies against data protection law.

Unfair clause detection
Given a sentence from a terms of service document, the task is to say whether it contains a potentially unfair clause. A clause is unfair when it creates a significant imbalance in rights and obligations, to the detriment of the consumer.
Unfairness category classification
Given a potentially unfair clause, the task is to assign its category, such as limitation of liability or unilateral termination. Some datasets also grade each clause as clearly fair, potentially unfair, or clearly unfair.
Explaining unfairness
Given a clause classified as unfair, the task is to retrieve the legal rationales that explain why it may be unfair. Rationales are short justifications written by legal experts.
Privacy policy assessment
Given a privacy policy, the task is to check each clause against the General Data Protection Regulation (GDPR), the EU data protection law. The check covers required information, lawful processing, and clear language.
Data practice identification
Given a privacy policy, the task is to identify the data processing practices it describes, such as data collection, transfer, and sharing.

Most legal language tools are built for English, while the European Union works in 24 official languages. Researchers transfer annotations, translate documents, and compare models on multilingual benchmarks.

Annotation projection
Given a document annotated in one language and its version in another language, the task is to transfer the labels onto the matching sentences. The two versions may not be exact translations.
Transfer through machine translation
Given legal documents in a new language, the task is to reuse an English system through machine translation. Either the training documents or each query document is translated.
Cross-lingual transfer
Given a model trained on one language, the task is to apply it to another language without new labels. It relies on representations shared across languages, such as multilingual embeddings.
Multilingual benchmarking
Given a collection of legal datasets in many languages, the task is to compare models with shared aggregate scores.

Large language models (LLMs), such as ChatGPT or GPT-4, can address a task from an instruction and a few examples. Their results on legal tasks are mixed, so studies compare them with smaller models trained on labelled data.

Prompting for unfair clause detection
Given a terms of service clause and an instruction, the task is to have an LLM say whether the clause is unfair. The prompt may include a few examples.
Example selection for legal argument mining
Given a legal argument mining task and a pool of labelled examples, the task is to choose which examples the LLM sees. The choice can change for each new input.
LLM annotation for training data
Given court decisions without labels, the task is to have an LLM label them. A smaller model is then trained on these labels and compared with the LLM.
Legal benchmarks for LLMs
Given a shared set of legal tasks written by experts, the task is to measure how well LLMs reason about law with few examples.

Our focus

Legal argumentation

We study how machines can find and classify the arguments that courts write in their decisions. This gives structured access to the reasoning behind a judgment.

Current

Built from the definitions in Section 4.1 of the paper. Dynamic graph-based selection picks new demonstrations for each query, the sentence to classify. A graph links training examples whose embeddings are close. Nodes far from the query are dropped. The Louvain method splits the remaining nodes into communities, and the node with the highest PageRank score is taken from each one. The picked examples go into the few-shot prompt of the LLM, which outputs the class label.QUERYSentence to classifyDYNAMIC GRAPH SELECTIONTraining graphclose pairs linkedNear queryfar nodes droppedCommunitiesLouvain methodOne per grouptop PageRankLLMfew-shot promptclass label

Dynamic Demonstrations Selection for Few-Shot Legal Argument Mining

Argument Mining and Empirical Legal Research Workshop (AMELR) · 2025

Choosing few-shot examples per query by similarity lets LLMs mine legal arguments in EU court rulings as well as or better than expert-chosen examples.

Past work

  • AMELIA - Argument Mining Evaluation on Legal documents in ItAlian: A CALAMITA Challenge
  • Argumentation Structure Prediction in CJEU Decisions on Fiscal State Aid
  • Detecting Arguments in CJEU Decisions on Fiscal State Aid
Read more

Interpretability

We build models that explain their decisions with the reasons a legal expert would give. This can help users check a prediction and decide whether to trust the tool.

Past work

  • Combining Transformers with Natural Language Explanations
  • Detecting and explaining unfairness in consumer contracts through memory networks
  • Explaining Potentially Unfair Clauses to the Consumer with the CLAUDETTE tool
  • Deep Learning for Detecting and Explaining Unfairness in Consumer Contracts
Read more

Unfair clause detection

We build systems that find clauses in online contracts and privacy policies that may harm consumers. Few people read these documents, so automatic tools can help consumers and lawyers.

Current

Built from Sections 4 and 5 of the paper. Each Terms of Service sentence goes to two kinds of detectors. Seven prompted LLMs, never trained on the task, use one of three prompt strategies: single-prompt, multi-prompt, or pipeline. Eight supervised baselines, a linear SVM and seven transformers, are fine-tuned on the 85 training documents. Both answer for which of the nine unfairness categories the sentence is unfair. On the test split, the best LLM, Qwen-32B with the pipeline, reaches a test macro-F1 of 0.72 (Table 4), shown as 72 on a 0 to 100 scale, and the best fine-tuned model, Custom-LegalBERT, reaches 77.9 (Table 3).TERMS OF SERVICEToS sentence7 PROMPTED LLMSSingle-promptMulti-promptPipeline8 BASELINESFine-tunedSVM, transformerson 85 training ToS9 categoriesBEST TEST MACRO-F1, 0 TO 10072Qwen-32B, pipeline77.9Custom-LegalBERT

Is It Worth Using LLMs for Unfair Clause Detection in Terms of Service?

International Conference on Artificial Intelligence and Law (ICAIL) · 2025

Seven large language models, prompted in three ways, face fine-tuned classifiers at spotting unfair clauses in Terms of Service, and the fine-tuned classifiers still win.

Past work

  • Detecting Vague Clauses in Italian Privacy Policies Using Transformers, LLMs, and Cross-Lingual Techniques
  • The force awakens: Artificial intelligence for consumer law
  • Consumer protection requires artificial intelligence
  • CLAUDETTE: an automated detector of potentially unfair clauses in online terms of service
  • Towards Consumer-Empowering Artificial Intelligence
  • Automated Processing of Privacy Policies Under the EU General Data Protection Regulation
Read more

Cross-linguality

We study how to move legal annotations and models from one language to another. In the European Union, the same document often exists in several languages.

Past work

  • Unfair clause detection in terms of service across multiple languages
  • LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain
  • Cross-lingual Annotation Projection in Legal Texts
Read more

Reading court decisions

We study how machines can find the parts and key statements of a court decision. Legal experts are few and costly, so we also study how to build training data with less of their time.

Current

Built from Sections 2 and 3 and Table 4 of the paper. Of 101 CJEU rulings on VAT, legal experts label 21 for validation and testing, and DeepSeek-R1 labels 80 for training. A fine-tuned BERT-based classifier learns from the LLM labels and labels each paragraph as a Judicial Interpretative Formula (JIF) or not. On the test split, LEGAL-BERT reaches 0.76 macro F1, against 0.75 for DeepSeek-R1.101 CJEU RULINGS ON VATExperts21 rulingsDeepSeek-R180 rulingsevaluationtrainClassifierfine-tuned BERTJIF or not JIFone label per paragraphMACRO F1, TEST SPLITLEGAL-BERT 0.76DeepSeek-R1 0.75

Automated Extraction of Judicial Interpretative Formulas in EU Case Law on VAT

International Conference on Legal Knowledge and Information Systems (JURIX) · 2025

A pipeline trains small classifiers on LLM-labelled EU tax rulings to find the Court's interpretative paragraphs, matching the LLM on expert-labelled test data.

Past work

  • TeamUnibo at SemEval-2023 Task 6: A transformer based approach to Rhetorical Roles prediction and NER in Legal Texts
  • Combining WordNet and Word Embeddings in Data Augmentation for Legal Texts
Read more

Legal argumentation

Argument mining is the automatic detection of arguments and their parts in text. In a court decision, a premise is a reason, and a conclusion is the point the court reaches from it. Premises can be legal, such as a rule or a precedent, or factual, such as an event in the case. An argument scheme is the pattern of reasoning that a legal premise follows, such as reasoning from a precedent. We ask whether models can recover these parts and their links, and whether similar decisions share similar arguments.

Current

Past work

Interpretability

A model is interpretable when people can understand why it gives a certain output. A legal rationale is a short justification, written by legal experts, of why a clause is unfair. A memory-augmented neural network stores such texts in an external memory and uses them while it classifies. We ask whether rationales can explain predictions to users without lowering accuracy.

Past work

Unfair clause detection

Terms of service are the contracts that govern the relation between an online platform and its users. A clause is potentially unfair when it may break consumer protection law, for instance by letting the provider change the contract alone. Unfair clause detection finds such clauses, and unfairness category classification assigns each one to a category. We ask how well machine learning, from traditional classifiers to LLMs, performs on this task and on related checks of privacy policies.

Current

Past work

Cross-linguality

Cross-linguality is the use of data or models in one language to solve a task in another language. Annotation projection copies the labels of a document onto its version in another language. The two versions are often not exact translations, so sentences do not match one to one. We ask whether projection or machine translation can replace the costly annotation of a new corpus for each language.

Past work

Reading court decisions

A court decision has functional parts, such as the facts, the arguments of the parties, and the ruling. Rhetorical role prediction labels each sentence with the part it belongs to. Legal named entity recognition finds names such as the petitioner, the court, or the statute. Some sentences state general principles that later decisions cite. In decisions of the Court of Justice of the European Union, these sentences are called judicial interpretative formulas. We ask how to find these elements and how to train models when labelled data are scarce.

Current

Past work