Automated Extraction of Judicial Interpretative Formulas in EU Case Law on VAT

Wed, 01 Jan 2025 00:00:00 +0000

Research setting

The Court of Justice of the European Union (CJEU) interprets EU law for the courts of the member states. A national court can send it a question, and the CJEU answers with a decision called a preliminary ruling. In its answers, the Court writes general interpretative statements that later rulings cite again and again: the paper calls them Judicial Interpretative Formulas (JIFs). JIFs are especially important in value added tax (VAT), where EU rules leave some aspects to national choice and CJEU rulings have shaped much of the law.

The task, as defined by the paper’s annotation guidelines (Section 2). Only the part of the ruling that holds the Court’s answer is read. Each numbered paragraph gets one label. The paragraph descriptions are illustrative, built from the four kinds of statement that make a paragraph a JIF.

The task is paragraph classification: a paragraph is a JIF when it contains at least one of the four kinds of statement shown above, and not a JIF otherwise.

Motivation

Finding JIFs by hand is slow work for legal experts, and no automatic method existed for VAT case law. Prior work on extracting legal principles has two weak points. Methods that learn from data need many examples labelled by experts, which are costly. Methods that prompt a large language model (LLM), a general text generator driven by instructions, often lack the stability, transparency, and reproducibility that legal work requires.

Approach	No costly expert labels for training	Stable and reproducible at deployment	Applied to CJEU VAT rulings
Machine learning on expert labels (Shulayeva et al., 2017; Valvoda and Ray, 2017)	no	yes	no
Prompting an LLM (Molinari et al., 2024)	yes	no	no
This paper (hybrid pipeline) This work	yes	yes	yes

How the paper positions its pipeline against prior work on extracting legal principles (Section 1 of the paper).

Objective. Build and evaluate a pipeline that extracts JIFs from CJEU rulings on VAT, using an LLM to label training data and a smaller, task-specific classifier to do the extraction.

Approach

Two legal experts wrote annotation guidelines and refined them on real rulings until no ambiguities remained. With the final guidelines, they labelled rulings independently and agreed almost perfectly. In total, experts labelled 21 rulings, which are kept for validation and testing only. This way, no model is ever evaluated on machine labels. An LLM, prompted with the guidelines, labelled 80 more rulings for training. Classifiers based on BERT, a family of compact language models that are fine-tuned (further trained) for one task, then learn from these machine labels.

Guidelines Experts define what a JIF is, with examples.
Expert labels 21 rulings for validation and testing.
LLM labels DeepSeek-R1 labels 80 rulings for training.
Classifier Fine-tuned on the LLM labels, it labels every paragraph as JIF or not JIF.

The extraction pipeline (Sections 2 and 3 of the paper). The last step trains a dedicated classifier on labels produced by an LLM instead of by experts.

Split	Rulings	Labels	JIF	Not JIF
Train	80	LLM	1015	1037
Validation	11	Experts	147	164
Test	10	Experts	142	164

The corpus, with paragraph counts per class. Source: Table 3 of the paper. Splits are made by ruling, so the paragraphs of one ruling never end up in two splits.

Results

Each model labels the paragraphs of the 10 test rulings, and its labels are compared with the expert labels. Precision is the share of predicted JIFs that are real JIFs. Recall is the share of real JIFs that the model finds. F1 combines the two, and macro F1 averages F1 over the two classes, JIF and not JIF. Agreement between the two experts is measured with Cohen’s kappa, where 1 means full agreement.

The three BERT-based models are DistilRoBERTa, DeBERTa, and LEGAL-BERT, a BERT model pre-trained on legal text. A simpler classifier, LinearSVC on word statistics (TF-IDF), is also trained on the LLM labels.

DeepSeek-R1 labelled the training data because it had the best F1 against the experts on the validation split. It scored 0.807, against 0.785 for Claude-3.7-sonnet and 0.727 for Gemini-1.5-pro. It also runs on the test split as a reference, next to a random baseline and a majority baseline that always predicts not JIF. Each BERT-based model is trained three times with different random seeds, and its scores are averaged. LinearSVC is deterministic, so it needs no repeated runs.

0.96Cohen’s kappa between the two experts, on two rulings labelled independently with the final guidelines
0.76macro F1 of LEGAL-BERT and DistilRoBERTa, against 0.75 for DeepSeek-R1
0.008standard deviation of LEGAL-BERT over three seed runs, the lowest of the BERT-based models

Macro F1 on the test split (higher is better)

Majority baseline 0.35
Random baseline 0.49
DeepSeek-R1 0.75
LinearSVC This work 0.72
LEGAL-BERT This work 0.76 ± 0.008
DistilRoBERTa This work 0.76 ± 0.020
DeBERTa This work 0.74 ± 0.012

Paragraph classification on the test split. Source: Table 4 of the paper. Whiskers show the standard deviation over three seed runs.

Show as table

System	Macro F1 on the test split
Majority baseline	0.35
Random baseline	0.49
DeepSeek-R1	0.75
LinearSVC (this work)	0.72
LEGAL-BERT (this work)	0.76 ± 0.008
DistilRoBERTa (this work)	0.76 ± 0.020
DeBERTa (this work)	0.74 ± 0.012

Model	JIF precision	JIF recall	JIF F1
DeepSeek-R1	0.73	0.73	0.74
LinearSVC	0.68	0.75	0.71
LEGAL-BERT	0.73	0.80	0.76
DistilRoBERTa	0.75	0.75	0.75
DeBERTa	0.69	0.82	0.75

Scores on the JIF class. Source: Table 4 of the paper.

Classifiers trained only on machine labels perform on par with the LLM that produced those labels, and two of them slightly beat it. Like the LLM, they tend to find more JIFs than they should rather than miss them, which the paper suggests may come from learning DeepSeek-R1’s labels. Their scores vary little across seed runs. LinearSVC is not far behind, which suggests that word choice is a strong cue for JIFs.

Takeaways