AMELIA - Argument Mining Evaluation on Legal documents in ItAlian: A CALAMITA Challenge

Sun, 01 Dec 2024 00:00:00 +0000

Research setting

Argument mining is the automatic analysis of arguments in text. An argument is a set of connected pieces of text, called argument components. A premise is a component that gives a reason for the argument, and a conclusion is the point that the premises argue for. CALAMITA is a collection of tests of the abilities of large language models (LLMs) in Italian. AMELIA is its challenge on legal arguments: a model classifies argument components from Italian tax decisions in three tasks.

The three AMELIA tasks on a real argument component of the dataset, shown in the English translation of Appendix A of the paper. Each task applies to the output of the previous one. Schema drawn from the task definitions in Section 2 of the paper.

The second task asks whether a premise is factual, legal, or both. A factual premise describes the events or the procedure of the case, and a legal premise states legal content such as rules, precedents, or principles. The third task labels each legal premise with one or more argument schemes, the patterns of legal reasoning it follows:

Scheme	The legal premise…
Rule	refers to codified law, such as an article or its text.
Precedent	refers to a previous decision of the Court of Cassation or the European Court of Justice.
Classification	defines a legal concept and qualifies a fact as having its properties.
Interpretative	states a new interpretation by the Court, which creates a new precedent.
Principle	refers explicitly to a principle of law, such as the principle of proportionality.

The five argument schemes for tax law. Source: Section 2 of the paper.

Motivation

Recognizing arguments is a first step toward reasoning, so even basic argument mining tasks tell us how well LLMs understand the logical relations expressed in text. Legal texts are a hard test, because legal professionals build and take apart arguments in formal documents with technical language and complex syntax. Many argument mining datasets exist in English, but resources for Italian are scarce. Legal language processing in Italian has focused on other tasks, and the authors know of no earlier argument mining challenge on Italian legal documents.

Resource	Italian	Legal documents	Argument mining
English argument mining datasets (Habernal et al., 2024; Niculae et al., 2017; Poudyal et al., 2020; Mayer et al., 2021; Accuosto and Saggion, 2020)	no	partially	yes
Demosthenes, decisions of the Court of Justice of the European Union (Grundler et al., 2022)	no	yes	yes
Italian argument mining on news comments and tweets (Basile et al., 2016; Lai et al., 2018)	yes	no	yes
Italian legal NLP, such as article retrieval, outcome prediction, contracts, and summarization	yes	yes	no
This paper (AMELIA) This work	yes	yes	yes

How AMELIA relates to the work the paper discusses (Section 1 of the paper).

Objective. Measure how well LLMs recognize and classify legal arguments in Italian, with three classification tasks on real tax decisions annotated by experts.

Approach

The data come from real decisions of Italian Regional Tax Commissions on Value Added Tax (VAT), from 2010 to 2022. The annotation guidelines adapt those of Demosthenes, an English corpus of the same authors, and keep its classes. One change was needed: in the Italian decisions a component may span several sentences, and one sentence may hold several components. The tasks assume that the argument components are already found, so a model only has to classify them.

Collect 225 Italian VAT decisions, from 2010 to 2022.
Extract Argument text from the reasoning part of each decision.
Mask Names, addresses, and places hidden; dates keep only the year.
Annotate Four tax law experts add the labels of the three tasks.
Split Three splits by decision; zero-shot and few-shot prompts.

How the AMELIA dataset was built and how models are tested (Section 3 of the paper).

The split follows the decisions, so all components of a decision fall in the same split, at a ratio of about 60:20:20. For each task the paper gives two prompts in Italian. The zero-shot prompt holds only the class definitions, and the few-shot prompt adds labeled examples from the training split. The dataset is public on Hugging Face under the CC BY 4.0 license.

Results

The paper presents the challenge and its data and reports no model results. Its results are the composition of the dataset (Table 1) and the evaluation protocol. The classes are highly unbalanced, so models are scored with the macro F1 score. It averages the F1 score over the classes, so rare classes count as much as frequent ones.

225Italian decisions on VAT
2910premises in task 1
401conclusions in task 1

Split	Rule	Precedent	Interpretative	Principle	Classification
Train	350	264	224	92	51
Validation	107	82	83	21	22
Test	118	73	67	31	27
Total	575	419	374	144	100

Legal premises labeled with each argument scheme. A premise may have several schemes. Source: Table 1 of the paper.

Split	Decisions	Premises	Conclusions	Factual	Legal
Train	135	1866	242	1254	812
Validation	44	528	81	315	266
Test	46	516	78	323	260
Total	225	2910	401	1892	1338

Composition of the dataset. Source: Table 1 of the paper.

Premises far outnumber conclusions, and the five schemes range from 575 legal premises for Rule to 100 for Classification. This is why the paper uses macro F1 and also reports the F1 score of each class. As a reference for difficulty only, the best macro F1 scores on Demosthenes were 0.88 for component classification, 0.85 for premise type, and 0.75 for scheme classification. The paper stresses that these scores are not directly comparable with AMELIA.

Takeaways