Dynamic Demonstrations Selection for Few-Shot Legal Argument Mining

Wed, 01 Jan 2025 00:00:00 +0000

Research setting

Argument mining (AM) is the automatic detection and classification of arguments in text. In court decisions, an argument is a chain of inferences in which one or more premises lead to a conclusion. The paper works on Demosthenes, a corpus of 40 English decisions by the Court of Justice of the European Union (CJEU) on Fiscal State Aid. Legal experts annotated it at three levels. Each level gives one classification task.

Argumentative sentence A sentence from a CJEU decision.
Argument component

Premise or conclusion.
Premise type

For premises. Factual, legal, or both.
Argument scheme

For legal premises. One or more of five schemes.

The three classification tasks of the paper, one per annotation level of the Demosthenes corpus (Sections 3 and 4).

Terms used on this page:

A factual premise describes events or procedural aspects of the case. A legal premise refers to legal content such as laws, precedents, principles, or their interpretation.
An argument scheme is the pattern of reasoning behind a legal premise. A premise can carry several schemes. The paper uses five:
- Rule: an EU norm applies to the case.
- Precedent: an earlier ruling of the Court applies.
- Authoritative: a statement by an authority, such as an Advocate General, supports the outcome.
- Classification: a legal concept is defined by its conditions.
- Interpretative: a meaning is given to a legal source.
A large language model (LLM) is a general-purpose text model that can solve a task from instructions in its prompt, without training.
Few-shot learning, or in-context learning, adds a few labeled examples to the prompt. These examples are called demonstrations, and a prompt with k of them is a k-shot prompt.
An embedding is a vector that represents the meaning of a sentence. Two sentences with close embeddings are similar.

First, it must be recalled that, according to the Court’s settled case-law,Precedent classification of a national measure as ‘State aid’,Classification within the meaning of Article 107(1) TFEU,Rule requires all the following conditions to be fulfilled …Classification

A legal premise from the corpus that carries three argument schemes at once (Appendix B.3 of the paper). The marked spans follow the rationale that the paper gives for each scheme.

Motivation

LLMs give mixed results in the legal domain, whose language is formal, layered, and often ambiguous. In few-shot learning, the choice of demonstrations shapes performance, and in the legal domain the choice is especially hard. Demonstrations must show the key reasoning patterns and cues, yet stay short because of the token limits of the models. Some prior methods fix one set of demonstrations for all queries. Others pick examples close to the query, but measure diversity only through labels or through the same similarity score.

Selection method	Chosen for each query	Similar to the query	Diverse in content
Static selection by experts or by the LLM (Grundler et al., 2024; Qin et al., 2024)	no	no	not discussed
k-NN selection (Liu et al., 2022)	yes	yes	no
Reinforcement learning selection (Wang et al., 2025)	yes	yes	labels only
Complementary examples (Ye et al., 2023)	yes	yes	partially
This paper (dynamic graph selection) This work	yes	yes	yes

How the paper relates to the demonstration selection methods it discusses (Section 2 of the paper).

Objective. Measure how demonstration selection affects LLMs on legal argument mining. Compare static selection with dynamic selection, including a new graph-based method that also seeks diversity.

Approach

The paper compares two families of selection, shown in the figure at the top of this page. Static selection fixes the demonstrations once. Legal experts choose them with explicit criteria, or the LLM itself chooses them from at most 50 training examples (self-selection). Dynamic selection picks new demonstrations from the training set for each query, the sentence to classify. Dynamic k-NN takes the k training examples whose embeddings are closest to the query. The new dynamic graph method adds diversity, in five steps.

Build a graph Nodes are training examples, linked when close.
Keep near nodes Drop nodes far from the query.
Find groups The Louvain method splits them into dense groups.
Pick one each The most central node, by PageRank.
Prompt the LLM The picks go into the prompt.

Dynamic graph selection (Section 4.1 of the paper). The two distance thresholds are set per task and number of demonstrations, in two configurations called c1 and c2 (Section 4.2).

A balanced variant of both dynamic methods includes about the same number of demonstrations for each class. The LLMs are Llama 3.1 (8B, instruction tuned) and Gemini 2.0 Flash, with 0, 5, or 10 demonstrations. Two trained classifiers serve as references: a LinearSVC on TF-IDF features and a fine-tuned LEGAL-BERT, one model per task.

Results

The paper tests 34 configurations on a held-out split of 8 decisions, with 345 premises and 22 conclusions. Scores are F1 per class and their macro average (macro-F1), from 0 to 1, where higher is better.

0.80argument scheme macro-F1, Gemini 5-shot with k-NN, against 0.68 with static expert
0.84premise type macro-F1, Gemini 10-shot with k-NN, against 0.81 with static expert
14demonstrations chosen by both Gemini and the experts, 11 for argument schemes

Macro-F1 on argument schemes (higher is better)

Gemini zero-shot 0.61
Static expert 0.71
Static self-selection 0.75
Dynamic k-NN 0.80
Dynamic graph c1 This work 0.79
Dynamic graph c2 This work 0.74
LEGAL-BERT (fine-tuned) 0.75
LinearSVC (trained) 0.83

Argument scheme classification with Gemini 2.0 Flash and 10 demonstrations, against zero-shot Gemini and the trained references. Source: Table 2 of the paper.

Show as table

System	Macro-F1 on argument schemes
Gemini zero-shot	0.61
Static expert	0.71
Static self-selection	0.75
Dynamic k-NN	0.80
Dynamic graph c1 (this work)	0.79
Dynamic graph c2 (this work)	0.74
LEGAL-BERT (fine-tuned)	0.75
LinearSVC (trained)	0.83

Model and selection	Argument component	Premise type	Argument scheme
LinearSVC (trained)	0.90	0.82	0.83
LEGAL-BERT (fine-tuned)	0.91	0.89	0.75
Gemini 10-shot, static expert	0.69	0.81	0.71
Gemini 10-shot, static self-selection	0.67	0.80	0.75
Gemini 10-shot, dynamic k-NN	0.65	0.84	0.80
Gemini 10-shot, dynamic k-NN balanced	0.67	0.82	0.80
Gemini 10-shot, dynamic graph c1	0.65	0.79	0.79
Gemini 10-shot, dynamic graph c2	0.67	0.83	0.74

Macro-F1 per task. Source: Table 2 of the paper.

Among the LLM configurations, dynamic k-NN is the best overall in all three tasks, which points to similarity as an important criterion in this domain. At least one configuration of the graph method performs close to k-NN in most settings. Neither c1 nor c2 is better overall. Balancing matters for argument components with Llama: with 5 demonstrations, k-NN drops to 0.14 macro-F1 without balancing and reaches 0.69 with it. The trained references still score best in every task. However, they use the whole training set and one model per task, while one LLM serves all tasks with 5 or 10 demonstrations.

Takeaways