Large Language Models | Language Technologies Lab

Hate Speech Detection with Argumentative Reasoning

Mon, 02 Mar 2026 00:00:00 +0000

Context

Hate speech often relies on implicit content: meaning that a text suggests without stating it. It also relies on subtle reasoning nuances. Argument mining is the problem of automatically detecting and extracting arguments from text. An argument usually combines a premise and a claim. The premise is a fact, and it supports the claim, a subjective conclusion.

Objective

This proposal applies argumentative reasoning to hate speech, to make its implicit content explicit. The resulting hate speech detection systems should be more interpretable and more user-friendly.

Argumentative reasoning makes the implicit content of a text explicit as a premise that supports a claim, toward hate speech detection. The schema uses no example text.

Rationalization via LLMs

Mon, 02 Mar 2026 00:00:00 +0000

Context

Selective rationalization explains a prediction with a rationale: a short part of the input text that a user can read and that faithfully shows what the prediction is based on. The Select-then-Predict Pipeline (SPP) builds this into the model: a generator selects the rationale, and a predictor classifies the rationale only.

Objective

Large language models (LLMs) can select a rationale and predict a label in a single answer when a prompt asks them to. This proposal evaluates how well they perform selective rationalization through prompting, compared with traditional SPP models.

The two settings this proposal compares, on an illustrative sentence that comes from no dataset. In the SPP, the label comes from the rationale alone. The prompted LLM reads the whole text and returns both.

Text Classification with Guidelines Only

Mon, 02 Mar 2026 00:00:00 +0000

Context

The standard way to train a machine learning model is to give it an annotated dataset: data paired with class labels. Annotators build this dataset after training on annotation guidelines, the rules that define how to assign the classes. They label the data with classes from a given class set. The model learns from the labels alone, without the guidelines, so it can learn any mapping from data to classes that fits the given data.

Objective

This proposal gives the annotation guidelines directly to the model. During training, the model receives the guidelines and unlabeled data, but no class labels.

The two settings this proposal compares. In standard training, only the annotators learn from the guidelines; in guidelines-only training, the model receives the guidelines and no class labels.

Transformers and LLMs for the Detection and Classification of Unfair Clauses

Mon, 02 Mar 2026 00:00:00 +0000

Context

Terms of Service and Privacy Policies are the documents that people accept when they use an online service. Some of their clauses can be unfair to the people who accept them. For several years, the lab has built tools that detect unfair clauses in such documents written in English, in the CLAUDETTE and PRIMA projects listed on the Projects page.

Objective

This proposal applies new methods to detect unfair clauses and to classify them. The current focus is on large language models (LLMs), and alternative techniques are also of interest.

The task on three illustrative clauses written for this page: the model marks each clause as fair or unfair, then classifies each unfair clause.

Sustainable Italian LLM Evaluation: Community Perspectives and Methodological Guidelines

Mon, 01 Sep 2025 00:00:00 +0000

Research setting

A large language model (LLM) is a neural network trained on large amounts of text that answers questions and follows instructions in natural language. Benchmarking an LLM means running it on tasks with known answers and scoring its outputs, so that models can be compared over time. Several LLMs trained for Italian have appeared recently. This position paper does not test a new model. It collects the experience of three Italian research groups and proposes how the community should evaluate these models.

Task item Marco Rossi è nato a Milano nel 1985. Which Milano?
Prompt The item with four options, from Milano, Texas to Milano, Italy.
LLM The model under evaluation.
Answer The model outputs one option letter, here B.
Score The answer is compared with the correct option.

One benchmark item, built from the multiple-choice example in Section 3.1 of the paper: the model must tell which Milano a sentence refers to.

Terms used on this page:

Native benchmark: a benchmark written directly in Italian, as opposed to one translated from English.
Probability-based evaluation reads the probability that the model gives to each predefined answer option and takes the most likely one as its choice.
Generative evaluation lets the model write a free-form answer and then checks that answer against criteria or a reference.
LLM-as-a-Judge is a generative evaluation in which another LLM grades the free-form answers.
Zero-shot and few-shot prompting give the model no solved examples or a few solved examples before the question.
A benchmark is saturated when the best models solve it almost fully, so it no longer separates them.

Motivation

Italian has rich inflection, a flexible word order, clitic pronouns, dropped subjects, and many regional varieties. A direct translation of English benchmarks cannot capture these features, nor Italian culture and institutions. Recent Italian LLM benchmarks are valuable. They are also isolated efforts, with limits in methodology, scope, sustainability, and coordination. The paper maps them onto its own evaluation framework and finds whole skill areas that no one tests.

Framework	Morphology	Pragmatics	Register	Ling. instructions	Task gen.	Cross-ling.
ITA-Bench	no	yes	no	no	no	yes
Evalita-LLM	no	yes	no	yes	no	no
ITALIC	yes	no	no	no	no	no
This paper This work	yes	yes	yes	yes	yes	yes

Coverage of six evaluation dimensions by three Italian LLM evaluation frameworks, from Table 1 of the paper. Ling. instructions is linguistic instruction following, Task gen. is task generalization, and Cross-ling. is cross-linguistic transfer. For this paper, a check means that its framework defines the dimension: the paper itself releases no dataset.

Objective. Give the Italian NLP community a shared framework and practical guidelines for evaluating Italian LLMs, and a plan to keep that evaluation running over time.

Approach

The paper organizes its guidelines around four questions. The first question sets the skills to test, through the three dimensions shown in the figure at the top of this page. The other three questions cover how to run a test, where to get the data, and how to keep the whole effort alive.

What to benchmark Linguistic competence, domain and knowledge, task generalization.
How to benchmark Task format, scoring strategy, and prompt variations.
Where to benchmark Translated, adapted, or native Italian data.
Sustainable benchmarking A living benchmark run by the community.

The four questions of the paper (Sections 2 to 5). The last one is the community plan that gives the paper its title.

Question	Main guideline of the paper
What	Task designers state which dimensions their task covers, so effort goes to the dimensions that few tasks test.
How: format	Multiple-choice questions are cheap and simple to score but far from real use. Open-ended generation is closer to real use but harder and costlier to score.
How: prompting	Zero-shot evaluation is the main measure. Few-shot prompting serves as a supplementary variation and a strong baseline.
How: variation	Prompts vary in register, explicitness, cultural framing, and random changes such as the order of options, to test robustness.
Where	Native development comes first for tasks that are critical for Italian. Translation and adaptation fill the remaining gaps.
Sustainable	Saturated or stale tasks leave the benchmark, and open models come first. Each model is reported with its parameters and training tokens. A rotating steering committee governs the effort, which runs on national compute such as CINECA’s Leonardo supercomputer.

Guidelines from Sections 2 to 5 of the paper.

Results

The paper reports no new model experiments. Its findings come from three sources: a coverage analysis of public datasets against the framework (Table 1), the authors’ experience with Italian benchmarks, and a preliminary cost analysis.

25 + 3public datasets and evaluation frameworks mapped onto the framework
3-5xmore compute for generative evaluation than for probability-based evaluation
500-750GPU hours per quarter to evaluate 10 models on 50 tasks

Coverage of 25 public datasets and 3 frameworks (ITA-Bench, Evalita-LLM, and ITALIC) by the dimensions of the framework. A check means covered, a cross means not covered (Table 1 of the paper).

Almost every dataset tests semantics, and many test lexical knowledge or a knowledge domain. No dataset or framework in the table tests register adaptation or task generalization, and only ITA-Bench tests cross-linguistic transfer. The paper calls for more tasks on pragmatics, register, and cross-linguistic transfer.

Translation can also change what a task measures. In the Italian version of a WinoGrande item, grammar alone gives the answer away, because GPS is masculine and mappa (map) is feminine.

Il GPSCandidate e la mappaCandidate mi hanno aiutato a tornare a casa. Mi sono perso quando laFeminine ___ è stata capovolta.Feminine

A WinoGrande item translated into Italian, from Section 4 of the paper. The correct answer is mappa. The highlighted feminine forms rule out GPS without any common sense.

Two more observations come from the authors’ experience. Italian models often perform better when prompted in English with the instruction to answer in Italian. This suggests that they benefit from higher-quality English training data. For scoring free-form answers, no open-weight LLM-as-a-Judge model is trained explicitly for Italian yet.

Takeaways

Know what a benchmark measures.Mapping each task onto the three dimensions of the framework shows the skills that Italian benchmarks miss, such as register adaptation.
Translation is not enough.A translated task can lose difficulty or cultural content. Native Italian tasks matter most for cultural knowledge, idioms, and pragmatic language use.
Benchmarks need to stay alive.A living benchmark retires saturated tasks and puts open models first. Run on national compute under community governance, it can keep Italian evaluation rigorous over time.

Detecting Vague Clauses in Italian Privacy Policies Using Transformers, LLMs, and Cross-Lingual Techniques

Wed, 01 Jan 2025 00:00:00 +0000

Research setting

In the European Union, the General Data Protection Regulation (GDPR) requires a privacy policy to tell users which personal data a service processes. The paper studies the clauses of a policy that describe these categories of data, called CAT clauses. A legal expert labels each CAT clause as sufficiently informative, when the data are described in full, or insufficiently informative, also called vague, in all other cases.

The label follows from three annotated elements. A Category is the term that names the data, such as “geolocation data”. A Specification is an optional list introduced by words such as “such as” or “namely”, and each item of the list is a Subcategory, such as “GPS information”. Each element is either Open, when it is vague or open-ended, or Closed, when it is concrete or exhaustive.

We receive information about youOpen category from merchants as well as payment and transaction fulfillment providers, such asOpen specification payment confirmation details,Closed subcategory and information about the delivery of products you have purchasedClosed subcategory through our shopping features.

An insufficiently informative clause from the TikTok policy (Figure 1 of the paper). The Category is Open and the list is open-ended, so the clause is vague even though its two Subcategories are Closed. The Specification element spans the whole list; only its opening words are marked here.

Category	Specification	Subcategories	Label
Closed	none	none	Sufficient
Closed	any	any	Sufficient
Open	Closed	Closed	Sufficient
Open	none	none	Insufficient
Open	Open	any	Insufficient
Open	Closed	Open	Insufficient

The six rules that label a clause as sufficiently or insufficiently informative. “Any” means Open or Closed. Source: Table 2 of the paper.

Motivation

Empirical studies show that many privacy policies still fall short of the GDPR, and vague descriptions of data leave users unable to know what is collected about them. Automatic tools could flag such clauses, but previous work on insufficiently informative clauses covers English documents only. The EU has 24 official languages, and annotated legal corpora are scarce in many of them. So it matters whether a resource built for one language can serve another.

Work	Italian privacy policies	Insufficiently informative clauses	Cross-lingual transfer
Vague terms (Lebanoff and Liu, 2018; Malik et al., 2023)	no	partially	no
Vague data clauses in English (Grundler et al., 2024)	no	yes	no
Cross-lingual legal NLP (Chalkidis et al., 2021; Galassi et al., 2024)	no	no	yes
This paper This work	yes	yes	yes

How the paper relates to the prior work it discusses (Section 2 of the paper).

Objective. Find the most effective way to detect insufficiently informative clauses in Italian, and test whether annotated data can be transferred between English and Italian in both directions.

Approach

The authors build the first Italian corpus for this task: 30 privacy policies of the same online platforms as the 30 English policies of Grundler et al. (2024). A legal expert, an Italian native speaker trained on the English corpus, annotated it. In the last training round on English policies, the expert agreed with the English gold labels with a Cohen’s κ of 0.77 on the label. The corpus has 5,862 sentences and 875 CAT clauses, of which 220 are sufficiently informative.

Task	Input	Output
CAT classification	A sentence	CAT or not
Informativeness classification	A CAT clause	Sufficiently or insufficiently informative
Category and Subcategory detection	A CAT clause	The spans of Categories and Subcategories
Specification detection	A CAT clause	The spans of Specifications
Type classification	A Category or Subcategory	Open or Closed
Kind classification	A Category or Subcategory	One of 30 kinds of data, such as payment

The six tasks, each solved independently (Section 4.1 of the paper).

Two families of models solve each task. Four BERT models (three Italian models and one multilingual model) are fine-tuned on the Italian training split. Two large language models (LLMs), Gemini 2.0 Flash and Llama-3.1-8B-Instruct, answer from a prompt. The prompt has no examples (zero-shot) or 15 examples (few-shot, 30 for Kind), picked at random or as the nearest neighbours of the input. The cross-lingual experiments use no Italian training data at all.

English corpus 30 annotated English policies.
Transfer technique Three ways to reach Italian.
- Training set translation Translate the training data, train Italian models.
- Multilingual model Train a multilingual model in English.
- Test set translation Translate Italian test clauses to English.
Italian test set Scored on the new Italian corpus.

Transfer from English to Italian (Section 4.3.1 of the paper). Translation uses the open-source Opus-MT models. The reverse direction applies the same idea with the Italian corpus as training data and the English test set.

Results

Each system is scored on the Italian test split, which holds 20% of the policies, with the F1 score of each class and their average, the macro F1 (from 0 to 1, higher is better). BERT scores are averages over three fine-tuning runs.

0.79macro F1 on informativeness for Italian BERT trained on Italian data
0.80macro F1 on informativeness with no Italian training data (DeBERTa, test set translation)
0.84macro F1 on informativeness in English for Italian BERT trained on the Italian corpus

Macro F1 for informativeness classification (higher is better)

Gemini zero-shot 0.64
Llama zero-shot 0.64
Llama few-shot (nearest examples) 0.71
Gemini few-shot (nearest examples) 0.72
Multilingual BERT trained in English 0.65 ± 0.109
UmBERTo 0.78 ± 0.043
Italian BERT 0.79 ± 0.016
DeBERTa, test set translation 0.80 ± 0.005

Informativeness classification on the Italian test set. Source: Table 3 of the paper (whiskers show the standard deviation over three runs).

Show as table

System	Macro F1 for informativeness classification
Gemini zero-shot	0.64
Llama zero-shot	0.64
Llama few-shot (nearest examples)	0.71
Gemini few-shot (nearest examples)	0.72
Multilingual BERT trained in English	0.65 ± 0.109
UmBERTo	0.78 ± 0.043
Italian BERT	0.79 ± 0.016
DeBERTa, test set translation	0.80 ± 0.005

Setting	Model	CAT	Informativeness	Type	Kind
Zero-shot	Gemini	0.85	0.64	0.67	0.56
Few-shot, nearest examples	Gemini	0.86	0.72	0.76	0.54
Fine-tuned on Italian data	Italian BERT	0.86	0.79	0.80	0.44
Multilingual model	BERT multilingual	0.68	0.65	0.73	0.30
Training set translation	UmBERTo	0.85	0.80	0.73	0.32
Test set translation	DeBERTa	0.85	0.80	0.79	0.43

Macro F1 of the four classification tasks on the Italian test set for selected systems. Source: Table 3 of the paper.

Fine-tuned BERT models beat the LLMs on vagueness: Italian BERT reaches 0.79, against 0.72 for the best LLM. The one exception is Kind classification, with 30 classes, where Gemini zero-shot scores best with 0.56. In the detection tasks, Italian BERT and UmBERTo are always the best models, with up to 0.86 macro F1 for Specifications with BIO tags (Table 5 of the paper). Translating the Italian test set and reusing English models reaches the top informativeness score of 0.80, while a multilingual model trained in English drops to 0.65.

Training data for the English test set	Model	CAT	Informativeness	Type	Kind
Original English corpus	LEGAL-BERT	0.86	0.84	0.80	0.46
Italian corpus, translated to English	LEGAL-BERT	0.86	0.80	0.78	0.36
Italian corpus, test set translated to Italian	Italian BERT	0.85	0.84	0.77	0.35

Macro F1 on the English test set. Source: Table 4 of the paper.

In the other direction, models trained on the Italian corpus match the English-trained model in three of the four classification tasks, and fall 10 points behind only on Kind classification.

Takeaways

Fine-tuned BERT models beat LLMs on vagueness.Italian BERT reaches 0.79 macro F1 on informativeness, against 0.72 for the best few-shot LLM. LLMs lead only on Kind classification, which has 30 classes.
English resources can serve Italian.Translating the test set and reusing English models matches training on Italian data, with no new corpus and no new training. A multilingual model trained in English transfers poorly.
The Italian corpus generalizes to English.Trained on Italian data, models match the English-trained model in three of the four classification tasks. The corpus and code are public.

Dynamic Demonstrations Selection for Few-Shot Legal Argument Mining

Wed, 01 Jan 2025 00:00:00 +0000

Research setting

Argument mining (AM) is the automatic detection and classification of arguments in text. In court decisions, an argument is a chain of inferences in which one or more premises lead to a conclusion. The paper works on Demosthenes, a corpus of 40 English decisions by the Court of Justice of the European Union (CJEU) on Fiscal State Aid. Legal experts annotated it at three levels. Each level gives one classification task.

Argumentative sentence A sentence from a CJEU decision.
Argument component

Premise or conclusion.
Premise type

For premises. Factual, legal, or both.
Argument scheme

For legal premises. One or more of five schemes.

The three classification tasks of the paper, one per annotation level of the Demosthenes corpus (Sections 3 and 4).

Terms used on this page:

A factual premise describes events or procedural aspects of the case. A legal premise refers to legal content such as laws, precedents, principles, or their interpretation.
An argument scheme is the pattern of reasoning behind a legal premise. A premise can carry several schemes. The paper uses five:
- Rule: an EU norm applies to the case.
- Precedent: an earlier ruling of the Court applies.
- Authoritative: a statement by an authority, such as an Advocate General, supports the outcome.
- Classification: a legal concept is defined by its conditions.
- Interpretative: a meaning is given to a legal source.
A large language model (LLM) is a general-purpose text model that can solve a task from instructions in its prompt, without training.
Few-shot learning, or in-context learning, adds a few labeled examples to the prompt. These examples are called demonstrations, and a prompt with k of them is a k-shot prompt.
An embedding is a vector that represents the meaning of a sentence. Two sentences with close embeddings are similar.

First, it must be recalled that, according to the Court’s settled case-law,Precedent classification of a national measure as ‘State aid’,Classification within the meaning of Article 107(1) TFEU,Rule requires all the following conditions to be fulfilled …Classification

A legal premise from the corpus that carries three argument schemes at once (Appendix B.3 of the paper). The marked spans follow the rationale that the paper gives for each scheme.

Motivation

LLMs give mixed results in the legal domain, whose language is formal, layered, and often ambiguous. In few-shot learning, the choice of demonstrations shapes performance, and in the legal domain the choice is especially hard. Demonstrations must show the key reasoning patterns and cues, yet stay short because of the token limits of the models. Some prior methods fix one set of demonstrations for all queries. Others pick examples close to the query, but measure diversity only through labels or through the same similarity score.

Selection method	Chosen for each query	Similar to the query	Diverse in content
Static selection by experts or by the LLM (Grundler et al., 2024; Qin et al., 2024)	no	no	not discussed
k-NN selection (Liu et al., 2022)	yes	yes	no
Reinforcement learning selection (Wang et al., 2025)	yes	yes	labels only
Complementary examples (Ye et al., 2023)	yes	yes	partially
This paper (dynamic graph selection) This work	yes	yes	yes

How the paper relates to the demonstration selection methods it discusses (Section 2 of the paper).

Objective. Measure how demonstration selection affects LLMs on legal argument mining. Compare static selection with dynamic selection, including a new graph-based method that also seeks diversity.

Approach

The paper compares two families of selection, shown in the figure at the top of this page. Static selection fixes the demonstrations once. Legal experts choose them with explicit criteria, or the LLM itself chooses them from at most 50 training examples (self-selection). Dynamic selection picks new demonstrations from the training set for each query, the sentence to classify. Dynamic k-NN takes the k training examples whose embeddings are closest to the query. The new dynamic graph method adds diversity, in five steps.

Build a graph Nodes are training examples, linked when close.
Keep near nodes Drop nodes far from the query.
Find groups The Louvain method splits them into dense groups.
Pick one each The most central node, by PageRank.
Prompt the LLM The picks go into the prompt.

Dynamic graph selection (Section 4.1 of the paper). The two distance thresholds are set per task and number of demonstrations, in two configurations called c1 and c2 (Section 4.2).

A balanced variant of both dynamic methods includes about the same number of demonstrations for each class. The LLMs are Llama 3.1 (8B, instruction tuned) and Gemini 2.0 Flash, with 0, 5, or 10 demonstrations. Two trained classifiers serve as references: a LinearSVC on TF-IDF features and a fine-tuned LEGAL-BERT, one model per task.

Results

The paper tests 34 configurations on a held-out split of 8 decisions, with 345 premises and 22 conclusions. Scores are F1 per class and their macro average (macro-F1), from 0 to 1, where higher is better.

0.80argument scheme macro-F1, Gemini 5-shot with k-NN, against 0.68 with static expert
0.84premise type macro-F1, Gemini 10-shot with k-NN, against 0.81 with static expert
14demonstrations chosen by both Gemini and the experts, 11 for argument schemes

Macro-F1 on argument schemes (higher is better)

Gemini zero-shot 0.61
Static expert 0.71
Static self-selection 0.75
Dynamic k-NN 0.80
Dynamic graph c1 This work 0.79
Dynamic graph c2 This work 0.74
LEGAL-BERT (fine-tuned) 0.75
LinearSVC (trained) 0.83

Argument scheme classification with Gemini 2.0 Flash and 10 demonstrations, against zero-shot Gemini and the trained references. Source: Table 2 of the paper.

Show as table

System	Macro-F1 on argument schemes
Gemini zero-shot	0.61
Static expert	0.71
Static self-selection	0.75
Dynamic k-NN	0.80
Dynamic graph c1 (this work)	0.79
Dynamic graph c2 (this work)	0.74
LEGAL-BERT (fine-tuned)	0.75
LinearSVC (trained)	0.83

Model and selection	Argument component	Premise type	Argument scheme
LinearSVC (trained)	0.90	0.82	0.83
LEGAL-BERT (fine-tuned)	0.91	0.89	0.75
Gemini 10-shot, static expert	0.69	0.81	0.71
Gemini 10-shot, static self-selection	0.67	0.80	0.75
Gemini 10-shot, dynamic k-NN	0.65	0.84	0.80
Gemini 10-shot, dynamic k-NN balanced	0.67	0.82	0.80
Gemini 10-shot, dynamic graph c1	0.65	0.79	0.79
Gemini 10-shot, dynamic graph c2	0.67	0.83	0.74

Macro-F1 per task. Source: Table 2 of the paper.

Among the LLM configurations, dynamic k-NN is the best overall in all three tasks, which points to similarity as an important criterion in this domain. At least one configuration of the graph method performs close to k-NN in most settings. Neither c1 nor c2 is better overall. Balancing matters for argument components with Llama: with 5 demonstrations, k-NN drops to 0.14 macro-F1 without balancing and reaches 0.69 with it. The trained references still score best in every task. However, they use the whole training set and one model per task, while one LLM serves all tasks with 5 or 10 demonstrations.

Takeaways

Similarity is the key criterion.Picking demonstrations close to each query was the best LLM setting overall in all three tasks. In the paper’s comparison, it matches or outperforms demonstrations chosen by legal experts.
Expert curation is not required.Comparable or better results come without the cost and effort of manual selection. At least one configuration of the graph method performs close to k-NN, which leaves room to explore its thresholds.
Training still wins.LinearSVC and LEGAL-BERT obtain the best results in all three tasks. The paper plans to test how less labeled data affects each approach.

Generation and Evaluation of English Grammar Multiple-Choice Cloze Exercises

Sun, 01 Dec 2024 00:00:00 +0000

Research setting

Language teachers use multiple-choice cloze (MCC) exercises to train grammar. An MCC exercise has a body, a sentence with a gap; a key, the correct answer for the gap; and distractors, plausible but incorrect alternatives. Good distractors are homogeneous (same syntactic category as the key), plausible, and unambiguous (none of them could replace the key). Writing many such exercises by hand takes time and expert knowledge, so the paper asks a large language model (LLM) to write them.

Thanks to high technology, doctors can betterKey assess patients’ conditions.

An MCC exercise on comparisons from the fine-tuning dataset (Listing 2 of the paper), shown with its key filled in. The learner sees a gap in place of the key and chooses among better, best, good, and well; the last three are the distractors.

Motivation

Most earlier systems take sentences from existing texts and place the gap with fixed rules, such as the first verb or an adjective. Systems that write new sentences still constrain the model, for example with a fixed part-of-speech (POS) sequence, and their output tends to lack creativity. The only English MCC dataset, SC-Ques, has limited diversity, underrepresented topics, and frequent mistakes. Lastly, the field has no agreed automatic metric for the quality of a generated exercise.

Approach	Writes its own sentences	No fixed gap rule or POS sequence
Rule-based pipelines (Sumita et al.; Lin et al.)	no	no
Gap detection with conditional random fields (Goto et al.)	no	yes
Neural gap prediction (Bitew et al.; Matsumori et al.)	no	yes
Generation from a POS sequence, keyword, and topic (Chomphooyod et al.)	yes	no
This paper This work	yes	yes

How the paper relates to the prior work it discusses (Sections 1 and 3).

Objective. Find out how well an LLM can write accurate English grammar MCC exercises without predefined constraints or POS sequences, and measure their quality with an automatic metric checked against a human expert.

Approach

The base Llama 3 model, given a detailed prompt, struggled with some grammar topics and wrote poor distractors. The authors therefore fine-tuned it on a cleaned dataset of exercises, built from SC-Ques in the steps below.

SC-Ques About 300k English sentence completion exercises written by teachers.
Cleaning Drop items with several gaps, fewer than two distractors, or noise.
Topics A rule-based Pattern Matcher links each key to one of 19 grammar topics.
Filtering Remove similar exercises per topic and cap how often one key appears.
Balancing Add synthetic WH-question exercises written with GPT-4.

Building the fine-tuning dataset (Section 5 of the paper). After the removal of similar exercises, 30% of the original data was left.

Each training example pairs a grammar topic and a number of distractors with an exercise in JSON: the full sentence, the sentence with the gap, the key, and the distractors. Two metrics then score the output. Structural compliance (SC) is a rule-based check of the exercise form. The gap must sit in the intended place, the key must have the right grammatical form, and every distractor must be homogeneous with the key. Self-BLEU measures how much the exercises of one topic repeat each other’s word sequences (2 to 5 words long); lower means more diverse.

1. Fine-tune

Grammar topic Data
Llama 3 weights Frozen
LoRA adapters Trained
Exercise Data
Structural check

Llama 3 8B Instruct is quantized to 4-bit precision and kept frozen. Small low-rank (LoRA) adapters on its attention layers learn to write the exercise for a given topic, for 3 epochs, in two hours on one GPU.

2. Generate

Grammar topic Data
Llama 3 weights Frozen
LoRA adapters Frozen
Exercise Data
Structural check

For each of the 19 topics, the model writes 50 exercises with one distractor each, sampling at temperature 0.7.

3. Check

Grammar topic
Llama 3 weights
LoRA adapters
Exercise Data
Structural check Frozen

The structural check tests the gap and key (pertinence) and the distractor (homogeneity). For inflectional topics, the distractor must share the key’s lemma; for free morpheme topics, whose options form a small closed set, it must come from a list of admitted words.

Trained with gradients Kept fixed Data

Training, generation, and checking (Sections 6 to 8 of the paper).

Results

The 950 generated exercises were scored with the automatic metrics and reviewed by a computational linguist with a background in language teaching. The expert judged each exercise correct (EC) only if it was plausible, unambiguous, coherent with common sense, and acceptable (free of stereotypes and inappropriate content). The expert also judged structural compliance, which serves as the gold reference for the automatic check (SC_A).

79%of the 950 exercises judged ready for learners by the expert
95%F1 of the automatic structural check against the expert (precision 98%, recall 91%)
0.07average self-BLEU, a sign of low repetition

Exercise correctness (EC) per grammar topic (higher is better)

articles 0.74
comparison adjectives 0.72
conditional statements 0.66
future simple 0.90
modal verbs 0.70
infinitive and gerund verbs 0.86
passive tenses 0.74
past continuous 0.88
past perfect 0.82
past simple 0.82
personal pronouns 0.74
possessive adjectives 0.72
prepositions 0.72
present continuous 0.88
present perfect 0.84
present simple 0.86
quantifiers 0.84
relative clauses 0.74
WH- question 0.90

Share of exercises judged correct by the expert, per grammar topic. Source: Table 1 of the paper (50 exercises per topic).

Show as table

System	Exercise correctness (EC) per grammar topic
articles	0.74
comparison adjectives	0.72
conditional statements	0.66
future simple	0.90
modal verbs	0.70
infinitive and gerund verbs	0.86
passive tenses	0.74
past continuous	0.88
past perfect	0.82
past simple	0.82
personal pronouns	0.74
possessive adjectives	0.72
prepositions	0.72
present continuous	0.88
present perfect	0.84
present simple	0.86
quantifiers	0.84
relative clauses	0.74
WH- question	0.90

Average over the 19 topics	Value
Structural compliance, automatic check (SC_A)	0.85
Structural compliance, expert (SC_H)	0.92
Self-BLEU (lower is better)	0.07
Exercise correctness, expert (EC)	0.79
Wrong exercises that break common sense	0.75
Wrong exercises with an ambiguous distractor	0.19
Wrong exercises that are not acceptable	0.05
Wrong exercises that are not plausible	0.01

Averages from Table 1 (first four rows) and from the error analysis on wrong exercises in Table 2 of the paper.

Most generated exercises are well formed and varied, and the automatic check agrees closely with the expert. By the structural scores, the model does better on free morpheme topics, where keys and distractors come from a small set of words. On verb tense topics, the Pattern Matcher sometimes mislabels verbs, so the check can wrongly reject an exercise. Among the wrong exercises, a sentence that makes no sense is the most frequent error, and an ambiguous distractor is the second. Biased or trivial exercises are almost absent.

Takeaways

A fine-tuned LLM writes usable grammar exercises.Without predefined constraints or POS sequences, 79% of the generated exercises met every requirement for classroom use. The authors found that high-quality fine-tuning data was a key factor.
Structure can be checked automatically.The rule-based structural check reached 95% F1 against the expert, so it can screen exercises before a human review.
Meaning is the open problem.Most wrong exercises had a sentence that breaks common sense, and some distractors could replace the key, especially for verb tenses.

TWOLAR: A TWO-Step LLM-Augmented Distillation Method for Passage Reranking

Mon, 01 Jan 2024 00:00:00 +0000