Italian | Language Technologies Lab

Do Large Language Models understand how to be judges?

Wed, 01 Oct 2025 00:00:00 +0000

Research setting

An LLM judge is a large language model (LLM) that reads a generated text and grades it, instead of a human reviewer. This paper studies LLM judges for news summaries. The judge receives an Italian news article and a summary, and it gives a score from 1 to 5 with a written explanation. The score follows a rubric, a list of quality criteria in which each criterion has a definition, sub-criteria, and a description of every score level. A good judge gives scores close to those of a human expert.

Article and summary An Italian news article and one summary of it.
Rubric prompt One criterion, its sub-criteria, and examples.
LLM judge Reads both texts and applies the criterion.
Score and explanation A score from 1 to 5 and the reason for it.

The judging task studied in the paper. The prompt asks for a JSON answer with a score and an explanation in Italian (Appendix A of the paper).

Terms used on this page:

Few-shot prompting shows the model a few solved examples inside the prompt, without any training.
MAE (mean absolute error) is the average distance between the model score and the expert score. Lower is better.
Spearman’s ρ measures whether the model ranks the summaries in the same order as the expert. It ranges from -1 to 1, and 0 means no relation.
Positive bias means that a judge gives higher scores than the expert.

Motivation

Overlap metrics such as ROUGE and BLEU count shared words, so they miss whether a summary is faithful or useful to its reader. Human evaluation captures these qualities, but it is costly, inconsistent, and hard to scale. LLM judges promise consistent and cheap scores, but only if they understand the language and the structure of the criteria they receive. Public summarization benchmarks are also a weak test, because their reference summaries often miss editorial standards and models may have seen them during training.

Approach	No judge training	One model per judgment	Criteria with explicit sub-criteria	Test texts unseen in training
Prompted judges	yes	yes	not discussed	not discussed
Fine-tuned judges	no	yes	not discussed	not discussed
Multi-agent judges	not discussed	no	not discussed	not discussed
G-Eval criteria	not discussed	not discussed	partially	not discussed
This paper This work	yes	yes	yes	yes

How the paper positions itself against the related work it discusses (Sections 2 and 3 of the paper).

Objective. Test whether LLMs, prompted with a detailed editorial rubric, understand its criteria well enough to score summaries and explain their scores as a human expert would.

Approach

The rubric comes from the practice of professional editors and has five criteria.

Criterion	What the judge checks
Coherence	The summary reads as a logical, connected narrative.
Consistency	Every factual claim is supported by the article.
Fluency	Grammar, spelling, punctuation, and phrasing are correct.
Relevance	The summary keeps the core points and drops filler.
Ordering	Key points follow the order of the article.

The authors pick 10 Italian news articles published after the training cutoff of every tested model. GPT-4o writes 50 summaries, each made to show a set quality level on one criterion, and humans review and correct them. An expert scores every summary on all five criteria, which gives 250 reference scores. The judges are 24 models: open models of five families (DeepSeek, Gemma 3, Llama 3, Phi 4, Qwen 3) from 0.6B to 14B parameters, and seven OpenAI GPT models. No judge is fine-tuned: each one only sees the rubric prompt with few-shot examples.

1. Write the summaries

Article Data
GPT-4o Frozen
Summary Human
Expert
LLM judge

GPT-4o summarizes each article several times, with a target quality level for one criterion in each summary. Humans check and correct the results.

2. Score with the expert

Article Data
GPT-4o
Summary Data
Expert Human
LLM judge

An expert annotator scores each of the 50 summaries from 1 to 5 on all five criteria.

3. Score with the judges

Article Data
GPT-4o
Summary Data
Expert
LLM judge Frozen

Each judge model receives the rubric prompt and few-shot examples, then scores every summary on every criterion.

4. Compare the scores

Article
GPT-4o
Summary Data
Expert Data
LLM judge Data

The judge scores are compared with the expert scores through MAE and Spearman’s ρ.

Human work Frozen Data

How the test is built and run. No model is trained at any step (Sections 3 and 4 of the paper).

Results

The authors compare each judge with the expert over the 250 scores, through MAE and Spearman’s ρ. They also compare the judges with one another, through Spearman’s ρ between their scores.

0.277highest Spearman’s ρ with the expert (GPT o4 mini)
1.04lowest MAE on the 1 to 5 scale (GPT o4 mini)
0.810Spearman’s ρ between two judges, GPT-4.1 and GPT-4o

Family	Model	MAE	Spearman’s ρ	p-value
DeepSeek	1.5B	1.47	0.008	0.894
	7B	1.71	0.031	0.627
	8B	1.21	-0.018	0.773
	14B	1.05	0.031	0.628
Gemma 3	1B	1.84	-0.078	0.222
	4B	1.38	-0.179	0.005
	12B	1.53	0.010	0.878
Llama 3	1B	1.34	-0.005	0.936
	3B	1.51	0.027	0.666
	8B	1.47	-0.039	0.535
Phi 4	3.8B	1.44	0.219	0.000
	14B	1.31	0.010	0.874
Qwen 3	0.6B	1.36	0.137	0.031
	1.7B	1.24	0.040	0.528
	4B	1.54	0.048	0.453
	8B	1.49	0.065	0.303
	14B	1.46	0.114	0.072
GPT	4o	1.41	0.064	0.314
	4o mini	1.15	-0.032	0.611
	4.1	1.25	0.095	0.136
	4.1 mini	1.42	0.173	0.006
	4.1 nano	1.24	0.034	0.594
	o3 mini	1.31	0.238	0.000
	o4 mini	1.04	0.277	0.000

Agreement of each judge with the expert (Table 1 of the paper).

Agreement with the expert is weak. The best Spearman’s ρ is 0.277, and most values are close to 0. A larger model often makes smaller errors, as in DeepSeek, where MAE drops from 1.47 at 1.5B to 1.05 at 14B, but its ranking does not improve. In the Phi 4 family the smaller model ranks better, with ρ = 0.219 at 3.8B against 0.010 at 14B. Large judges agree more with one another than with the expert, up to ρ = 0.810 between GPT-4.1 and GPT-4o.

Score counts of the expert (blue) and of three judges (orange) for fluency and coherence. The judges pile up on scores 4 and 5 (Figure 6 of the paper).

Almost every judge gives higher scores than the expert, most of all for fluency and coherence. The few-shot examples cover the whole 1 to 5 scale in equal amounts, so the authors trace this positive bias to training data and alignment rather than to the prompt.

Takeaways

Smaller errors do not mean better ranking.Larger models often score closer to the expert on average, but they do not order the summaries more like the expert. The best Spearman’s ρ is 0.277.
Judges are too generous.Almost every judge rates summaries higher than the expert, most of all for fluency and coherence. Their high scores do not carry the weight of an expert score.
Agreement among judges grows with size.Small models often disagree with one another, while large models, mostly within one family, converge. Consistency across several judges appears only above a certain size.

Detecting Vague Clauses in Italian Privacy Policies Using Transformers, LLMs, and Cross-Lingual Techniques

Wed, 01 Jan 2025 00:00:00 +0000

Research setting

In the European Union, the General Data Protection Regulation (GDPR) requires a privacy policy to tell users which personal data a service processes. The paper studies the clauses of a policy that describe these categories of data, called CAT clauses. A legal expert labels each CAT clause as sufficiently informative, when the data are described in full, or insufficiently informative, also called vague, in all other cases.

The label follows from three annotated elements. A Category is the term that names the data, such as “geolocation data”. A Specification is an optional list introduced by words such as “such as” or “namely”, and each item of the list is a Subcategory, such as “GPS information”. Each element is either Open, when it is vague or open-ended, or Closed, when it is concrete or exhaustive.

We receive information about youOpen category from merchants as well as payment and transaction fulfillment providers, such asOpen specification payment confirmation details,Closed subcategory and information about the delivery of products you have purchasedClosed subcategory through our shopping features.

An insufficiently informative clause from the TikTok policy (Figure 1 of the paper). The Category is Open and the list is open-ended, so the clause is vague even though its two Subcategories are Closed. The Specification element spans the whole list; only its opening words are marked here.

Category	Specification	Subcategories	Label
Closed	none	none	Sufficient
Closed	any	any	Sufficient
Open	Closed	Closed	Sufficient
Open	none	none	Insufficient
Open	Open	any	Insufficient
Open	Closed	Open	Insufficient

The six rules that label a clause as sufficiently or insufficiently informative. “Any” means Open or Closed. Source: Table 2 of the paper.

Motivation

Empirical studies show that many privacy policies still fall short of the GDPR, and vague descriptions of data leave users unable to know what is collected about them. Automatic tools could flag such clauses, but previous work on insufficiently informative clauses covers English documents only. The EU has 24 official languages, and annotated legal corpora are scarce in many of them. So it matters whether a resource built for one language can serve another.

Work	Italian privacy policies	Insufficiently informative clauses	Cross-lingual transfer
Vague terms (Lebanoff and Liu, 2018; Malik et al., 2023)	no	partially	no
Vague data clauses in English (Grundler et al., 2024)	no	yes	no
Cross-lingual legal NLP (Chalkidis et al., 2021; Galassi et al., 2024)	no	no	yes
This paper This work	yes	yes	yes

How the paper relates to the prior work it discusses (Section 2 of the paper).

Objective. Find the most effective way to detect insufficiently informative clauses in Italian, and test whether annotated data can be transferred between English and Italian in both directions.

Approach

The authors build the first Italian corpus for this task: 30 privacy policies of the same online platforms as the 30 English policies of Grundler et al. (2024). A legal expert, an Italian native speaker trained on the English corpus, annotated it. In the last training round on English policies, the expert agreed with the English gold labels with a Cohen’s κ of 0.77 on the label. The corpus has 5,862 sentences and 875 CAT clauses, of which 220 are sufficiently informative.

Task	Input	Output
CAT classification	A sentence	CAT or not
Informativeness classification	A CAT clause	Sufficiently or insufficiently informative
Category and Subcategory detection	A CAT clause	The spans of Categories and Subcategories
Specification detection	A CAT clause	The spans of Specifications
Type classification	A Category or Subcategory	Open or Closed
Kind classification	A Category or Subcategory	One of 30 kinds of data, such as payment

The six tasks, each solved independently (Section 4.1 of the paper).

Two families of models solve each task. Four BERT models (three Italian models and one multilingual model) are fine-tuned on the Italian training split. Two large language models (LLMs), Gemini 2.0 Flash and Llama-3.1-8B-Instruct, answer from a prompt. The prompt has no examples (zero-shot) or 15 examples (few-shot, 30 for Kind), picked at random or as the nearest neighbours of the input. The cross-lingual experiments use no Italian training data at all.

English corpus 30 annotated English policies.
Transfer technique Three ways to reach Italian.
- Training set translation Translate the training data, train Italian models.
- Multilingual model Train a multilingual model in English.
- Test set translation Translate Italian test clauses to English.
Italian test set Scored on the new Italian corpus.

Transfer from English to Italian (Section 4.3.1 of the paper). Translation uses the open-source Opus-MT models. The reverse direction applies the same idea with the Italian corpus as training data and the English test set.

Results

Each system is scored on the Italian test split, which holds 20% of the policies, with the F1 score of each class and their average, the macro F1 (from 0 to 1, higher is better). BERT scores are averages over three fine-tuning runs.

0.79macro F1 on informativeness for Italian BERT trained on Italian data
0.80macro F1 on informativeness with no Italian training data (DeBERTa, test set translation)
0.84macro F1 on informativeness in English for Italian BERT trained on the Italian corpus

Macro F1 for informativeness classification (higher is better)

Gemini zero-shot 0.64
Llama zero-shot 0.64
Llama few-shot (nearest examples) 0.71
Gemini few-shot (nearest examples) 0.72
Multilingual BERT trained in English 0.65 ± 0.109
UmBERTo 0.78 ± 0.043
Italian BERT 0.79 ± 0.016
DeBERTa, test set translation 0.80 ± 0.005

Informativeness classification on the Italian test set. Source: Table 3 of the paper (whiskers show the standard deviation over three runs).

Show as table

System	Macro F1 for informativeness classification
Gemini zero-shot	0.64
Llama zero-shot	0.64
Llama few-shot (nearest examples)	0.71
Gemini few-shot (nearest examples)	0.72
Multilingual BERT trained in English	0.65 ± 0.109
UmBERTo	0.78 ± 0.043
Italian BERT	0.79 ± 0.016
DeBERTa, test set translation	0.80 ± 0.005

Setting	Model	CAT	Informativeness	Type	Kind
Zero-shot	Gemini	0.85	0.64	0.67	0.56
Few-shot, nearest examples	Gemini	0.86	0.72	0.76	0.54
Fine-tuned on Italian data	Italian BERT	0.86	0.79	0.80	0.44
Multilingual model	BERT multilingual	0.68	0.65	0.73	0.30
Training set translation	UmBERTo	0.85	0.80	0.73	0.32
Test set translation	DeBERTa	0.85	0.80	0.79	0.43

Macro F1 of the four classification tasks on the Italian test set for selected systems. Source: Table 3 of the paper.

Fine-tuned BERT models beat the LLMs on vagueness: Italian BERT reaches 0.79, against 0.72 for the best LLM. The one exception is Kind classification, with 30 classes, where Gemini zero-shot scores best with 0.56. In the detection tasks, Italian BERT and UmBERTo are always the best models, with up to 0.86 macro F1 for Specifications with BIO tags (Table 5 of the paper). Translating the Italian test set and reusing English models reaches the top informativeness score of 0.80, while a multilingual model trained in English drops to 0.65.

Training data for the English test set	Model	CAT	Informativeness	Type	Kind
Original English corpus	LEGAL-BERT	0.86	0.84	0.80	0.46
Italian corpus, translated to English	LEGAL-BERT	0.86	0.80	0.78	0.36
Italian corpus, test set translated to Italian	Italian BERT	0.85	0.84	0.77	0.35

Macro F1 on the English test set. Source: Table 4 of the paper.

In the other direction, models trained on the Italian corpus match the English-trained model in three of the four classification tasks, and fall 10 points behind only on Kind classification.

Takeaways

Fine-tuned BERT models beat LLMs on vagueness.Italian BERT reaches 0.79 macro F1 on informativeness, against 0.72 for the best few-shot LLM. LLMs lead only on Kind classification, which has 30 classes.
English resources can serve Italian.Translating the test set and reusing English models matches training on Italian data, with no new corpus and no new training. A multilingual model trained in English transfers poorly.
The Italian corpus generalizes to English.Trained on Italian data, models match the English-trained model in three of the four classification tasks. The corpus and code are public.

AMELIA - Argument Mining Evaluation on Legal documents in ItAlian: A CALAMITA Challenge

Sun, 01 Dec 2024 00:00:00 +0000

Research setting

Argument mining is the automatic analysis of arguments in text. An argument is a set of connected pieces of text, called argument components. A premise is a component that gives a reason for the argument, and a conclusion is the point that the premises argue for. CALAMITA is a collection of tests of the abilities of large language models (LLMs) in Italian. AMELIA is its challenge on legal arguments: a model classifies argument components from Italian tax decisions in three tasks.

The three AMELIA tasks on a real argument component of the dataset, shown in the English translation of Appendix A of the paper. Each task applies to the output of the previous one. Schema drawn from the task definitions in Section 2 of the paper.

The second task asks whether a premise is factual, legal, or both. A factual premise describes the events or the procedure of the case, and a legal premise states legal content such as rules, precedents, or principles. The third task labels each legal premise with one or more argument schemes, the patterns of legal reasoning it follows:

Scheme	The legal premise…
Rule	refers to codified law, such as an article or its text.
Precedent	refers to a previous decision of the Court of Cassation or the European Court of Justice.
Classification	defines a legal concept and qualifies a fact as having its properties.
Interpretative	states a new interpretation by the Court, which creates a new precedent.
Principle	refers explicitly to a principle of law, such as the principle of proportionality.

The five argument schemes for tax law. Source: Section 2 of the paper.

Motivation

Recognizing arguments is a first step toward reasoning, so even basic argument mining tasks tell us how well LLMs understand the logical relations expressed in text. Legal texts are a hard test, because legal professionals build and take apart arguments in formal documents with technical language and complex syntax. Many argument mining datasets exist in English, but resources for Italian are scarce. Legal language processing in Italian has focused on other tasks, and the authors know of no earlier argument mining challenge on Italian legal documents.

Resource	Italian	Legal documents	Argument mining
English argument mining datasets (Habernal et al., 2024; Niculae et al., 2017; Poudyal et al., 2020; Mayer et al., 2021; Accuosto and Saggion, 2020)	no	partially	yes
Demosthenes, decisions of the Court of Justice of the European Union (Grundler et al., 2022)	no	yes	yes
Italian argument mining on news comments and tweets (Basile et al., 2016; Lai et al., 2018)	yes	no	yes
Italian legal NLP, such as article retrieval, outcome prediction, contracts, and summarization	yes	yes	no
This paper (AMELIA) This work	yes	yes	yes

How AMELIA relates to the work the paper discusses (Section 1 of the paper).

Objective. Measure how well LLMs recognize and classify legal arguments in Italian, with three classification tasks on real tax decisions annotated by experts.

Approach

The data come from real decisions of Italian Regional Tax Commissions on Value Added Tax (VAT), from 2010 to 2022. The annotation guidelines adapt those of Demosthenes, an English corpus of the same authors, and keep its classes. One change was needed: in the Italian decisions a component may span several sentences, and one sentence may hold several components. The tasks assume that the argument components are already found, so a model only has to classify them.

Collect 225 Italian VAT decisions, from 2010 to 2022.
Extract Argument text from the reasoning part of each decision.
Mask Names, addresses, and places hidden; dates keep only the year.
Annotate Four tax law experts add the labels of the three tasks.
Split Three splits by decision; zero-shot and few-shot prompts.

How the AMELIA dataset was built and how models are tested (Section 3 of the paper).

The split follows the decisions, so all components of a decision fall in the same split, at a ratio of about 60:20:20. For each task the paper gives two prompts in Italian. The zero-shot prompt holds only the class definitions, and the few-shot prompt adds labeled examples from the training split. The dataset is public on Hugging Face under the CC BY 4.0 license.

Results

The paper presents the challenge and its data and reports no model results. Its results are the composition of the dataset (Table 1) and the evaluation protocol. The classes are highly unbalanced, so models are scored with the macro F1 score. It averages the F1 score over the classes, so rare classes count as much as frequent ones.

225Italian decisions on VAT
2910premises in task 1
401conclusions in task 1

Split	Rule	Precedent	Interpretative	Principle	Classification
Train	350	264	224	92	51
Validation	107	82	83	21	22
Test	118	73	67	31	27
Total	575	419	374	144	100

Legal premises labeled with each argument scheme. A premise may have several schemes. Source: Table 1 of the paper.

Split	Decisions	Premises	Conclusions	Factual	Legal
Train	135	1866	242	1254	812
Validation	44	528	81	315	266
Test	46	516	78	323	260
Total	225	2910	401	1892	1338

Composition of the dataset. Source: Table 1 of the paper.

Premises far outnumber conclusions, and the five schemes range from 575 legal premises for Rule to 100 for Classification. This is why the paper uses macro F1 and also reports the F1 score of each class. As a reference for difficulty only, the best macro F1 scores on Demosthenes were 0.88 for component classification, 0.85 for premise type, and 0.75 for scheme classification. The paper stresses that these scores are not directly comparable with AMELIA.

Takeaways

A first argument mining challenge on Italian legal text.AMELIA tests whether LLMs can classify the components, premise types, and reasoning schemes of real Italian tax decisions.
Expert labels on real decisions.Four tax law experts annotated 225 anonymized VAT decisions, and the data are public under CC BY 4.0.
Rare classes decide the score.The classes are highly unbalanced, so macro F1 is the metric. The paper also notes the small size, the 2010 to 2022 time span, and the details lost to anonymization as limits.

Language is Scary when Over-Analyzed: Unpacking Implied Misogynistic Reasoning with Argumentation Theory-Driven Prompts

Fri, 01 Nov 2024 00:00:00 +0000

Research setting

Misogyny detection is a task of hate speech detection: a model reads a social media message and decides whether it is misogynous. Many misogynous messages are implicit. They contain no slur and no insult, and their hate lies in an unstated assumption about women, called an implied assumption. Classifiers trained mostly on explicit hate tend to miss these messages, as the example below shows.

An existing hate speech classifier (bert-hateXplain) flags the explicit message but misses the implicit one (Figure 1 of the paper). Its implied assumptions are that women are not as capable as men and should be told what to do.

The paper reads such messages as arguments, following the argument model of Toulmin. In this model, the claim is what the message states, and the warrant is the unstated link that makes the claim acceptable. A large language model (LLM) is a general text model that follows written instructions, called prompts. In the zero-shot setting the prompt gives only the instructions, and in the few-shot setting it also gives a few solved examples.

Motivation

Most work on hate speech has focused on explicit messages. Some English datasets pair implicit hateful messages with human-written implied statements, but few efforts generate these statements automatically. No study had evaluated whether widely used LLMs can reason their way to good implied assumptions, and Italian had no dataset for implicit misogyny.

Line of work	Targets implicit hate	Generates implied assumptions automatically	Argumentative reasoning framing
Explicit hate speech datasets and systems	no	no	no
Implicit hate datasets with implied statements (Sap et al., 2020; ElSherief et al., 2021)	yes	Human-written only	no
Concept-based explanation generation (Ji et al., 2020; Yang et al., 2023)	partially	yes	no
LLMs on argumentative tasks (Wachsmuth et al., 2024; Chen et al., 2023; Gorur et al., 2024)	no	no	yes
This paper This work	yes	yes	yes

How the paper relates to the prior work it discusses (Sections 1 and 2 of the paper).

Objective. Frame implicit misogyny detection as an argumentative reasoning task, and test whether LLMs can generate the missing warrant of a message and use it to classify the message, in English and Italian.

Approach

The paper prompts the LLM to write out the hidden reasoning before it gives its label. In the Toulmin setting, the LLM states the claim and the warrant, and then decides whether the message is misogynous based on the warrant. In the Assumption setting, the LLM writes the implied assumptions directly, without a claim, and then classifies. The Toulmin prompts ask for the reasoning step by step, a technique called chain-of-thought prompting. In both settings the generated text then informs the decision, a technique called knowledge-augmented prompting.

Message Women football commentators annoy me so much.
Claim Women football commentators are annoying.
Warrant The LLM writes the hidden link. Women do not understand sport.
Label Misogynous, decided from the warrant.

The Toulmin setting on the example of Figure 2 of the paper. The Assumption setting skips the claim and asks for the implied assumptions directly.

Each setting runs zero-shot and few-shot, with prompts in English and in Italian. Two instruction-tuned LLMs are tested: Llama3-8B and Mistral-7B-v02. The baselines are fine-tuned classifiers (bert-hateXplain for English, ALBERTo for Italian) and the same LLMs asked for a label with no reasoning.

Results

The experiments use two datasets of implicit misogyny. ImplicIT-Mis is a new Italian dataset of 1,120 Facebook comments, all misogynous, where 150 comments carry 225 human-written implied assumptions. SBIC+ has 2,409 English messages from two existing corpora, each with human-written warrants. Because every message is misogynous, classification is measured by recall, the share of messages labeled misogynous, from 0 to 1. The generated warrants were also checked by hand on 300 messages, 150 per language, for the best model in each language.

0.725recall of Llama3-8B few-shot with Toulmin warrants on ImplicIT-Mis, against 0.480 with implied assumptions
68% / 50%of generated implied assumptions are wrong, in Italian and in English
Allcorrect Italian labels in the manual check were reached for the wrong reasons

Recall on ImplicIT-Mis (higher is better)

ALBERTo (fine-tuned) 0.380
Llama3-8B, label only 0.738
Llama3-8B, Assumption This work 0.480
Llama3-8B, Toulmin This work 0.725
Mistral-7B-v02, label only 0.259
Mistral-7B-v02, Assumption This work 0.461
Mistral-7B-v02, Toulmin This work 0.556

Classification recall on the Italian ImplicIT-Mis dataset with few-shot prompts and the fine-tuned baseline. Source: Table 1 of the paper.

Show as table

System	Recall on ImplicIT-Mis
ALBERTo (fine-tuned)	0.380
Llama3-8B, label only	0.738
Llama3-8B, Assumption (this work)	0.480
Llama3-8B, Toulmin (this work)	0.725
Mistral-7B-v02, label only	0.259
Mistral-7B-v02, Assumption (this work)	0.461
Mistral-7B-v02, Toulmin (this work)	0.556

Setting	Model	ImplicIT-Mis (IT)	SBIC+ (EN)
Fine-tuned	bert-hateXplain / ALBERTo	0.380	0.342
Zero-shot, label only	Llama3-8B	0.588	0.609
Few-shot, label only	Llama3-8B	0.738	0.719
Few-shot, Assumption	Llama3-8B	0.480	0.616
Few-shot, Assumption	Mistral-7B-v02	0.461	0.685
Zero-shot, Toulmin	Llama3-8B	0.557	0.452
Few-shot, Toulmin	Llama3-8B	0.725	0.594
Few-shot, Toulmin	Mistral-7B-v02	0.556	0.604

Classification recall, selected rows. ALBERTo is the Italian baseline and bert-hateXplain the English one. Source: Table 1 of the paper.

LLMs generally beat the fine-tuned classifiers, and Llama3-8B with a plain few-shot prompt gives the best recall in both languages. In Italian, the Toulmin warrant comes close to that best score and gains 24 points over implied assumptions in the few-shot setting, while in English the warrant-based prompt falls behind. The manual check shows that a correct label does not mean a correct explanation. Only 35% of the Italian warrants were correct, and about half of the English implied assumptions were wrong. The most frequent errors were wrong inferences, missing knowledge of the people a message refers to, figurative language, and sarcasm.

Error in the generated warrant	Italian	English
Wrong inference	42	34
Lack of reference	17	0
Metaphorical and figurative language	14	1
Opposite intention	12	9
Wrong translation	10	N/A
Denial of misogyny	3	4
Sarcasm or irony	2	26

Error categories in the manually checked samples. Source: Table 3 of the paper.

Takeaways

A right label is not right reasoning.In the Italian manual check, every correctly classified message came with a wrong warrant, so classification scores cannot certify the explanation.
LLMs lean on stereotypes, not inference.The models fill the gap with common stereotypes about women and with spurious correlations. They fail on metaphors, sarcasm, and references to people they do not know.
Warrants help in Italian, not in English.Toulmin prompts raised Italian few-shot recall by 24 points over implied assumptions. The paper suggests checking the generated warrants with humans in the loop before using them.

SubjectivITA: An Italian Corpus for Subjectivity Detection in Newspapers

Fri, 01 Jan 2021 00:00:00 +0000