Sustainable Italian LLM Evaluation: Community Perspectives and Methodological Guidelines

Mon, 01 Sep 2025 00:00:00 +0000

Research setting

A large language model (LLM) is a neural network trained on large amounts of text that answers questions and follows instructions in natural language. Benchmarking an LLM means running it on tasks with known answers and scoring its outputs, so that models can be compared over time. Several LLMs trained for Italian have appeared recently. This position paper does not test a new model. It collects the experience of three Italian research groups and proposes how the community should evaluate these models.

Task item Marco Rossi è nato a Milano nel 1985. Which Milano?
Prompt The item with four options, from Milano, Texas to Milano, Italy.
LLM The model under evaluation.
Answer The model outputs one option letter, here B.
Score The answer is compared with the correct option.

One benchmark item, built from the multiple-choice example in Section 3.1 of the paper: the model must tell which Milano a sentence refers to.

Terms used on this page:

Native benchmark: a benchmark written directly in Italian, as opposed to one translated from English.
Probability-based evaluation reads the probability that the model gives to each predefined answer option and takes the most likely one as its choice.
Generative evaluation lets the model write a free-form answer and then checks that answer against criteria or a reference.
LLM-as-a-Judge is a generative evaluation in which another LLM grades the free-form answers.
Zero-shot and few-shot prompting give the model no solved examples or a few solved examples before the question.
A benchmark is saturated when the best models solve it almost fully, so it no longer separates them.

Motivation

Italian has rich inflection, a flexible word order, clitic pronouns, dropped subjects, and many regional varieties. A direct translation of English benchmarks cannot capture these features, nor Italian culture and institutions. Recent Italian LLM benchmarks are valuable. They are also isolated efforts, with limits in methodology, scope, sustainability, and coordination. The paper maps them onto its own evaluation framework and finds whole skill areas that no one tests.

Framework	Morphology	Pragmatics	Register	Ling. instructions	Task gen.	Cross-ling.
ITA-Bench	no	yes	no	no	no	yes
Evalita-LLM	no	yes	no	yes	no	no
ITALIC	yes	no	no	no	no	no
This paper This work	yes	yes	yes	yes	yes	yes

Coverage of six evaluation dimensions by three Italian LLM evaluation frameworks, from Table 1 of the paper. Ling. instructions is linguistic instruction following, Task gen. is task generalization, and Cross-ling. is cross-linguistic transfer. For this paper, a check means that its framework defines the dimension: the paper itself releases no dataset.

Objective. Give the Italian NLP community a shared framework and practical guidelines for evaluating Italian LLMs, and a plan to keep that evaluation running over time.

Approach

The paper organizes its guidelines around four questions. The first question sets the skills to test, through the three dimensions shown in the figure at the top of this page. The other three questions cover how to run a test, where to get the data, and how to keep the whole effort alive.

What to benchmark Linguistic competence, domain and knowledge, task generalization.
How to benchmark Task format, scoring strategy, and prompt variations.
Where to benchmark Translated, adapted, or native Italian data.
Sustainable benchmarking A living benchmark run by the community.

The four questions of the paper (Sections 2 to 5). The last one is the community plan that gives the paper its title.

Question	Main guideline of the paper
What	Task designers state which dimensions their task covers, so effort goes to the dimensions that few tasks test.
How: format	Multiple-choice questions are cheap and simple to score but far from real use. Open-ended generation is closer to real use but harder and costlier to score.
How: prompting	Zero-shot evaluation is the main measure. Few-shot prompting serves as a supplementary variation and a strong baseline.
How: variation	Prompts vary in register, explicitness, cultural framing, and random changes such as the order of options, to test robustness.
Where	Native development comes first for tasks that are critical for Italian. Translation and adaptation fill the remaining gaps.
Sustainable	Saturated or stale tasks leave the benchmark, and open models come first. Each model is reported with its parameters and training tokens. A rotating steering committee governs the effort, which runs on national compute such as CINECA’s Leonardo supercomputer.

Guidelines from Sections 2 to 5 of the paper.

Results

The paper reports no new model experiments. Its findings come from three sources: a coverage analysis of public datasets against the framework (Table 1), the authors’ experience with Italian benchmarks, and a preliminary cost analysis.

25 + 3public datasets and evaluation frameworks mapped onto the framework
3-5xmore compute for generative evaluation than for probability-based evaluation
500-750GPU hours per quarter to evaluate 10 models on 50 tasks

Coverage of 25 public datasets and 3 frameworks (ITA-Bench, Evalita-LLM, and ITALIC) by the dimensions of the framework. A check means covered, a cross means not covered (Table 1 of the paper).

Almost every dataset tests semantics, and many test lexical knowledge or a knowledge domain. No dataset or framework in the table tests register adaptation or task generalization, and only ITA-Bench tests cross-linguistic transfer. The paper calls for more tasks on pragmatics, register, and cross-linguistic transfer.

Translation can also change what a task measures. In the Italian version of a WinoGrande item, grammar alone gives the answer away, because GPS is masculine and mappa (map) is feminine.

Il GPSCandidate e la mappaCandidate mi hanno aiutato a tornare a casa. Mi sono perso quando laFeminine ___ è stata capovolta.Feminine

A WinoGrande item translated into Italian, from Section 4 of the paper. The correct answer is mappa. The highlighted feminine forms rule out GPS without any common sense.

Two more observations come from the authors’ experience. Italian models often perform better when prompted in English with the instruction to answer in Italian. This suggests that they benefit from higher-quality English training data. For scoring free-form answers, no open-weight LLM-as-a-Judge model is trained explicitly for Italian yet.

Takeaways