Overview of the CLEF-2025 CheckThat! Lab: Subjectivity, fact-checking, claim normalization, and retrieval

Wed, 01 Jan 2025 00:00:00 +0000

Research setting

Fact-checkers decide whether a claim is true, and much of their work can be supported by language technology. CheckThat! is an evaluation lab at CLEF, the Conference and Labs of the Evaluation Forum. Each year it runs shared tasks: the organizers release a dataset, teams build systems for it, and every system is scored on the same test data. The tasks follow the verification pipeline below, plus auxiliary tasks that support its stages.

Check-worthiness estimation Decide whether a statement deserves to be checked.
Verified claim retrieval Find claims that fact-checkers have already verified.
Supporting evidence retrieval Collect evidence about the claim from the Web.
Claim verification Decide whether the claim is true or false.

The four core stages of the verification pipeline, as described in Section 1 and Figure 1 of the paper.

Motivation

Claims rarely arrive as clean, neutral sentences. They hide in subjective text and in noisy social media posts, they rest on numbers, or they point to scientific studies without citing them. Up to 2022 the lab mainly focused on the core stages; the 2023 and 2024 editions added auxiliary tasks such as subjectivity. The 2025 edition adds three new tasks for these harder inputs and covers more languages than any earlier edition.

Edition	Subjectivity	Claim normalization	Numerical claims	Scientific web discourse	20 languages or more
2018 to 2020 (core verification pipeline)	no	no	no	no	no
2021 and 2022 (multilinguality, fake news detection)	no	no	no	no	no
2023 and 2024 (subjectivity, bias, authority finding, persuasion)	yes	no	no	no	no
This paper (2025 edition) This work	yes	yes	yes	yes	yes

How the 2025 edition relates to the earlier editions of the lab (Section 2, Table 1, and Section 6 of the paper).

Objective. Offer four tasks that support different stages of the verification pipeline, with datasets in mono-, multi-, and cross-lingual settings. Then compare the systems of the participating teams.

Approach

Each task has its own dataset with training, development, and test splits. A zero-shot setting means that a test language has no training data, so systems must transfer what they learned from other languages. Teams submit runs, which are sets of predictions on the test data. Each task scores the runs with its own metric.

Four tasks Each task supports one stage of the pipeline.
Datasets Training, development, and test splits.
Runs Teams submit predictions on the test data.
Scoring Each task scores the runs with its own metric.

How the 2025 lab ran (Sections 3 to 5 of the paper).

Task	Input	Output	Languages	Metric
T1 Subjectivity	A sentence from a news article	Subjective or objective	Arabic, Bulgarian, English, German, Italian, multilingual; zero-shot: Greek, Polish, Romanian, Ukrainian	Macro F1
T2 Claim normalization	A social media post	A clear, verifiable statement, the normalized claim	13 with training data, 7 zero-shot	METEOR
T3 Numerical claims	A claim with numbers or dates, plus the top 100 pieces of evidence	True, False, or Conflicting	English, Spanish, Arabic	Macro F1
T4a Scientific web discourse detection	A post from X	Any of: a scientific claim, a reference to a study, a mention of a scientific entity	Not stated	Macro F1
T4b Claim-source retrieval	A post that mentions a paper without a link	The mentioned paper among 7,718 CORD-19 papers	Not stated	MRR@5

The 2025 tasks. Source: Sections 3 and 4 of the paper.

Results

Each task was scored on its own test data. Macro F1 averages the F1 score over the classes, from 0 to 1. METEOR, from 0 to 1, measures how close a generated claim is to the reference claim. MRR@5 rewards a system that ranks the right paper high among its top five answers, from 0 to 1. The baselines include BM25, a classic keyword-based ranking method.

83teams took part, out of 177 registered, and 54 wrote system papers
20+languages across the four tasks
0.68best MRR@5 in claim-source retrieval, against 0.43 for the BM25 baseline

Best macro F1 in Task 1 (subjectivity) (higher is better)

Arabic 0.6884
English 0.8052
German 0.8520
Italian 0.8104
Multilingual 0.7550
Greek (zero-shot) 0.5067
Polish (zero-shot) 0.6922
Romanian (zero-shot) 0.8126
Ukrainian (zero-shot) 0.6424

Best macro F1 per language in Task 1. Greek, Polish, Romanian, and Ukrainian are zero-shot languages. Source: Table 6 of the paper.

Show as table

System	Best macro F1 in Task 1 (subjectivity)
Arabic	0.6884
English	0.8052
German	0.8520
Italian	0.8104
Multilingual	0.7550
Greek (zero-shot)	0.5067
Polish (zero-shot)	0.6922
Romanian (zero-shot)	0.8126
Ukrainian (zero-shot)	0.6424

Task	Teams	Metric	Best score	Baseline
T1 Subjectivity, English	21	Macro F1	0.8052	Not reported
T2 Claim normalization, English	18	METEOR	0.4569	Not reported
T3 Numerical claims, English	13	Macro F1 (0 to 100)	59.54	Not reported
T4a Scientific web discourse detection	10	Macro F1	0.7998	0.7668 (DeBERTa-v3)
T4b Claim-source retrieval	30	MRR@5	0.68	0.43 (BM25)

Participating teams and best score per task. Teams count every language of a task. Source: Sections 5.1 to 5.4 and Tables 6, 9, 11, and 12 of the paper.

Most teams fine-tuned pretrained transformer language models, and some also prompted or fine-tuned large language models (LLMs). For subjectivity, the best scores in the zero-shot languages vary widely, from 0.5067 in Greek to 0.8126 in Romanian. For numerical claims, most teams used LLMs to decompose each claim into simpler parts, retrieved evidence with BM25, and re-ranked it. For claim-source retrieval, most teams combined keyword-based (sparse) or embedding-based (dense) retrieval with re-ranking, and LLM re-rankers did not always beat fine-tuned transformers.

Takeaways