Overview of the CLEF-2024 CheckThat! lab: check-worthiness, subjectivity, persuasion, roles, authorities, and adversarial robustness

Mon, 01 Jan 2024 00:00:00 +0000

Research setting

Fact-checkers decide whether a claim is true, and language technology can support many steps of their work. CheckThat! is an evaluation lab at CLEF, the Conference and Labs of the Evaluation Forum. Each year it runs shared tasks: the organizers release a dataset, teams build systems for it, and every system is scored on the same test data. The tasks follow the verification pipeline below, plus auxiliary tasks that support its stages.

Check-worthiness estimation Decide whether a statement deserves to be checked.
Verified claim retrieval Find claims that fact-checkers have already verified.
Supporting evidence retrieval Collect evidence about the claim from the Web.
Claim verification Decide whether the claim is true or false.

The four core stages of the verification pipeline, as described in Section 1 and Figure 1 of the paper.

Motivation

The first five editions of the lab focused on the core stages of the pipeline. From 2023, the lab also opened to auxiliary tasks that help each stage. Claims also hide in subjective sentences, in persuasive news, and in memes. A classifier that flags misinformation can itself be fooled. The 2024 edition adds tasks for these cases, and three of its six tasks ran for the first time.

Edition	Check-worthiness	Subjectivity	Persuasion	Meme roles	Authority evidence	Adversarial attacks
2018 to 2020	yes	no	no	no	no	no
2021 and 2022	yes	no	no	no	no	no
2023	yes	yes	no	no	partially	no
This paper (2024) This work	yes	yes	yes	yes	yes	yes

How the 2024 edition relates to the earlier editions of the lab (Sections 1 to 3 and Table 1 of the paper). Partial: the 2023 edition asked to find authorities. It did not ask to retrieve evidence from them.

Objective. Offer six tasks that support different steps of the verification pipeline, in mono-, multi-, and cross-lingual settings over fifteen languages. Then compare the systems of the participating teams.

Approach

Each task has its own dataset, mostly with training, development, and test splits. Teams submit runs, which are sets of predictions on the test data. Each task scores the runs with its own metric and compares them with a baseline, a simple reference system built by the organizers. In Task 6 the roles change: teams attack victim models, which are classifiers that detect misinformation. An adversarial example is a text changed slightly so that the victim model changes its decision while the meaning stays the same.

Six tasks Each task supports one step of the pipeline.
Datasets Training, development, and test data per language.
Runs Teams submit predictions on the test data.
Scoring Each task scores the runs against a baseline.

How the 2024 lab ran (Sections 3 to 5 of the paper).

Task	Input	Output	Test languages	Metric
T1 Check-worthiness	A statement from a tweet, a transcript, or a political debate	Whether it needs fact-checking	Arabic, Dutch, English	F1 of the check-worthy class
T2 Subjectivity	A sentence from a news article	Subjective or objective	Arabic, Bulgarian, English, German, Italian, multilingual	Macro F1
T3 Persuasion techniques	A news article	Text spans that use one of 23 persuasion techniques	Arabic, Bulgarian, English, Portuguese, Slovene	Micro F1 with partial span matching
T4 Roles in memes	A meme and an entity in it	Hero, villain, victim, or other	Bulgarian, English, code-mixed Hindi-English	F1
T5 Evidence from authorities	A rumor in a tweet and the timelines of authorities, the sources that know the matter	Evidence tweets, then supported, refuted, or unverifiable	Arabic, English	MAP, macro F1
T6 Adversarial robustness	A text and a victim model	An adversarial example	Not stated	BODEGA score

The 2024 tasks. Source: Sections 3, 4, and 5 of the paper.

Results

Each task was scored on its own test data, against its own baseline. F1 combines precision and recall, from 0 to 1, and macro F1 averages it over the classes. MAP (mean average precision), from 0 to 1, rewards a system that ranks the evidence tweets high in its list. The BODEGA score, from the framework of the same name, rates adversarial examples automatically, and Task 6 also checked a sample by hand.

46teams took part, out of 130 registered
15languages across the six tasks
0.879best macro F1 in English rumor verification, against 0.495 for the baseline

Task	Teams	Metric	Best score	Baseline
T1 Check-worthiness, English	26	F1 of the check-worthy class	0.802	Not reported
T2 Subjectivity, English	15 (all languages)	Macro F1	0.744	Not reported
T3 Persuasion techniques, English	2 (all languages)	Micro F1	0.092	0.009
T4 Roles in memes	0	F1	No runs	0.58
T5 Evidence retrieval, English	5	MAP	0.604	0.335
T5 Rumor verification, English	5	Macro F1	0.879	0.495
T6 Adversarial robustness	6	BODEGA score	0.7458	0.4261 (BERT-ATTACK, an earlier attack method)

Participating teams and best official score per task. Source: Sections 5.1 to 5.6 and Tables 5, 7, 10, 12, 14, and 15 of the paper. The best Task 5 retrieval run came from a team of the task organizers.

Most teams fine-tuned pretrained transformer models, and many also used large language models (LLMs). In Task 5, systems fine-tuned on the task data retrieved evidence best, but only the two LLM-based systems beat the baseline in English rumor verification. Task 4 attracted no participants, so only its baseline is available. Task 6 shows a gap between automatic and manual evaluation, as the chart below shows.

Adversarial examples with preserved meaning (manual check) (higher is better)

SINAI 99%
MMU_NLP 96%
TurQUaz 62%
Palöri 14%
OpenFact 11%
TextTrojaners 7%

Share of adversarial examples whose meaning annotators judged preserved. Each team had 100 successful examples from the fact-checking domain of Task 6. The two top teams of the automatic BODEGA ranking (OpenFact, TextTrojaners) score lowest here. Source: Section 5.6 of the paper.

Show as table

System	Adversarial examples with preserved meaning (manual check)
SINAI	99%
MMU_NLP	96%
TurQUaz	62%
Palöri	14%
OpenFact	11%
TextTrojaners	7%

Takeaways