Multimodal Argument Mining | Language Technologies Lab

Overview of MM-ArgFallacy2025 on Multimodal Argumentative Fallacy Detection and Classification in Political Debates

Tue, 01 Jul 2025 00:00:00 +0000

Research setting

Argument mining is the automatic extraction of arguments from discourse. Multimodal argument mining (MAM) adds the audio of the speech to the text, because paralinguistic features, such as how a sentence is spoken, carry cues that text alone misses. A fallacy is an argument with a flaw in its reasoning, and political debates contain many of them. MM-ArgFallacy2025 is a shared task, a public challenge where teams build systems for the same task and are ranked on the same hidden data. It has two subtasks on sentences from U.S. presidential debates: argumentative fallacy detection (AFD) and argumentative fallacy classification (AFC).

The two subtasks on a real annotated example labeled Ad Hominem (Table 1 of the paper). Each sentence comes as text only, audio only, or text with its aligned audio. Systems may also use the previous sentences of the debate as context. Schema drawn from the paper’s task definition (Section 3).

Motivation

Detecting and classifying fallacies can help to analyse behaviour in dialogue, to stop misinformation in fact-checking systems, and to evaluate the reasoning of generative models. The voice may help, since paralinguistic features can be associated with specific fallacy types. MAM has mainly addressed finding arguments, labeling their parts, and linking them, where audio has proved effective. Fallacy detection and classification with audio are still underexplored.

Resource	Audio aligned with text	Fallacy labels	Secret test set from new debates
UKDebates (Lippi and Torroni, 2016)	yes	no	no
M-Arg (Mestre et al., 2021)	yes	no	no
MM-USED (Mancini et al., 2022)	yes	no	no
Fallacies in debates (Goffredo et al., 2022)	no	yes	no
MM-USED-fallacy (Mancini et al., 2024)	yes	yes	no
This paper (MM-ArgFallacy2025) This work	yes	yes	yes

How the shared task relates to the debate datasets the paper discusses (Sections 2 and 4 and Table 2 of the paper).

Objective. Provide a common platform to develop and evaluate systems that detect and classify fallacies in political debates from text, audio, or both, and report what the participating systems reveal.

Approach

The organizers released MM-USED-fallacy for training, with sentences and aligned audio from past U.S. presidential debates. It holds 18,925 instances for detection and 3,388 for classification. Three supplementary debate datasets, labeled for other argument mining tasks, were also offered for training on several related tasks at once.

For the secret test set, the organizers took two 2024 presidential debates: Trump against Biden and Trump against Harris. They aligned the transcripts to the audio with WhisperX, a speech recognition model. Two annotators with argument mining expertise labeled the fallacies with the guidelines of the training data. Their agreement, measured as Cohen’s kappa, is 0.4787 for detection and 0.4954 for classification, both moderate.

Training data MM-USED-fallacy, past presidential debates with aligned audio.
New debates Two 2024 debates, transcripts aligned to audio.
Annotation Two annotators label fallacies and their types.
Evaluation Team runs scored on the secret test set.

How the shared task was built and scored (Sections 4 and 5 of the paper).

Fallacy type	Test instances
Appeal to Emotion	142
Ad Hominem	46
Appeal to Authority	16
False Cause	16
Slogan	9
Slippery Slope	0
Total fallacious	229

The secret test set for classification (Table 3 of the paper). For detection, the test set holds 229 fallacies and 1,946 non-fallacious sentences.

The organizers ran two baselines in every input mode. The first is a BiLSTM, a recurrent network, that reads GloVe word embeddings and MFCCs, standard acoustic features. The second is a transformer model that uses RoBERTa for text and WavLM for audio.

Results

Five teams took part, three in classification and two in detection, with 25 valid runs in total. Every system is scored with F1, which combines precision and recall into a value from 0 to 1. Detection uses binary F1. Classification uses macro F1, the average F1 over fallacy types, so rare types weigh as much as frequent ones.

0.4856best classification macro F1, text only (Team NUST)
0.3559best classification macro F1, audio only (Team AlessioPittiglio)
0.4611best classification macro F1, text and audio (Team NUST)

Macro F1 on the secret test set, text only (higher is better)

Team NUST 0.4856
Baseline BiLSTM 0.4721
Team AlessioPittiglio 0.4444
Baseline Transformer 0.3925
Team CASS 0.1432

Fallacy classification with text only. Source: Table 4 of the paper.

Show as table

System	Macro F1 on the secret test set, text only
Team NUST	0.4856
Baseline BiLSTM	0.4721
Team AlessioPittiglio	0.4444
Baseline Transformer	0.3925
Team CASS	0.1432

System	Text	Audio	Both
Classification (macro F1)
Team NUST	0.4856	0.1588	0.4611
Team AlessioPittiglio	0.4444	0.3559	0.4403
Team CASS	0.1432	0.0864	0.1432
Baseline BiLSTM	0.4721	0.1582	0.2191
Baseline Transformer	0.3925	0.0643	0.3816
Detection (binary F1)
Team Ambali_Yashovardhan	0.2534	0.2095	0.2244
Team EvaAdriana	0.2195	0.1690	0.1931
Baseline BiLSTM	0.2462	0.0000	0.2337
Baseline Transformer	0.2770	0.0000	0.2848

All results by input mode, where Both means text with its aligned audio. Source: Table 4 of the paper.

Text alone gives the best classification scores. Only Team NUST beats both baselines with text, which shows that even a simple BiLSTM is a strong competitor. Audio alone stays well below text, although the best audio-only run improves over earlier work. Adding audio to text does not beat the best text-only score in classification. In detection, the transformer baseline ranks first with text and with text and audio. Most teams used transformer models, mainly RoBERTa variants. Team NUST countered the rarity of some fallacy types by generating extra training examples with GPT-4.

Takeaways

Text carries most of the signal.Text-only systems gave the best classification scores, and adding audio brought at most small gains. Transformer models, particularly RoBERTa variants, proved the most effective.
Audio needs better fusion.Audio-only scores improved over earlier work but stay below text. The paper argues that simple fusion, which joins the text and audio outputs only at the end, misses how words and voice interact. It calls for models that learn both modalities jointly.
Data scale and imbalance limit the task.Only 9.2% of detection sentences are fallacious, Appeal to Emotion makes up 59% of fallacies, and Slippery Slope is absent from the test set. Larger and more balanced data across more debates is the next step.

Multimodal Fallacy Classification in Political Debates

Fri, 01 Mar 2024 00:00:00 +0000

Research setting

Argument mining is the automatic analysis of arguments in text, such as finding claims and the reasons that support them. A fallacy is a deceptive, misleading, or invalid argument, and fallacy classification assigns each fallacious passage to a fallacy category. Most argument mining work reads only the words. Spoken debates also carry paralinguistic cues, such as accent, rhythm, and tone of voice. This paper treats fallacy classification in US presidential debates as a multimodal task, where each fallacy comes with its text and its audio clip.

Debate passage One passage, in two modalities.
- Text Transcript sentences
- Audio The matching recording
Classifier Combines what both modalities say.
Fallacy category One of six categories.

The task: a fallacious passage from a debate, as text and as audio, goes in, and one of six fallacy categories comes out.

Fallacy category	What it does	Example from the corpus
Appeal to Emotion	Uses loaded language.	“the same kind of woolly thinking”
Appeal to Authority	Uses an expert’s opinion as evidence.	“As George Will said the other day, …”
Ad Hominem	Attacks the arguer instead of the position.	“Governor Carter apparently doesn’t know the facts.”
False Cause	Takes correlation for causation.	“We won the Cold War because we invested and we went forward.”
Slippery Slope	Claims exaggerated outcomes for an action.	“And if we don’t act today, the problem will be valued in the trillions.”
Slogans	Brief, striking phrases that evoke excitement.	“We have to practice what we preach.”

The six fallacy categories (Section 3.1 and Table 1 of the paper).

Terms used on this page:

A snippet is the annotated fallacy within a debate passage.
A pretrained encoder is a large model trained beforehand on other data that turns text or audio into vectors. BERT, RoBERTa, and SBERT encode text; Wav2Vec and CLAP encode audio.
F1 balances precision and recall for one category. Macro-F1 averages F1 over the six categories, so rare categories count as much as frequent ones. It ranges from 0 to 1, and higher is better.

Motivation

Analyses of political speech link fallacies to the voice. Accent stereotypes go with Ad Hominem, a staccato rhythm with Appeal to Authority, and an angry tone with Appeal to Emotion. Multimodal argument mining already uses debate audio for argument detection, component classification, and relation classification. Fallacy classification, however, had only been studied on text, and no corpus paired annotated fallacies with audio.

Study	Political debates	Uses audio	Fallacy classification
Claim detection from speech (Lippi and Torroni, 2016)	yes	yes	no
Multimodal extensions of USED (Mancini et al., 2022; Mestre et al., 2023)	yes	yes	no
Text-only fallacy classification, USED-fallacy (Goffredo et al., 2022)	yes	no	yes
This paper (MM-USED-fallacy) This work	yes	yes	yes

How the paper positions itself against the related work it discusses (Sections 1 and 2 of the paper).

Objective. Build the first corpus for multimodal fallacy classification, and test whether adding audio to text improves the classification of fallacies in political debates.

Approach

The authors start from USED-fallacy, a text corpus of US presidential debates from 1960 to 2016 with span-level fallacy annotations. They map each annotated span to the sentences it overlaps. Sentence-level timestamps from an earlier multimodal version of the same debates then cut the matching audio clips. Fuzzy string matching handles small transcript differences, and misaligned samples are removed. The result, MM-USED-fallacy, holds 1,891 text-audio pairs from 28 debates.

Spans Fallacy spans from USED-fallacy.
Sentences Label every sentence that overlaps a fallacy span.
Audio Timestamps cut the clips; misaligned samples are removed.
Corpus MM-USED-fallacy, 1,891 text-audio pairs from 28 debates.

How MM-USED-fallacy is built (Section 3.2 of the paper).

The classifier keeps the architecture shown at the top of this page. A text module and an audio module each encode the passage, their outputs are concatenated, and a stack of dense layers predicts the category. The study compares three text encoders and two audio encoders, alone and in every text-audio pair.

Encoders One module per modality.
- Text module BERT, RoBERTa, or SBERT
- Audio module Wav2Vec or CLAP, then a BiLSTM
Concatenation Joins the two encodings.
Classification module Dense layers predict the category.

The classifier and the encoders compared (Section 4 and Figure 1 of the paper).

Results

The authors compare text-only, audio-only, and text-audio models on 1,063 unique passage and snippet pairs, after removing duplicate snippets. They use leave-one-out cross-validation over debates: each of the 28 runs tests on one debate and trains on the others. Scores are macro-F1 averaged over the 28 runs, next to a majority-class baseline and a random baseline.

0.40best average macro-F1, text plus audio (BERT + Wav2Vec, RoBERTa + CLAP)
0.38best average macro-F1, text only (RoBERTa)
+8 pointsgain from adding Wav2Vec audio to BERT (0.32 to 0.40)

Average macro-F1 on MM-USED-fallacy (higher is better)

Random baseline 0.12 ± 0.05
Majority baseline 0.20 ± 0.17
Audio only, Wav2Vec 0.13 ± 0.07
Audio only, CLAP 0.12 ± 0.08
Text only, BERT 0.32 ± 0.13
Text only, RoBERTa 0.38 ± 0.18
Text only, SBERT 0.31 ± 0.18
BERT + Wav2Vec This work 0.40 ± 0.17
BERT + CLAP This work 0.36 ± 0.17
RoBERTa + Wav2Vec This work 0.39 ± 0.19
RoBERTa + CLAP This work 0.40 ± 0.19
SBERT + Wav2Vec This work 0.23 ± 0.11
SBERT + CLAP This work 0.24 ± 0.10

Average macro-F1 over the 28 runs; whiskers show the standard deviation reported in the paper. Source: Table 6 of the paper.

Show as table

System	Average macro-F1 on MM-USED-fallacy
Random baseline	0.12 ± 0.05
Majority baseline	0.20 ± 0.17
Audio only, Wav2Vec	0.13 ± 0.07
Audio only, CLAP	0.12 ± 0.08
Text only, BERT	0.32 ± 0.13
Text only, RoBERTa	0.38 ± 0.18
Text only, SBERT	0.31 ± 0.18
BERT + Wav2Vec (this work)	0.40 ± 0.17
BERT + CLAP (this work)	0.36 ± 0.17
RoBERTa + Wav2Vec (this work)	0.39 ± 0.19
RoBERTa + CLAP (this work)	0.40 ± 0.19
SBERT + Wav2Vec (this work)	0.23 ± 0.11
SBERT + CLAP (this work)	0.24 ± 0.10

Model	Appeal to Emotion	Appeal to Authority	Ad Hominem	False Cause	Slippery Slope	Slogans
BERT, text only	0.70	0.45	0.15	0.28	0.22	0.06
BERT + Wav2Vec	0.80	0.50	0.13	0.35	0.23	0.04
RoBERTa, text only	0.53	0.50	0.32	0.29	0.30	0.17
RoBERTa + CLAP	0.74	0.45	0.23	0.37	0.31	0.12

F1 per fallacy category, averaged over the 28 runs (Table 6 of the paper).

Audio alone is weak, close to the random baseline, but adding it to BERT or RoBERTa raises the average macro-F1. The gain is statistically significant for BERT (p-value below 0.05) but not for RoBERTa, and SBERT drops when audio is added. The gains are uneven across categories: Appeal to Emotion and False Cause improve the most, while text-only models are as good or better on Slogans, the rarest category. Results also vary across debates, and recent debates tend to benefit more from audio.

Takeaways

A first multimodal fallacy corpus.MM-USED-fallacy pairs 1,891 annotated fallacy passages from 28 US presidential debates with their audio. The corpus and code are public.
Audio helps text, not alone.Adding audio raises average macro-F1 from 0.32 to 0.40 for BERT, a statistically significant gain. For RoBERTa it rises from 0.38 to 0.40, a gain that is not significant, and audio-only models stay near the random baseline.
The benefit depends on the fallacy.Appeal to Emotion gains the most from audio, while Slogans do not. The annotations come from text only, so a new annotation that listens to the audio could reveal more.

Multimodal Argument Mining: A Case Study in Political Debates

Sat, 01 Jan 2022 00:00:00 +0000