Audio | Language Technologies Lab

MAMKit: A Comprehensive Multimodal Argument Mining Toolkit

Thu, 01 Aug 2024 00:00:00 +0000

Research setting

Argument mining is the automatic analysis of arguments in text, such as finding claims, the premises that support them, and the relations between them. Political debates are a natural source of arguments, and they are spoken: the voice of a speaker carries paralinguistic cues, such as tone and pauses, that a transcript loses. Multimodal argument mining (MAM) studies arguments from both the text and the audio of each sentence. The paper covers four MAM tasks:

Argumentative sentence detection (ASD) decides whether a sentence contains an argument or a component, such as a claim.
Argumentative component classification (ACC) labels an argumentative sentence as a claim or a premise.
Argumentative relation classification (ARC) labels a pair of sentences as support, attack, or no relation.
Argumentative fallacy classification (AFC) assigns a sentence already known to be a fallacy, a flawed argument, to one category of a fallacy taxonomy.

Debate Transcripts and audio recordings of political debates.
Aligned sentence Each sentence is paired with its audio snippet.
- Text Sentence transcript
- Audio Speech snippet
Multimodal model Encodes and fuses text and audio.
Task label One of the four tasks.
- ASD, ACC Sentence labels
- ARC Pair labels
- AFC Fallacy category

The MAM setting of the paper (Sections 3.3 to 3.5). Each debate sentence comes with its audio snippet. A model reads both and predicts the label of one task.

Motivation

Early MAM results suggest that the audio of a speaker helps argument analysis. However, MAM datasets and models are spread across different sites and repositories, each with its own way of loading and rebuilding them. This makes a fair comparison between models hard, and it slows the evaluation of new ones. Toolkits with a shared interface exist for other areas of AI, but none supports argument mining. Past MAM models also mostly merged unimodal models in a standard way and left recent audio encoders and fusion methods unexplored.

Resource	Built for argument mining	Shared interface for datasets and models	Recent audio encoders and fusion methods
Earlier MAM models and datasets, each in its own repository	yes	no	no
Language-vision libraries (LAVIS, MMF, X-modaler, UniLM)	no	yes	Other modalities
Multimodal and audio tools (TorchMultimodal, ViLMedic, pyannote, Muskits)	no	yes	Other tasks
Task-specific NLP libraries (LogiTorch, TextBox 2.0, DeepPavlov, and others)	no	yes	Text only
This paper (MAMKit) This work	yes	yes	yes

How MAMKit relates to the prior work discussed in the paper (Sections 1 and 2).

Objective. Give MAM researchers one open, standardized platform where datasets and models can be loaded, compared, and reproduced, and use it to test new audio encoders and fusion methods.

Approach

MAMKit is an open-source PyTorch toolkit organized in five packages, shown in the figure at the top of this page. The data package loads, splits, and preprocesses each dataset for text-only, audio-only, or text-audio input. The models and modules packages define the models and their shared layers. The configs package stores the exact settings of published experiments, so a user can rebuild a model and its data processing from a single configuration key. The utility package wraps every model for training and evaluation with PyTorch Lightning.

Every model in MAMKit follows the same structure: a text module, an audio module, a fusion layer that merges the two, and a classification head for the task. Early fusion merges the two modalities near the input, while late fusion merges the outputs of two separate unimodal branches.

Text module GloVe with a BiLSTM, BERT, or RoBERTa.
Audio module Speech features, then a BiLSTM or a transformer.
- Features Wav2Vec 2.0, HuBERT, WavLM, or MFCCs
Fusion layer Merges text and audio.
- Concat Concatenation, early or late
- Average Late
- Crossmodal attention One modality attends to the other
Classification head Predicts the task label.

The structure shared by all MAMKit models (Section 3.3 and Table 1 of the paper). Wav2Vec 2.0, HuBERT, and WavLM are pre-trained speech models. MFCCs are classic hand-crafted audio features.

MAMKit includes three models from earlier MAM work and three new ones.

Model	Origin	Text module	Audio module	Fusion
BiLSTM	Earlier work	GloVe + BiLSTM	Wav2Vec 2.0 or MFCCs + BiLSTM	Concatenation, late
MM-BERT	Earlier work	BERT	Wav2Vec 2.0, HuBERT, or WavLM + BiLSTM	Concatenation, late
MM-RoBERTa	Earlier work	RoBERTa	Wav2Vec 2.0, HuBERT, or WavLM + BiLSTM	Concatenation, late
CSA	This paper	BERT	Wav2Vec 2.0, HuBERT, or WavLM + transformer	Concatenation, early
Ensemble	This paper	BERT	Wav2Vec 2.0, HuBERT, or WavLM + transformer	Average, late
Mul-TA	This paper	BERT	Wav2Vec 2.0, HuBERT, or WavLM + transformer	Crossmodal attention

Models available in MAMKit. Source: Table 1 of the paper.

CSA joins the text and audio sequences into one sequence and applies self-attention. Audio sequences are often about ten times longer than text, so CSA rescales the attention scores to give text and audio the same total weight.
Ensemble trains a text-only and an audio-only model and averages their predicted probabilities with a learned weight. The weight is bounded between 0.3 and 0.7, so the ensemble cannot ignore either modality.
Mul-TA adapts a multimodal transformer to text and audio. In each crossmodal attention layer, one modality queries the other, so the two sequences need no alignment.

The CSA model joins text and audio embeddings into one sequence for a transformer with rescaled attention (Figure 2 of the paper).

Results

The paper evaluates every MAMKit model on every supported task and dataset:

UKDebates has 386 sentences from a 2015 UK Prime Ministerial election debate, labeled as containing a claim or not (ASD).
M-Argγ has 2,443 sentence pairs from the 2020 US presidential debates, kept for their high annotator agreement and labeled for relations (ARC).
MM-USED has 26,781 sentences from 39 US presidential debates, from 1960 to 2016, labeled for detection (ASD) and for claims and premises (ACC). It is the largest MAM resource to date.
MM-USED-fallacy has 1,891 sentences labeled with six fallacy categories (AFC).

The metric is macro F1, the F1 score averaged over classes, except on UKDebates, where it is the binary F1 of the claim class. Scores range from 0 to 1, and higher is better. Each score is the best configuration of a model, averaged over repeated runs.

.697macro F1 of Mul-TA on MM-USED (ACC), against .679 for its text-only module
.657macro F1 of an audio-only BiLSTM on MM-USED-fallacy (AFC), the best of all models
.692binary F1 of text-only RoBERTa on UKDebates (ASD), above every text-audio model

Macro F1 on MM-USED-fallacy (AFC) (higher is better)

BiLSTM, text only .525 ± .113
BERT, text only .594 ± .122
RoBERTa, text only .615 ± .097
BiLSTM, audio only .657 ± .000
Transformer, audio only .629 ± .162
BiLSTM, text and audio .572 ± .099
MM-BERT .599 ± .128
MM-RoBERTa .624 ± .074
CSA This work .582 ± .114
Ensemble This work .612 ± .134
Mul-TA This work .605 ± .110

Fallacy classification, the only task where audio-only models lead. Source: Table 2 of the paper (mean and standard deviation over runs).

Show as table

System	Macro F1 on MM-USED-fallacy (AFC)
BiLSTM, text only	.525 ± .113
BERT, text only	.594 ± .122
RoBERTa, text only	.615 ± .097
BiLSTM, audio only	.657 ± .000
Transformer, audio only	.629 ± .162
BiLSTM, text and audio	.572 ± .099
MM-BERT	.599 ± .128
MM-RoBERTa	.624 ± .074
CSA (this work)	.582 ± .114
Ensemble (this work)	.612 ± .134
Mul-TA (this work)	.605 ± .110

Model	Input	UKDebates (ASD)	M-Argγ (ARC)	MM-USED (ASD)	MM-USED (ACC)	MM-USED-fallacy (AFC)
BiLSTM	Text	.552	.120	.811	.663	.525
BERT	Text	.654	.132	.824	.679	.594
RoBERTa	Text	.692	.172	.839	.680	.615
BiLSTM	Audio	.393	.024	.774	.596	.657
Transformer	Audio	.455	.000	.771	.526	.629
BiLSTM	Both	.533	.084	.815	.667	.572
MM-BERT	Both	.662	.160	.841	.680	.599
MM-RoBERTa	Both	.687	.178	.837	.678	.624
CSA (this paper)	Both	.663	.160	.833	.693	.582
Ensemble (this paper)	Both	.586	.011	.826	.681	.612
Mul-TA (this paper)	Both	.616	.098	.837	.697	.605

Best score of each model on each task. Input is text, audio, or both. Source: Table 2 of the paper.

On UKDebates, adding audio brings no benefit: text-audio models score the same as or below their text-only modules. On M-Argγ, audio-only models fail to learn the task, and only MM-BERT and CSA score slightly above their text-only modules. On MM-USED, audio-only models come close to text-only ones, but text-audio models improve on their text modules by at most 1.8 points on a 0 to 100 scale. On MM-USED-fallacy, audio-only models lead, and text-audio models fall between text-only and audio-only ones. The paper suggests one cause: every dataset was first annotated on transcripts and only later aligned to audio, so the labels do not reflect acoustic cues.

Takeaways

MAM now has a shared platform.MAMKit offers 4 datasets and 6 models behind one interface, and it rebuilds published experiments from a configuration key.
New audio encoders and fusion methods gave small gains.The three new models did not bring the improvement the authors hoped for, and text-only models remain strong baselines.
Annotation should start from the audio.Labels made on transcripts miss auditory cues, so progress in MAM needs annotation processes that include them.

Multimodal Fallacy Classification in Political Debates

Fri, 01 Mar 2024 00:00:00 +0000

Research setting

Argument mining is the automatic analysis of arguments in text, such as finding claims and the reasons that support them. A fallacy is a deceptive, misleading, or invalid argument, and fallacy classification assigns each fallacious passage to a fallacy category. Most argument mining work reads only the words. Spoken debates also carry paralinguistic cues, such as accent, rhythm, and tone of voice. This paper treats fallacy classification in US presidential debates as a multimodal task, where each fallacy comes with its text and its audio clip.

Debate passage One passage, in two modalities.
- Text Transcript sentences
- Audio The matching recording
Classifier Combines what both modalities say.
Fallacy category One of six categories.

The task: a fallacious passage from a debate, as text and as audio, goes in, and one of six fallacy categories comes out.

Fallacy category	What it does	Example from the corpus
Appeal to Emotion	Uses loaded language.	“the same kind of woolly thinking”
Appeal to Authority	Uses an expert’s opinion as evidence.	“As George Will said the other day, …”
Ad Hominem	Attacks the arguer instead of the position.	“Governor Carter apparently doesn’t know the facts.”
False Cause	Takes correlation for causation.	“We won the Cold War because we invested and we went forward.”
Slippery Slope	Claims exaggerated outcomes for an action.	“And if we don’t act today, the problem will be valued in the trillions.”
Slogans	Brief, striking phrases that evoke excitement.	“We have to practice what we preach.”

The six fallacy categories (Section 3.1 and Table 1 of the paper).

Terms used on this page:

A snippet is the annotated fallacy within a debate passage.
A pretrained encoder is a large model trained beforehand on other data that turns text or audio into vectors. BERT, RoBERTa, and SBERT encode text; Wav2Vec and CLAP encode audio.
F1 balances precision and recall for one category. Macro-F1 averages F1 over the six categories, so rare categories count as much as frequent ones. It ranges from 0 to 1, and higher is better.

Motivation

Analyses of political speech link fallacies to the voice. Accent stereotypes go with Ad Hominem, a staccato rhythm with Appeal to Authority, and an angry tone with Appeal to Emotion. Multimodal argument mining already uses debate audio for argument detection, component classification, and relation classification. Fallacy classification, however, had only been studied on text, and no corpus paired annotated fallacies with audio.

Study	Political debates	Uses audio	Fallacy classification
Claim detection from speech (Lippi and Torroni, 2016)	yes	yes	no
Multimodal extensions of USED (Mancini et al., 2022; Mestre et al., 2023)	yes	yes	no
Text-only fallacy classification, USED-fallacy (Goffredo et al., 2022)	yes	no	yes
This paper (MM-USED-fallacy) This work	yes	yes	yes

How the paper positions itself against the related work it discusses (Sections 1 and 2 of the paper).

Objective. Build the first corpus for multimodal fallacy classification, and test whether adding audio to text improves the classification of fallacies in political debates.

Approach

The authors start from USED-fallacy, a text corpus of US presidential debates from 1960 to 2016 with span-level fallacy annotations. They map each annotated span to the sentences it overlaps. Sentence-level timestamps from an earlier multimodal version of the same debates then cut the matching audio clips. Fuzzy string matching handles small transcript differences, and misaligned samples are removed. The result, MM-USED-fallacy, holds 1,891 text-audio pairs from 28 debates.

Spans Fallacy spans from USED-fallacy.
Sentences Label every sentence that overlaps a fallacy span.
Audio Timestamps cut the clips; misaligned samples are removed.
Corpus MM-USED-fallacy, 1,891 text-audio pairs from 28 debates.

How MM-USED-fallacy is built (Section 3.2 of the paper).

The classifier keeps the architecture shown at the top of this page. A text module and an audio module each encode the passage, their outputs are concatenated, and a stack of dense layers predicts the category. The study compares three text encoders and two audio encoders, alone and in every text-audio pair.

Encoders One module per modality.
- Text module BERT, RoBERTa, or SBERT
- Audio module Wav2Vec or CLAP, then a BiLSTM
Concatenation Joins the two encodings.
Classification module Dense layers predict the category.

The classifier and the encoders compared (Section 4 and Figure 1 of the paper).

Results

The authors compare text-only, audio-only, and text-audio models on 1,063 unique passage and snippet pairs, after removing duplicate snippets. They use leave-one-out cross-validation over debates: each of the 28 runs tests on one debate and trains on the others. Scores are macro-F1 averaged over the 28 runs, next to a majority-class baseline and a random baseline.

0.40best average macro-F1, text plus audio (BERT + Wav2Vec, RoBERTa + CLAP)
0.38best average macro-F1, text only (RoBERTa)
+8 pointsgain from adding Wav2Vec audio to BERT (0.32 to 0.40)

Average macro-F1 on MM-USED-fallacy (higher is better)

Random baseline 0.12 ± 0.05
Majority baseline 0.20 ± 0.17
Audio only, Wav2Vec 0.13 ± 0.07
Audio only, CLAP 0.12 ± 0.08
Text only, BERT 0.32 ± 0.13
Text only, RoBERTa 0.38 ± 0.18
Text only, SBERT 0.31 ± 0.18
BERT + Wav2Vec This work 0.40 ± 0.17
BERT + CLAP This work 0.36 ± 0.17
RoBERTa + Wav2Vec This work 0.39 ± 0.19
RoBERTa + CLAP This work 0.40 ± 0.19
SBERT + Wav2Vec This work 0.23 ± 0.11
SBERT + CLAP This work 0.24 ± 0.10

Average macro-F1 over the 28 runs; whiskers show the standard deviation reported in the paper. Source: Table 6 of the paper.

Show as table

System	Average macro-F1 on MM-USED-fallacy
Random baseline	0.12 ± 0.05
Majority baseline	0.20 ± 0.17
Audio only, Wav2Vec	0.13 ± 0.07
Audio only, CLAP	0.12 ± 0.08
Text only, BERT	0.32 ± 0.13
Text only, RoBERTa	0.38 ± 0.18
Text only, SBERT	0.31 ± 0.18
BERT + Wav2Vec (this work)	0.40 ± 0.17
BERT + CLAP (this work)	0.36 ± 0.17
RoBERTa + Wav2Vec (this work)	0.39 ± 0.19
RoBERTa + CLAP (this work)	0.40 ± 0.19
SBERT + Wav2Vec (this work)	0.23 ± 0.11
SBERT + CLAP (this work)	0.24 ± 0.10

Model	Appeal to Emotion	Appeal to Authority	Ad Hominem	False Cause	Slippery Slope	Slogans
BERT, text only	0.70	0.45	0.15	0.28	0.22	0.06
BERT + Wav2Vec	0.80	0.50	0.13	0.35	0.23	0.04
RoBERTa, text only	0.53	0.50	0.32	0.29	0.30	0.17
RoBERTa + CLAP	0.74	0.45	0.23	0.37	0.31	0.12

F1 per fallacy category, averaged over the 28 runs (Table 6 of the paper).

Audio alone is weak, close to the random baseline, but adding it to BERT or RoBERTa raises the average macro-F1. The gain is statistically significant for BERT (p-value below 0.05) but not for RoBERTa, and SBERT drops when audio is added. The gains are uneven across categories: Appeal to Emotion and False Cause improve the most, while text-only models are as good or better on Slogans, the rarest category. Results also vary across debates, and recent debates tend to benefit more from audio.

Takeaways

A first multimodal fallacy corpus.MM-USED-fallacy pairs 1,891 annotated fallacy passages from 28 US presidential debates with their audio. The corpus and code are public.
Audio helps text, not alone.Adding audio raises average macro-F1 from 0.32 to 0.40 for BERT, a statistically significant gain. For RoBERTa it rises from 0.38 to 0.40, a gain that is not significant, and audio-only models stay near the random baseline.
The benefit depends on the fallacy.Appeal to Emotion gains the most from audio, while Slogans do not. The annotations come from text only, so a new annotation that listens to the audio could reveal more.

Multimodal Argument Mining: A Case Study in Political Debates

Sat, 01 Jan 2022 00:00:00 +0000