Corpus | Language Technologies Lab

A Corpus for Sentence-Level Subjectivity Detection on English News Articles

Wed, 01 May 2024 00:00:00 +0000

Research setting

Subjectivity detection decides whether a sentence states facts or carries the opinion of its author. The paper defines a sentence as subjective when its content is based on or influenced by personal feelings, tastes, or opinions, and as objective otherwise. It supports tasks such as sentiment analysis, bias detection, and fact-checking, where opinions must be separated from facts that can be checked. Training a classifier for this task needs a corpus: a collection of sentences labeled by human annotators who follow written annotation guidelines.

The task on two example sentences from the annotation guidelines (Table 8 of the paper). A model reads one sentence at a time and returns OBJ (objective) or SUBJ (subjective).

Two terms recur on this page:

Inter-annotator agreement (IAA) measures how consistently annotators label the same sentences. The paper uses Krippendorff’s alpha, where 1 means perfect agreement and 0 means agreement no better than chance.
Macro-F1 is the F1 score (the balance of precision and recall) computed for each class and then averaged, so objective and subjective sentences count equally. It ranges from 0 to 1, and higher is better.

Motivation

Perceiving subjectivity is hard even for experts. It depends on interpretation, background knowledge, and personal bias. The standard way to build corpora spots subjective words from a lexicon, a fixed list of opinion words for one language and domain. Moving such resources to another language needs extra tools such as machine translation. Guideline-based alternatives exist, but they struggle with ambiguous sentences and annotator bias.

Approach	No lexicons or machine translation	Disagreements discussed and resolved	Sentence level	Hand-labeled corpora in two languages
Lexicon-based corpus creation (Riloff and Wiebe, 2003; Wiebe and Riloff, 2005)	no	not discussed	yes	no
Evidentiality-based annotation (Wiebe et al., 1999b)	not discussed	certainty rating instead	yes	no
Multilingual detection through translation (Banea et al.)	no	not discussed	not discussed	no
This paper (NewsSD-ENG) This work	yes	yes	yes	yes

How the paper positions itself against the related work it discusses (Sections 1 and 6 of the paper).

Objective. Write annotation guidelines for sentence-level subjectivity that do not depend on language-specific cues. Use them to build an English news corpus, and test whether they carry over to Italian.

Approach

The authors follow the prescriptive paradigm: annotators share one written belief about what counts as subjective, and every disagreement leads to a discussion and a clearer rule. Seven annotators with near-native English and a linguistics or computing background label the sentences. The articles come from 8 British news outlets and cover controversial political affairs, such as law, civil rights, and economics. The guidelines start from those of SubjectivITA, an earlier Italian news corpus, and are refined in two pilot studies.

Starting guidelines Adapted from the Italian SubjectivITA guidelines.
First pilot 270 sentences, IAA 0.40, edge cases discussed.
Second pilot With article context, IAA 0.38. Without it, 0.53.
Final guidelines Isolated sentences, five subjective cases, clear objective cases.

How the guidelines were built (Section 2 of the paper).

A sentence is subjective when its author expresses an explicit opinion, uses sarcasm or irony, exhorts, uses discriminating or downgrading words, or uses rhetorical figures to convey an opinion. Opinions of a third party, such as quotes, are objective. The author’s own stated emotions are objective too, since the author is a reliable source about their own feelings. Annotators see each sentence alone, because article context made them label ambiguous sentences as subjective more often and increased their workload.

Two annotators Each sentence is labeled independently by two annotators.
Discussion The pair discusses ambiguous sentences to agree.
Third annotator Breaks the tie in the remaining cases.

The three-stage annotation of the final corpus (Section 3 of the paper).

To test whether the guidelines carry over to other languages, the authors re-annotate quotes and emotions in SubjectivITA with the same criteria, creating NewsSD-ITA. They then train classifiers in three settings:

Setting	Training data	Test data
Monolingual	One language (en→en or it→it)	Same language
Multilingual	English and Italian together (en+it)	English or Italian
Cross-lingual	One language only	The other language, with no training examples in it

Results

The authors first measure corpus quality with IAA, and then compare classifiers by macro-F1 on the test sets of NewsSD-ENG (English) and NewsSD-ITA (Italian), averaged over three seed runs. Two classifiers use tf-idf word features: a support vector machine (SVM) and a logistic regressor (LR). Two are multilingual transformers: M-BERT and M-SBERT. A random and a majority-class baseline complete the comparison.

NewsSD-ENG holds 1,049 sentences from 23 articles: 638 objective and 411 subjective.

0.83IAA after the pair discussion, up from 0.51 after independent labeling
-0.21average IAA of a lexicon-based tool (TextBlob) with the annotators
0.80macro-F1 of M-BERT on English when trained on English and Italian, against 0.75 on English alone

Macro-F1 on the English test set (higher is better)

M-SBERT, monolingual 0.69
M-SBERT, multilingual 0.71
M-SBERT, cross-lingual 0.67
M-BERT, monolingual 0.75
M-BERT, multilingual 0.80
M-BERT, cross-lingual 0.60

Macro-F1 of the two transformers on the NewsSD-ENG test set in each setting. Source: Table 4 of the paper (average over three seed runs).

Show as table

System	Macro-F1 on the English test set
M-SBERT, monolingual	0.69
M-SBERT, multilingual	0.71
M-SBERT, cross-lingual	0.67
M-BERT, monolingual	0.75
M-BERT, multilingual	0.80
M-BERT, cross-lingual	0.60

Model	Setting	English macro-F1	English SUBJ F1	Italian macro-F1	Italian SUBJ F1
Majority baseline	Monolingual	0.33	0.00	0.42	0.00
Random baseline	Monolingual	0.50	0.50	0.47	0.36
SVM	Monolingual	0.44	0.24	0.59	0.34
LR	Monolingual	0.55	0.48	0.60	0.42
M-SBERT	Monolingual	0.69	0.69	0.69	0.56
M-BERT	Monolingual	0.75	0.71	0.74	0.59
SVM	Multilingual	0.49	0.34	0.60	0.35
LR	Multilingual	0.64	0.65	0.61	0.42
M-SBERT	Multilingual	0.71	0.76	0.69	0.56
M-BERT	Multilingual	0.80	0.80	0.77	0.66
M-SBERT	Cross-lingual	0.67	0.74	0.66	0.49
M-BERT	Cross-lingual	0.60	0.46	0.65	0.46

Macro-F1 and F1 on the subjective class (SUBJ). Source: Table 4 of the paper.

The annotation reaches near-perfect agreement after discussion, and a third annotator was needed for fewer than 10% of the sentences. The lexicon-based tool disagrees with the annotators, which suggests that lexicons do not suit this task. Training on English and Italian together never lowers performance and helps M-BERT most: macro-F1 rises by 5 points on English and 3 points on Italian. In the cross-lingual setting, M-SBERT stays close to its monolingual scores, while M-BERT loses performance.

Takeaways

A careful corpus for English news.NewsSD-ENG labels 1,049 news sentences with guidelines refined through pilot studies and discussion, reaching an IAA of 0.83.
The guidelines travel across languages.They rely on no lexicon or translation, and the matching Italian corpus can be added to training without any loss.
Mixing languages helps.Multilingual training gives the best macro-F1 on both test sets, 0.80 on English and 0.77 on Italian with M-BERT.

Multimodal Fallacy Classification in Political Debates

Fri, 01 Mar 2024 00:00:00 +0000

Research setting

Argument mining is the automatic analysis of arguments in text, such as finding claims and the reasons that support them. A fallacy is a deceptive, misleading, or invalid argument, and fallacy classification assigns each fallacious passage to a fallacy category. Most argument mining work reads only the words. Spoken debates also carry paralinguistic cues, such as accent, rhythm, and tone of voice. This paper treats fallacy classification in US presidential debates as a multimodal task, where each fallacy comes with its text and its audio clip.

Debate passage One passage, in two modalities.
- Text Transcript sentences
- Audio The matching recording
Classifier Combines what both modalities say.
Fallacy category One of six categories.

The task: a fallacious passage from a debate, as text and as audio, goes in, and one of six fallacy categories comes out.

Fallacy category	What it does	Example from the corpus
Appeal to Emotion	Uses loaded language.	“the same kind of woolly thinking”
Appeal to Authority	Uses an expert’s opinion as evidence.	“As George Will said the other day, …”
Ad Hominem	Attacks the arguer instead of the position.	“Governor Carter apparently doesn’t know the facts.”
False Cause	Takes correlation for causation.	“We won the Cold War because we invested and we went forward.”
Slippery Slope	Claims exaggerated outcomes for an action.	“And if we don’t act today, the problem will be valued in the trillions.”
Slogans	Brief, striking phrases that evoke excitement.	“We have to practice what we preach.”

The six fallacy categories (Section 3.1 and Table 1 of the paper).

Terms used on this page:

A snippet is the annotated fallacy within a debate passage.
A pretrained encoder is a large model trained beforehand on other data that turns text or audio into vectors. BERT, RoBERTa, and SBERT encode text; Wav2Vec and CLAP encode audio.
F1 balances precision and recall for one category. Macro-F1 averages F1 over the six categories, so rare categories count as much as frequent ones. It ranges from 0 to 1, and higher is better.

Motivation

Analyses of political speech link fallacies to the voice. Accent stereotypes go with Ad Hominem, a staccato rhythm with Appeal to Authority, and an angry tone with Appeal to Emotion. Multimodal argument mining already uses debate audio for argument detection, component classification, and relation classification. Fallacy classification, however, had only been studied on text, and no corpus paired annotated fallacies with audio.

Study	Political debates	Uses audio	Fallacy classification
Claim detection from speech (Lippi and Torroni, 2016)	yes	yes	no
Multimodal extensions of USED (Mancini et al., 2022; Mestre et al., 2023)	yes	yes	no
Text-only fallacy classification, USED-fallacy (Goffredo et al., 2022)	yes	no	yes
This paper (MM-USED-fallacy) This work	yes	yes	yes

How the paper positions itself against the related work it discusses (Sections 1 and 2 of the paper).

Objective. Build the first corpus for multimodal fallacy classification, and test whether adding audio to text improves the classification of fallacies in political debates.

Approach

The authors start from USED-fallacy, a text corpus of US presidential debates from 1960 to 2016 with span-level fallacy annotations. They map each annotated span to the sentences it overlaps. Sentence-level timestamps from an earlier multimodal version of the same debates then cut the matching audio clips. Fuzzy string matching handles small transcript differences, and misaligned samples are removed. The result, MM-USED-fallacy, holds 1,891 text-audio pairs from 28 debates.

Spans Fallacy spans from USED-fallacy.
Sentences Label every sentence that overlaps a fallacy span.
Audio Timestamps cut the clips; misaligned samples are removed.
Corpus MM-USED-fallacy, 1,891 text-audio pairs from 28 debates.

How MM-USED-fallacy is built (Section 3.2 of the paper).

The classifier keeps the architecture shown at the top of this page. A text module and an audio module each encode the passage, their outputs are concatenated, and a stack of dense layers predicts the category. The study compares three text encoders and two audio encoders, alone and in every text-audio pair.

Encoders One module per modality.
- Text module BERT, RoBERTa, or SBERT
- Audio module Wav2Vec or CLAP, then a BiLSTM
Concatenation Joins the two encodings.
Classification module Dense layers predict the category.

The classifier and the encoders compared (Section 4 and Figure 1 of the paper).

Results

The authors compare text-only, audio-only, and text-audio models on 1,063 unique passage and snippet pairs, after removing duplicate snippets. They use leave-one-out cross-validation over debates: each of the 28 runs tests on one debate and trains on the others. Scores are macro-F1 averaged over the 28 runs, next to a majority-class baseline and a random baseline.

0.40best average macro-F1, text plus audio (BERT + Wav2Vec, RoBERTa + CLAP)
0.38best average macro-F1, text only (RoBERTa)
+8 pointsgain from adding Wav2Vec audio to BERT (0.32 to 0.40)

Average macro-F1 on MM-USED-fallacy (higher is better)

Random baseline 0.12 ± 0.05
Majority baseline 0.20 ± 0.17
Audio only, Wav2Vec 0.13 ± 0.07
Audio only, CLAP 0.12 ± 0.08
Text only, BERT 0.32 ± 0.13
Text only, RoBERTa 0.38 ± 0.18
Text only, SBERT 0.31 ± 0.18
BERT + Wav2Vec This work 0.40 ± 0.17
BERT + CLAP This work 0.36 ± 0.17
RoBERTa + Wav2Vec This work 0.39 ± 0.19
RoBERTa + CLAP This work 0.40 ± 0.19
SBERT + Wav2Vec This work 0.23 ± 0.11
SBERT + CLAP This work 0.24 ± 0.10

Average macro-F1 over the 28 runs; whiskers show the standard deviation reported in the paper. Source: Table 6 of the paper.

Show as table

System	Average macro-F1 on MM-USED-fallacy
Random baseline	0.12 ± 0.05
Majority baseline	0.20 ± 0.17
Audio only, Wav2Vec	0.13 ± 0.07
Audio only, CLAP	0.12 ± 0.08
Text only, BERT	0.32 ± 0.13
Text only, RoBERTa	0.38 ± 0.18
Text only, SBERT	0.31 ± 0.18
BERT + Wav2Vec (this work)	0.40 ± 0.17
BERT + CLAP (this work)	0.36 ± 0.17
RoBERTa + Wav2Vec (this work)	0.39 ± 0.19
RoBERTa + CLAP (this work)	0.40 ± 0.19
SBERT + Wav2Vec (this work)	0.23 ± 0.11
SBERT + CLAP (this work)	0.24 ± 0.10

Model	Appeal to Emotion	Appeal to Authority	Ad Hominem	False Cause	Slippery Slope	Slogans
BERT, text only	0.70	0.45	0.15	0.28	0.22	0.06
BERT + Wav2Vec	0.80	0.50	0.13	0.35	0.23	0.04
RoBERTa, text only	0.53	0.50	0.32	0.29	0.30	0.17
RoBERTa + CLAP	0.74	0.45	0.23	0.37	0.31	0.12

F1 per fallacy category, averaged over the 28 runs (Table 6 of the paper).

Audio alone is weak, close to the random baseline, but adding it to BERT or RoBERTa raises the average macro-F1. The gain is statistically significant for BERT (p-value below 0.05) but not for RoBERTa, and SBERT drops when audio is added. The gains are uneven across categories: Appeal to Emotion and False Cause improve the most, while text-only models are as good or better on Slogans, the rarest category. Results also vary across debates, and recent debates tend to benefit more from audio.

Takeaways

A first multimodal fallacy corpus.MM-USED-fallacy pairs 1,891 annotated fallacy passages from 28 US presidential debates with their audio. The corpus and code are public.
Audio helps text, not alone.Adding audio raises average macro-F1 from 0.32 to 0.40 for BERT, a statistically significant gain. For RoBERTa it rises from 0.38 to 0.40, a gain that is not significant, and audio-only models stay near the random baseline.
The benefit depends on the fallacy.Appeal to Emotion gains the most from audio, while Slogans do not. The annotations come from text only, so a new annotation that listens to the audio could reveal more.

Detecting Arguments in CJEU Decisions on Fiscal State Aid

Sat, 01 Jan 2022 00:00:00 +0000

Multimodal Argument Mining: A Case Study in Political Debates

Sat, 01 Jan 2022 00:00:00 +0000

SubjectivITA: An Italian Corpus for Subjectivity Detection in Newspapers

Fri, 01 Jan 2021 00:00:00 +0000

Automated Processing of Privacy Policies Under the EU General Data Protection Regulation

Mon, 01 Jan 2018 00:00:00 +0000