A Corpus for Sentence-Level Subjectivity Detection on English News Articles

Wed, 01 May 2024 00:00:00 +0000

Research setting

Subjectivity detection decides whether a sentence states facts or carries the opinion of its author. The paper defines a sentence as subjective when its content is based on or influenced by personal feelings, tastes, or opinions, and as objective otherwise. It supports tasks such as sentiment analysis, bias detection, and fact-checking, where opinions must be separated from facts that can be checked. Training a classifier for this task needs a corpus: a collection of sentences labeled by human annotators who follow written annotation guidelines.

The task on two example sentences from the annotation guidelines (Table 8 of the paper). A model reads one sentence at a time and returns OBJ (objective) or SUBJ (subjective).

Two terms recur on this page:

Inter-annotator agreement (IAA) measures how consistently annotators label the same sentences. The paper uses Krippendorff’s alpha, where 1 means perfect agreement and 0 means agreement no better than chance.
Macro-F1 is the F1 score (the balance of precision and recall) computed for each class and then averaged, so objective and subjective sentences count equally. It ranges from 0 to 1, and higher is better.

Motivation

Perceiving subjectivity is hard even for experts. It depends on interpretation, background knowledge, and personal bias. The standard way to build corpora spots subjective words from a lexicon, a fixed list of opinion words for one language and domain. Moving such resources to another language needs extra tools such as machine translation. Guideline-based alternatives exist, but they struggle with ambiguous sentences and annotator bias.

Approach	No lexicons or machine translation	Disagreements discussed and resolved	Sentence level	Hand-labeled corpora in two languages
Lexicon-based corpus creation (Riloff and Wiebe, 2003; Wiebe and Riloff, 2005)	no	not discussed	yes	no
Evidentiality-based annotation (Wiebe et al., 1999b)	not discussed	certainty rating instead	yes	no
Multilingual detection through translation (Banea et al.)	no	not discussed	not discussed	no
This paper (NewsSD-ENG) This work	yes	yes	yes	yes

How the paper positions itself against the related work it discusses (Sections 1 and 6 of the paper).

Objective. Write annotation guidelines for sentence-level subjectivity that do not depend on language-specific cues. Use them to build an English news corpus, and test whether they carry over to Italian.

Approach

The authors follow the prescriptive paradigm: annotators share one written belief about what counts as subjective, and every disagreement leads to a discussion and a clearer rule. Seven annotators with near-native English and a linguistics or computing background label the sentences. The articles come from 8 British news outlets and cover controversial political affairs, such as law, civil rights, and economics. The guidelines start from those of SubjectivITA, an earlier Italian news corpus, and are refined in two pilot studies.

Starting guidelines Adapted from the Italian SubjectivITA guidelines.
First pilot 270 sentences, IAA 0.40, edge cases discussed.
Second pilot With article context, IAA 0.38. Without it, 0.53.
Final guidelines Isolated sentences, five subjective cases, clear objective cases.

How the guidelines were built (Section 2 of the paper).

A sentence is subjective when its author expresses an explicit opinion, uses sarcasm or irony, exhorts, uses discriminating or downgrading words, or uses rhetorical figures to convey an opinion. Opinions of a third party, such as quotes, are objective. The author’s own stated emotions are objective too, since the author is a reliable source about their own feelings. Annotators see each sentence alone, because article context made them label ambiguous sentences as subjective more often and increased their workload.

Two annotators Each sentence is labeled independently by two annotators.
Discussion The pair discusses ambiguous sentences to agree.
Third annotator Breaks the tie in the remaining cases.

The three-stage annotation of the final corpus (Section 3 of the paper).

To test whether the guidelines carry over to other languages, the authors re-annotate quotes and emotions in SubjectivITA with the same criteria, creating NewsSD-ITA. They then train classifiers in three settings:

Setting	Training data	Test data
Monolingual	One language (en→en or it→it)	Same language
Multilingual	English and Italian together (en+it)	English or Italian
Cross-lingual	One language only	The other language, with no training examples in it

Results

The authors first measure corpus quality with IAA, and then compare classifiers by macro-F1 on the test sets of NewsSD-ENG (English) and NewsSD-ITA (Italian), averaged over three seed runs. Two classifiers use tf-idf word features: a support vector machine (SVM) and a logistic regressor (LR). Two are multilingual transformers: M-BERT and M-SBERT. A random and a majority-class baseline complete the comparison.

NewsSD-ENG holds 1,049 sentences from 23 articles: 638 objective and 411 subjective.

0.83IAA after the pair discussion, up from 0.51 after independent labeling
-0.21average IAA of a lexicon-based tool (TextBlob) with the annotators
0.80macro-F1 of M-BERT on English when trained on English and Italian, against 0.75 on English alone

Macro-F1 on the English test set (higher is better)

M-SBERT, monolingual 0.69
M-SBERT, multilingual 0.71
M-SBERT, cross-lingual 0.67
M-BERT, monolingual 0.75
M-BERT, multilingual 0.80
M-BERT, cross-lingual 0.60

Macro-F1 of the two transformers on the NewsSD-ENG test set in each setting. Source: Table 4 of the paper (average over three seed runs).

Show as table

System	Macro-F1 on the English test set
M-SBERT, monolingual	0.69
M-SBERT, multilingual	0.71
M-SBERT, cross-lingual	0.67
M-BERT, monolingual	0.75
M-BERT, multilingual	0.80
M-BERT, cross-lingual	0.60

Model	Setting	English macro-F1	English SUBJ F1	Italian macro-F1	Italian SUBJ F1
Majority baseline	Monolingual	0.33	0.00	0.42	0.00
Random baseline	Monolingual	0.50	0.50	0.47	0.36
SVM	Monolingual	0.44	0.24	0.59	0.34
LR	Monolingual	0.55	0.48	0.60	0.42
M-SBERT	Monolingual	0.69	0.69	0.69	0.56
M-BERT	Monolingual	0.75	0.71	0.74	0.59
SVM	Multilingual	0.49	0.34	0.60	0.35
LR	Multilingual	0.64	0.65	0.61	0.42
M-SBERT	Multilingual	0.71	0.76	0.69	0.56
M-BERT	Multilingual	0.80	0.80	0.77	0.66
M-SBERT	Cross-lingual	0.67	0.74	0.66	0.49
M-BERT	Cross-lingual	0.60	0.46	0.65	0.46

Macro-F1 and F1 on the subjective class (SUBJ). Source: Table 4 of the paper.

The annotation reaches near-perfect agreement after discussion, and a third annotator was needed for fewer than 10% of the sentences. The lexicon-based tool disagrees with the annotators, which suggests that lexicons do not suit this task. Training on English and Italian together never lowers performance and helps M-BERT most: macro-F1 rises by 5 points on English and 3 points on Italian. In the cross-lingual setting, M-SBERT stays close to its monolingual scores, while M-BERT loses performance.

Takeaways

A careful corpus for English news.NewsSD-ENG labels 1,049 news sentences with guidelines refined through pilot studies and discussion, reaching an IAA of 0.83.
The guidelines travel across languages.They rely on no lexicon or translation, and the matching Italian corpus can be added to training without any loss.
Mixing languages helps.Multilingual training gives the best macro-F1 on both test sets, 0.80 on English and 0.77 on Italian with M-BERT.

Overview of the CLEF-2024 CheckThat! Lab Task 2 on Subjectivity in News Articles

Mon, 01 Jan 2024 00:00:00 +0000

Overview of the CLEF-2023 CheckThat! Lab: Task 2 on Subjectivity Detection

Sun, 01 Jan 2023 00:00:00 +0000

News Articles | Language Technologies Lab