Multilingual | Language Technologies Lab

A Corpus for Sentence-Level Subjectivity Detection on English News Articles

Wed, 01 May 2024 00:00:00 +0000

Research setting

Subjectivity detection decides whether a sentence states facts or carries the opinion of its author. The paper defines a sentence as subjective when its content is based on or influenced by personal feelings, tastes, or opinions, and as objective otherwise. It supports tasks such as sentiment analysis, bias detection, and fact-checking, where opinions must be separated from facts that can be checked. Training a classifier for this task needs a corpus: a collection of sentences labeled by human annotators who follow written annotation guidelines.

The task on two example sentences from the annotation guidelines (Table 8 of the paper). A model reads one sentence at a time and returns OBJ (objective) or SUBJ (subjective).

Two terms recur on this page:

Inter-annotator agreement (IAA) measures how consistently annotators label the same sentences. The paper uses Krippendorff’s alpha, where 1 means perfect agreement and 0 means agreement no better than chance.
Macro-F1 is the F1 score (the balance of precision and recall) computed for each class and then averaged, so objective and subjective sentences count equally. It ranges from 0 to 1, and higher is better.

Motivation

Perceiving subjectivity is hard even for experts. It depends on interpretation, background knowledge, and personal bias. The standard way to build corpora spots subjective words from a lexicon, a fixed list of opinion words for one language and domain. Moving such resources to another language needs extra tools such as machine translation. Guideline-based alternatives exist, but they struggle with ambiguous sentences and annotator bias.

Approach	No lexicons or machine translation	Disagreements discussed and resolved	Sentence level	Hand-labeled corpora in two languages
Lexicon-based corpus creation (Riloff and Wiebe, 2003; Wiebe and Riloff, 2005)	no	not discussed	yes	no
Evidentiality-based annotation (Wiebe et al., 1999b)	not discussed	certainty rating instead	yes	no
Multilingual detection through translation (Banea et al.)	no	not discussed	not discussed	no
This paper (NewsSD-ENG) This work	yes	yes	yes	yes

How the paper positions itself against the related work it discusses (Sections 1 and 6 of the paper).

Objective. Write annotation guidelines for sentence-level subjectivity that do not depend on language-specific cues. Use them to build an English news corpus, and test whether they carry over to Italian.

Approach

The authors follow the prescriptive paradigm: annotators share one written belief about what counts as subjective, and every disagreement leads to a discussion and a clearer rule. Seven annotators with near-native English and a linguistics or computing background label the sentences. The articles come from 8 British news outlets and cover controversial political affairs, such as law, civil rights, and economics. The guidelines start from those of SubjectivITA, an earlier Italian news corpus, and are refined in two pilot studies.

Starting guidelines Adapted from the Italian SubjectivITA guidelines.
First pilot 270 sentences, IAA 0.40, edge cases discussed.
Second pilot With article context, IAA 0.38. Without it, 0.53.
Final guidelines Isolated sentences, five subjective cases, clear objective cases.

How the guidelines were built (Section 2 of the paper).

A sentence is subjective when its author expresses an explicit opinion, uses sarcasm or irony, exhorts, uses discriminating or downgrading words, or uses rhetorical figures to convey an opinion. Opinions of a third party, such as quotes, are objective. The author’s own stated emotions are objective too, since the author is a reliable source about their own feelings. Annotators see each sentence alone, because article context made them label ambiguous sentences as subjective more often and increased their workload.

Two annotators Each sentence is labeled independently by two annotators.
Discussion The pair discusses ambiguous sentences to agree.
Third annotator Breaks the tie in the remaining cases.

The three-stage annotation of the final corpus (Section 3 of the paper).

To test whether the guidelines carry over to other languages, the authors re-annotate quotes and emotions in SubjectivITA with the same criteria, creating NewsSD-ITA. They then train classifiers in three settings:

Setting	Training data	Test data
Monolingual	One language (en→en or it→it)	Same language
Multilingual	English and Italian together (en+it)	English or Italian
Cross-lingual	One language only	The other language, with no training examples in it

Results

The authors first measure corpus quality with IAA, and then compare classifiers by macro-F1 on the test sets of NewsSD-ENG (English) and NewsSD-ITA (Italian), averaged over three seed runs. Two classifiers use tf-idf word features: a support vector machine (SVM) and a logistic regressor (LR). Two are multilingual transformers: M-BERT and M-SBERT. A random and a majority-class baseline complete the comparison.

NewsSD-ENG holds 1,049 sentences from 23 articles: 638 objective and 411 subjective.

0.83IAA after the pair discussion, up from 0.51 after independent labeling
-0.21average IAA of a lexicon-based tool (TextBlob) with the annotators
0.80macro-F1 of M-BERT on English when trained on English and Italian, against 0.75 on English alone

Macro-F1 on the English test set (higher is better)

M-SBERT, monolingual 0.69
M-SBERT, multilingual 0.71
M-SBERT, cross-lingual 0.67
M-BERT, monolingual 0.75
M-BERT, multilingual 0.80
M-BERT, cross-lingual 0.60

Macro-F1 of the two transformers on the NewsSD-ENG test set in each setting. Source: Table 4 of the paper (average over three seed runs).

Show as table

System	Macro-F1 on the English test set
M-SBERT, monolingual	0.69
M-SBERT, multilingual	0.71
M-SBERT, cross-lingual	0.67
M-BERT, monolingual	0.75
M-BERT, multilingual	0.80
M-BERT, cross-lingual	0.60

Model	Setting	English macro-F1	English SUBJ F1	Italian macro-F1	Italian SUBJ F1
Majority baseline	Monolingual	0.33	0.00	0.42	0.00
Random baseline	Monolingual	0.50	0.50	0.47	0.36
SVM	Monolingual	0.44	0.24	0.59	0.34
LR	Monolingual	0.55	0.48	0.60	0.42
M-SBERT	Monolingual	0.69	0.69	0.69	0.56
M-BERT	Monolingual	0.75	0.71	0.74	0.59
SVM	Multilingual	0.49	0.34	0.60	0.35
LR	Multilingual	0.64	0.65	0.61	0.42
M-SBERT	Multilingual	0.71	0.76	0.69	0.56
M-BERT	Multilingual	0.80	0.80	0.77	0.66
M-SBERT	Cross-lingual	0.67	0.74	0.66	0.49
M-BERT	Cross-lingual	0.60	0.46	0.65	0.46

Macro-F1 and F1 on the subjective class (SUBJ). Source: Table 4 of the paper.

The annotation reaches near-perfect agreement after discussion, and a third annotator was needed for fewer than 10% of the sentences. The lexicon-based tool disagrees with the annotators, which suggests that lexicons do not suit this task. Training on English and Italian together never lowers performance and helps M-BERT most: macro-F1 rises by 5 points on English and 3 points on Italian. In the cross-lingual setting, M-SBERT stays close to its monolingual scores, while M-BERT loses performance.

Takeaways

A careful corpus for English news.NewsSD-ENG labels 1,049 news sentences with guidelines refined through pilot studies and discussion, reaching an IAA of 0.83.
The guidelines travel across languages.They rely on no lexicon or translation, and the matching Italian corpus can be added to training without any loss.
Mixing languages helps.Multilingual training gives the best macro-F1 on both test sets, 0.80 on English and 0.77 on Italian with M-BERT.

Unfair clause detection in terms of service across multiple languages

Mon, 01 Apr 2024 00:00:00 +0000

Research setting

Terms of Service (ToS) are the contracts that online services ask consumers to accept. Some of their clauses may be unfair to consumers under European consumer law, for example a clause that limits the liability of the provider. Unfair clause detection is the task of finding these clauses automatically, one sentence at a time. Legal experts give each relevant clause one of nine categories, such as choice of law (law) or jurisdiction (j). They also give it a level: 1 for clearly fair, 2 for potentially unfair, and 3 for clearly unfair.

This Agreement is governed by the laws of England and Wales and is subject to the exclusive jurisdiction of courts of England and Wales.Unfair (law2, j3) If you are a resident of Northern Ireland you may also bring proceedings in Northern Ireland, and if you are a resident of Scotland, you may also bring proceedings in Scotland.Fair (j1)

A real annotated clause from the English Klarna ToS, quoted in Appendix A of the paper. The first sentence is a potentially unfair choice of law (law2) and a clearly unfair jurisdiction clause (j3). The second sentence is a clearly fair jurisdiction clause (j1).

CLAUDETTE is such a detector for English, and it labels a sentence as positive when the experts marked it as unfair or potentially unfair.

Motivation

Most legal language technology is built for English. The European Union works in 24 official languages, and many ToS exist in several language versions. Annotating a new corpus for every language takes time and legal experts who are native speakers of that language. The language versions of one contract also differ: some are translations of the English text, while others add clauses for one country. For example, the Italian Klarna ToS replaces the clauses above with Italian law and the court of the consumer’s home, which the experts labeled as clearly fair.

Work	Legal documents	Unfair clauses in ToS	Annotation projection	Machine translation as a strategy	Several target languages
MULTI-EURLEX (Chalkidis et al., 2021)	yes	no	no	no	yes
Annotation projection for ToS (Galassi et al., 2020)	yes	yes	yes	no	no
Machine translation vs. new models (Isbister et al., 2021)	no	no	no	yes	yes
This paper This work	yes	yes	yes	yes	yes

How the paper relates to the work it discusses in Section 2. Columns list the properties that the paper uses to position itself.

Objective. Find out whether an English unfair clause detector can be extended to other languages without annotating a new corpus for each one. The paper compares four strategies on German, Italian, and Polish.

Approach

The authors first built a parallel corpus of the same 50 ToS in English, German, Italian, and Polish. Legal experts annotated all 200 documents. They then compared four strategies to obtain a detector for a target language, shown in the figure at the top of this page. Annotation projection copies the labels of each English sentence onto the matching sentence of the target-language version of the same contract. Machine translation means an off-the-shelf automatic translation tool.

1. New corpus

English corpus
Machine translation
Target-language corpus New labels
Classifier Trained

Legal experts annotate a new corpus in the target language, and a new classifier is trained on it. The English corpus and the English system are not used.

2. Projection

English corpus Data
Machine translation Used as is
Target-language corpus Data
Classifier Trained

The labels of the English corpus are projected onto the original target-language versions of the same contracts. Sentences are matched with sentence embeddings and dynamic time warping. A new classifier is trained on the projected labels.

3. Translate training set

English corpus Data
Machine translation Used as is
Target-language corpus Data
Classifier Trained

The English training documents are translated into the target language and keep their original labels. A new classifier is trained on the translated documents.

4. Translate queries

English corpus Data
Machine translation Used as is
Target-language corpus Data
Classifier Used as is

No new classifier is trained. At prediction time, each target-language sentence is translated into English and classified by the English system.

New annotation by legal experts Trained for the target language Used as is, no training Existing data

The four strategies (Section 4, Figure 1, and Table 2 of the paper). Each tab shows which resources a strategy needs, and whether a classifier is trained for the target language.

All strategies use the same classifier, a linear support vector machine (SVM) on word unigrams and bigrams. It beat sentence embeddings from ELMo and BERT in a preliminary test (Table 3 of the paper). Three translation tools of different quality are compared: Google Translate, Opus-MT, and the older statistical system Apache Joshua.

Results

Each strategy was tested on the German, Italian, and Polish documents with 5-fold cross-validation over documents. Precision is the share of flagged sentences that are truly unfair, and recall is the share of unfair sentences that are flagged. The measure is F1, from 0 to 1, which balances the two and is averaged over the five folds.

0.677F1 on German when queries are translated with Google Translate, against 0.638 with a new German corpus
0.662F1 on Italian when queries are translated with Opus-MT, against 0.621 with a new Italian corpus
0.645F1 on Polish when queries are translated with Google Translate, against 0.630 with a new Polish corpus

F1 on German ToS (higher is better)

1. New corpus
0.638
1. Projection
0.620
1. Translate training set (G)
0.620
1. Translate queries (G)
0.677
1. Translate queries (O)
0.676
1. Translate queries (J)
0.538

F1 of each strategy on German. G is Google Translate, O is Opus-MT, and J is Apache Joshua. Source: Table 4 of the paper.

Show as table

System	F1 on German ToS
1. New corpus	0.638
2. Projection	0.620
3. Translate training set (G)	0.620
4. Translate queries (G)	0.677
4. Translate queries (O)	0.676
4. Translate queries (J)	0.538

Strategy	German F1	Italian F1	Polish F1
1. New corpus	0.638	0.621	0.630
2. Projection	0.620	0.604	0.619
3. Translate training set (G)	0.620	0.612	0.608
4. Translate queries (G)	0.677	0.644	0.645
4. Translate queries (O)	0.676	0.662	0.575
4. Translate queries (J)	0.538	0.535	0.509

F1 of every strategy in each language. Source: Tables 4, 5, and 6 of the paper.

With Google Translate, translating the queries scored a higher F1 than a new corpus in all three languages. With Opus-MT, it did so in German and Italian but not in Polish. The gain was statistically significant (paired t-test, p < 0.05) for German with both tools and for Italian with Opus-MT. With the lower-quality Joshua translations, F1 dropped, and a new corpus was significantly better in all three languages. Projection was only slightly below a new corpus, with a significant gap for German only, and its loss came mostly from precision. Projection and training set translation performed alike in every language.

Takeaways

A new corpus per language is often not needed.An English detector with Google Translate for the queries scored a higher F1 than detectors trained on new German, Italian, and Polish corpora.
Translation quality decides the outcome.With the older Joshua system, F1 fell to between 0.509 and 0.538, and annotating a new corpus became the better choice.
Projection and translation are cheaper fallbacks.Projecting labels or translating the training set lost little F1 against a new corpus, and the authors release their four-language corpus and code for research.

Overview of the CLEF-2024 CheckThat! Lab Task 2 on Subjectivity in News Articles

Mon, 01 Jan 2024 00:00:00 +0000

LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain

Fri, 01 Dec 2023 00:00:00 +0000

On the Definition of Prescriptive Annotation Guidelines for Language-Agnostic Subjectivity Detection

Sun, 01 Jan 2023 00:00:00 +0000

Overview of the CLEF-2023 CheckThat! Lab: Task 2 on Subjectivity Detection

Sun, 01 Jan 2023 00:00:00 +0000