Unfair clause detection in terms of service across multiple languages

Mon, 01 Apr 2024 00:00:00 +0000

Research setting

Terms of Service (ToS) are the contracts that online services ask consumers to accept. Some of their clauses may be unfair to consumers under European consumer law, for example a clause that limits the liability of the provider. Unfair clause detection is the task of finding these clauses automatically, one sentence at a time. Legal experts give each relevant clause one of nine categories, such as choice of law (law) or jurisdiction (j). They also give it a level: 1 for clearly fair, 2 for potentially unfair, and 3 for clearly unfair.

This Agreement is governed by the laws of England and Wales and is subject to the exclusive jurisdiction of courts of England and Wales.Unfair (law2, j3) If you are a resident of Northern Ireland you may also bring proceedings in Northern Ireland, and if you are a resident of Scotland, you may also bring proceedings in Scotland.Fair (j1)

A real annotated clause from the English Klarna ToS, quoted in Appendix A of the paper. The first sentence is a potentially unfair choice of law (law2) and a clearly unfair jurisdiction clause (j3). The second sentence is a clearly fair jurisdiction clause (j1).

CLAUDETTE is such a detector for English, and it labels a sentence as positive when the experts marked it as unfair or potentially unfair.

Motivation

Most legal language technology is built for English. The European Union works in 24 official languages, and many ToS exist in several language versions. Annotating a new corpus for every language takes time and legal experts who are native speakers of that language. The language versions of one contract also differ: some are translations of the English text, while others add clauses for one country. For example, the Italian Klarna ToS replaces the clauses above with Italian law and the court of the consumer’s home, which the experts labeled as clearly fair.

Work	Legal documents	Unfair clauses in ToS	Annotation projection	Machine translation as a strategy	Several target languages
MULTI-EURLEX (Chalkidis et al., 2021)	yes	no	no	no	yes
Annotation projection for ToS (Galassi et al., 2020)	yes	yes	yes	no	no
Machine translation vs. new models (Isbister et al., 2021)	no	no	no	yes	yes
This paper This work	yes	yes	yes	yes	yes

How the paper relates to the work it discusses in Section 2. Columns list the properties that the paper uses to position itself.

Objective. Find out whether an English unfair clause detector can be extended to other languages without annotating a new corpus for each one. The paper compares four strategies on German, Italian, and Polish.

Approach

The authors first built a parallel corpus of the same 50 ToS in English, German, Italian, and Polish. Legal experts annotated all 200 documents. They then compared four strategies to obtain a detector for a target language, shown in the figure at the top of this page. Annotation projection copies the labels of each English sentence onto the matching sentence of the target-language version of the same contract. Machine translation means an off-the-shelf automatic translation tool.

1. New corpus

English corpus
Machine translation
Target-language corpus New labels
Classifier Trained

Legal experts annotate a new corpus in the target language, and a new classifier is trained on it. The English corpus and the English system are not used.

2. Projection

English corpus Data
Machine translation Used as is
Target-language corpus Data
Classifier Trained

The labels of the English corpus are projected onto the original target-language versions of the same contracts. Sentences are matched with sentence embeddings and dynamic time warping. A new classifier is trained on the projected labels.

3. Translate training set

English corpus Data
Machine translation Used as is
Target-language corpus Data
Classifier Trained

The English training documents are translated into the target language and keep their original labels. A new classifier is trained on the translated documents.

4. Translate queries

English corpus Data
Machine translation Used as is
Target-language corpus Data
Classifier Used as is

No new classifier is trained. At prediction time, each target-language sentence is translated into English and classified by the English system.

New annotation by legal experts Trained for the target language Used as is, no training Existing data

The four strategies (Section 4, Figure 1, and Table 2 of the paper). Each tab shows which resources a strategy needs, and whether a classifier is trained for the target language.

All strategies use the same classifier, a linear support vector machine (SVM) on word unigrams and bigrams. It beat sentence embeddings from ELMo and BERT in a preliminary test (Table 3 of the paper). Three translation tools of different quality are compared: Google Translate, Opus-MT, and the older statistical system Apache Joshua.

Results

Each strategy was tested on the German, Italian, and Polish documents with 5-fold cross-validation over documents. Precision is the share of flagged sentences that are truly unfair, and recall is the share of unfair sentences that are flagged. The measure is F1, from 0 to 1, which balances the two and is averaged over the five folds.

0.677F1 on German when queries are translated with Google Translate, against 0.638 with a new German corpus
0.662F1 on Italian when queries are translated with Opus-MT, against 0.621 with a new Italian corpus
0.645F1 on Polish when queries are translated with Google Translate, against 0.630 with a new Polish corpus

F1 on German ToS (higher is better)

1. New corpus
0.638
1. Projection
0.620
1. Translate training set (G)
0.620
1. Translate queries (G)
0.677
1. Translate queries (O)
0.676
1. Translate queries (J)
0.538

F1 of each strategy on German. G is Google Translate, O is Opus-MT, and J is Apache Joshua. Source: Table 4 of the paper.

Show as table

System	F1 on German ToS
1. New corpus	0.638
2. Projection	0.620
3. Translate training set (G)	0.620
4. Translate queries (G)	0.677
4. Translate queries (O)	0.676
4. Translate queries (J)	0.538

Strategy	German F1	Italian F1	Polish F1
1. New corpus	0.638	0.621	0.630
2. Projection	0.620	0.604	0.619
3. Translate training set (G)	0.620	0.612	0.608
4. Translate queries (G)	0.677	0.644	0.645
4. Translate queries (O)	0.676	0.662	0.575
4. Translate queries (J)	0.538	0.535	0.509

F1 of every strategy in each language. Source: Tables 4, 5, and 6 of the paper.

With Google Translate, translating the queries scored a higher F1 than a new corpus in all three languages. With Opus-MT, it did so in German and Italian but not in Polish. The gain was statistically significant (paired t-test, p < 0.05) for German with both tools and for Italian with Opus-MT. With the lower-quality Joshua translations, F1 dropped, and a new corpus was significantly better in all three languages. Projection was only slightly below a new corpus, with a significant gap for German only, and its loss came mostly from precision. Projection and training set translation performed alike in every language.

Takeaways

A new corpus per language is often not needed.An English detector with Google Translate for the queries scored a higher F1 than detectors trained on new German, Italian, and Polish corpora.
Translation quality decides the outcome.With the older Joshua system, F1 fell to between 0.509 and 0.538, and annotating a new corpus became the better choice.
Projection and translation are cheaper fallbacks.Projecting labels or translating the training set lost little F1 against a new corpus, and the authors release their four-language corpus and code for research.

Explaining Potentially Unfair Clauses to the Consumer with the CLAUDETTE tool

Wed, 01 Jan 2020 00:00:00 +0000

Consumer protection requires artificial intelligence

Mon, 01 Apr 2019 00:00:00 +0000

CLAUDETTE: an automated detector of potentially unfair clauses in online terms of service

Tue, 01 Jan 2019 00:00:00 +0000

Towards Consumer-Empowering Artificial Intelligence

Sun, 01 Jul 2018 00:00:00 +0000

Consumer Protection | Language Technologies Lab