Detecting Vague Clauses in Italian Privacy Policies Using Transformers, LLMs, and Cross-Lingual Techniques

Wed, 01 Jan 2025 00:00:00 +0000

Research setting

In the European Union, the General Data Protection Regulation (GDPR) requires a privacy policy to tell users which personal data a service processes. The paper studies the clauses of a policy that describe these categories of data, called CAT clauses. A legal expert labels each CAT clause as sufficiently informative, when the data are described in full, or insufficiently informative, also called vague, in all other cases.

The label follows from three annotated elements. A Category is the term that names the data, such as “geolocation data”. A Specification is an optional list introduced by words such as “such as” or “namely”, and each item of the list is a Subcategory, such as “GPS information”. Each element is either Open, when it is vague or open-ended, or Closed, when it is concrete or exhaustive.

We receive information about youOpen category from merchants as well as payment and transaction fulfillment providers, such asOpen specification payment confirmation details,Closed subcategory and information about the delivery of products you have purchasedClosed subcategory through our shopping features.

An insufficiently informative clause from the TikTok policy (Figure 1 of the paper). The Category is Open and the list is open-ended, so the clause is vague even though its two Subcategories are Closed. The Specification element spans the whole list; only its opening words are marked here.

Category	Specification	Subcategories	Label
Closed	none	none	Sufficient
Closed	any	any	Sufficient
Open	Closed	Closed	Sufficient
Open	none	none	Insufficient
Open	Open	any	Insufficient
Open	Closed	Open	Insufficient

The six rules that label a clause as sufficiently or insufficiently informative. “Any” means Open or Closed. Source: Table 2 of the paper.

Motivation

Empirical studies show that many privacy policies still fall short of the GDPR, and vague descriptions of data leave users unable to know what is collected about them. Automatic tools could flag such clauses, but previous work on insufficiently informative clauses covers English documents only. The EU has 24 official languages, and annotated legal corpora are scarce in many of them. So it matters whether a resource built for one language can serve another.

Work	Italian privacy policies	Insufficiently informative clauses	Cross-lingual transfer
Vague terms (Lebanoff and Liu, 2018; Malik et al., 2023)	no	partially	no
Vague data clauses in English (Grundler et al., 2024)	no	yes	no
Cross-lingual legal NLP (Chalkidis et al., 2021; Galassi et al., 2024)	no	no	yes
This paper This work	yes	yes	yes

How the paper relates to the prior work it discusses (Section 2 of the paper).

Objective. Find the most effective way to detect insufficiently informative clauses in Italian, and test whether annotated data can be transferred between English and Italian in both directions.

Approach

The authors build the first Italian corpus for this task: 30 privacy policies of the same online platforms as the 30 English policies of Grundler et al. (2024). A legal expert, an Italian native speaker trained on the English corpus, annotated it. In the last training round on English policies, the expert agreed with the English gold labels with a Cohen’s κ of 0.77 on the label. The corpus has 5,862 sentences and 875 CAT clauses, of which 220 are sufficiently informative.

Task	Input	Output
CAT classification	A sentence	CAT or not
Informativeness classification	A CAT clause	Sufficiently or insufficiently informative
Category and Subcategory detection	A CAT clause	The spans of Categories and Subcategories
Specification detection	A CAT clause	The spans of Specifications
Type classification	A Category or Subcategory	Open or Closed
Kind classification	A Category or Subcategory	One of 30 kinds of data, such as payment

The six tasks, each solved independently (Section 4.1 of the paper).

Two families of models solve each task. Four BERT models (three Italian models and one multilingual model) are fine-tuned on the Italian training split. Two large language models (LLMs), Gemini 2.0 Flash and Llama-3.1-8B-Instruct, answer from a prompt. The prompt has no examples (zero-shot) or 15 examples (few-shot, 30 for Kind), picked at random or as the nearest neighbours of the input. The cross-lingual experiments use no Italian training data at all.

English corpus 30 annotated English policies.
Transfer technique Three ways to reach Italian.
- Training set translation Translate the training data, train Italian models.
- Multilingual model Train a multilingual model in English.
- Test set translation Translate Italian test clauses to English.
Italian test set Scored on the new Italian corpus.

Transfer from English to Italian (Section 4.3.1 of the paper). Translation uses the open-source Opus-MT models. The reverse direction applies the same idea with the Italian corpus as training data and the English test set.

Results

Each system is scored on the Italian test split, which holds 20% of the policies, with the F1 score of each class and their average, the macro F1 (from 0 to 1, higher is better). BERT scores are averages over three fine-tuning runs.

0.79macro F1 on informativeness for Italian BERT trained on Italian data
0.80macro F1 on informativeness with no Italian training data (DeBERTa, test set translation)
0.84macro F1 on informativeness in English for Italian BERT trained on the Italian corpus

Macro F1 for informativeness classification (higher is better)

Gemini zero-shot 0.64
Llama zero-shot 0.64
Llama few-shot (nearest examples) 0.71
Gemini few-shot (nearest examples) 0.72
Multilingual BERT trained in English 0.65 ± 0.109
UmBERTo 0.78 ± 0.043
Italian BERT 0.79 ± 0.016
DeBERTa, test set translation 0.80 ± 0.005

Informativeness classification on the Italian test set. Source: Table 3 of the paper (whiskers show the standard deviation over three runs).

Show as table

System	Macro F1 for informativeness classification
Gemini zero-shot	0.64
Llama zero-shot	0.64
Llama few-shot (nearest examples)	0.71
Gemini few-shot (nearest examples)	0.72
Multilingual BERT trained in English	0.65 ± 0.109
UmBERTo	0.78 ± 0.043
Italian BERT	0.79 ± 0.016
DeBERTa, test set translation	0.80 ± 0.005

Setting	Model	CAT	Informativeness	Type	Kind
Zero-shot	Gemini	0.85	0.64	0.67	0.56
Few-shot, nearest examples	Gemini	0.86	0.72	0.76	0.54
Fine-tuned on Italian data	Italian BERT	0.86	0.79	0.80	0.44
Multilingual model	BERT multilingual	0.68	0.65	0.73	0.30
Training set translation	UmBERTo	0.85	0.80	0.73	0.32
Test set translation	DeBERTa	0.85	0.80	0.79	0.43

Macro F1 of the four classification tasks on the Italian test set for selected systems. Source: Table 3 of the paper.

Fine-tuned BERT models beat the LLMs on vagueness: Italian BERT reaches 0.79, against 0.72 for the best LLM. The one exception is Kind classification, with 30 classes, where Gemini zero-shot scores best with 0.56. In the detection tasks, Italian BERT and UmBERTo are always the best models, with up to 0.86 macro F1 for Specifications with BIO tags (Table 5 of the paper). Translating the Italian test set and reusing English models reaches the top informativeness score of 0.80, while a multilingual model trained in English drops to 0.65.

Training data for the English test set	Model	CAT	Informativeness	Type	Kind
Original English corpus	LEGAL-BERT	0.86	0.84	0.80	0.46
Italian corpus, translated to English	LEGAL-BERT	0.86	0.80	0.78	0.36
Italian corpus, test set translated to Italian	Italian BERT	0.85	0.84	0.77	0.35

Macro F1 on the English test set. Source: Table 4 of the paper.

In the other direction, models trained on the Italian corpus match the English-trained model in three of the four classification tasks, and fall 10 points behind only on Kind classification.

Takeaways