Annotation Guidelines | Language Technologies Lab

Let Guidelines Guide You: A Prescriptive Guideline-Centered Data Annotation Methodology

Fri, 18 Sep 2026 00:00:00 +0000

Research setting

Content warning: this page contains an example of offensive content from the paper.

Many datasets for text classification are labeled by human annotators who follow written annotation guidelines. A guideline is one textual criterion, such as “The entry derogates women by comparing them to non-human entities”. A class set is the list of labels of a task, such as sexist and non-sexist. In the prescriptive paradigm, all annotators must apply one shared set of guidelines, and the standard annotation methodology (SAM) asks them to record only a class label for each text.

(a) A guideline set and a class set for sexism detection. (b) In SAM, the annotator knows how guidelines map to classes and records only the label. (c) In GCAM, the annotator sees only the guidelines and records which ones apply; a fixed mapping then gives the label (panels a to c of Figure 1 of the paper).

Motivation

When only the label is recorded, nobody knows which guideline the annotator used, or whether they used one at all. Agreement between annotators is then measured on labels only, which can hide disagreement on the guidelines. The labels are also tied to one class set, so a new formulation of the task needs a new round of human annotation. Finally, models never see the guidelines, so it is unclear whether they learn the task that the guidelines describe.

Annotation method	Records the guidelines used for each text	Annotators do not see the class set	Reuse with a new class set without new annotation	Annotators report one kind of annotation only
Standard annotation, SAM (Rottger et al., 2022)	no	no	no	yes
Guidelines plus labels (Jikeli et al., 2023)	yes	no	no	no
This paper (GCAM) This work	yes	yes	yes	yes

How GCAM relates to the annotation methods discussed in the paper (Sections 1, 2 and 4).

Objective. Define an annotation methodology in which annotators report the guidelines that apply to each text instead of a class label, and compare it with standard annotation in a human study and in experiments with machine learning models.

Approach

The Guideline-Centered Annotation Methodology (GCAM) splits annotation into two stages. First, annotators see only the guidelines and select the subset of guidelines that applies to each text. They never see the class set, so their choice must rest on the guidelines. Second, a class grounding function, fixed by design and hidden from annotators, maps the selected guidelines to class labels. Swapping this function gives labels for another class set from the same human annotation.

1. Annotate with guidelines

Input Data
Annotator Human work
Selected Data
Grounding
Labels

The input is a text and the guideline set, with no class set. The annotator reports the selected guidelines that apply. “Women are evil” gets guideline g1.

2. Ground to classes

Input
Annotator
Selected Data
Grounding Fixed
Labels Data

The class grounding function maps each selected guideline to a class, with no annotator involved. Guideline g1 maps to sexist.

3. Reuse for another task

Input
Annotator
Selected Data
Grounding Swapped
Labels Data

A different grounding function maps the same guidelines to another class set, for example animosity, threats, and derogation. No new human annotation is needed.

Swapped for a new class set Human work Fixed by design, not shown to annotators Data

The two stages of GCAM and the reuse of its annotations (Section 3 and Figure 1 of the paper).

The paper tests GCAM in two ways. In a human study, four annotators label 200 news sentences as subjective or objective with 12 guidelines, split in two batches of 100: one pair uses SAM and the other uses GCAM. In model experiments, encoder models and three open LLMs (Mistral, Llama, Phi) are trained or prompted to predict labels or guidelines on two datasets that record both: ToS-100 (unfair clauses in Terms of Service, five unfairness categories) and EDOS (online sexism).

Results

The human study measures inter-annotator agreement with Krippendorff’s alpha, on labels for both pairs and on selected guidelines for the GCAM pair, and the annotation time per batch. The model experiments report macro-F1, the average F1 score over classes, on labels and on guidelines, as the mean of 10 seed runs for encoders.

66 of 115guideline disagreements between GCAM annotators that still agree on the label
~150 minper batch of 100 sentences with GCAM, against ~60 min with SAM
80.14macro-F1 on ToS-A for the guideline-trained encoder, against 71.61 with labels only

Krippendorff’s alpha, as reported in the paper (higher is better)

SAM, class labels 57.90
GCAM, class labels This work 50.97
GCAM, guidelines This work 33.93

Inter-annotator agreement on all 200 sentences of the human study (Appendix B of the paper).

Show as table

System	Krippendorff's alpha, as reported in the paper
SAM, class labels	57.90
GCAM, class labels (this work)	50.97
GCAM, guidelines (this work)	33.93

Agreement on labels is comparable for the two methods (54.00 and 62.00 for SAM, 48.18 and 54.00 for GCAM, on the two batches). Agreement on guidelines is much lower (29.28 and 38.74), so agreeing on a broad set of guidelines is notably harder. GCAM makes this visible: 66 of the 115 disagreements (about 57%) agree on the label while reporting different guidelines, so a high label agreement can hide different reasons. The price is time, since reporting guidelines takes longer than reporting labels.

Dataset	Binary (SAM), labels	Fine-grained (GCAM), labels	Text-based (GCAM), labels	Fine-grained (GCAM), guidelines	Text-based (GCAM), guidelines
ToS-A	71.61	65.31	80.14 ^{∗, ∗∗}	18.33	37.49 ^∗∗
ToS-CH	87.96	83.61	90.23 ^{ns, ∗∗}	34.58	44.92 ^∗∗
ToS-CR	79.02	67.85	78.14 ^{ns, ∗∗}	17.37	27.21 ^∗∗
ToS-LTD	75.87	73.70	80.39 ^{∗∗, ∗∗}	19.87	21.49 ^ns
ToS-TER	86.86	78.80	84.57 ^{ns, ∗}	10.16	20.04 ^∗∗
EDOS	81.74	71.67	79.87 ^{∗, ∗∗}	27.09	22.97 ^∗∗

Macro-F1 of encoder models (LegalBERT on ToS-100, RoBERTaHate on EDOS), mean of 10 seed runs (Table 2 of the paper, left part). Binary predicts labels, Fine-grained treats each guideline as a class, Text-based reads each guideline text next to the input. Best values in bold, as in the paper. Superscripts give the Wilcoxon significance of Text-based against Binary and Fine-grained on labels, and against Fine-grained on guidelines: ns for not significant, ∗ for p ≤ 0.05, ∗∗ for p ≤ 0.01.

On labels, the Text-based encoder significantly outperforms the label-only Binary model on ToS-A and ToS-LTD, shows no significant difference on ToS-CH, ToS-CR, and ToS-TER, and is significantly below it on EDOS (79.87 against 81.74). It significantly outperforms Fine-grained on labels on every dataset, which shows the value of the guideline text. On guidelines, it significantly outperforms Fine-grained on four of the five ToS targets, shows no significant difference on ToS-LTD, and is significantly below it on EDOS (22.97 against 27.09). Guidelines also refine error analysis: for the Text-based model on EDOS, 423 test predictions have the right label for the wrong guideline, a case SAM counts as correct (Figure 2 of the paper). Prompted zero-shot, the LLMs lose much of their performance when asked for guidelines, and in EDOS 60 to 71% of their predictions get both guideline and label wrong (Section 6.3).

Takeaways

Record the reason, not only the label.When annotators report guidelines, one can check whether they followed them and see why they disagree, even when their labels match.
Annotate once, reuse for many tasks.Labels come from a separate fixed mapping, so the same guideline annotations produce labels for several class sets without new human work.
Guidelines help models and their evaluation.Encoders trained with guideline text perform as well as label-only training, and guideline errors separate right answers from right answers for the wrong reason.

Text Classification with Guidelines Only

Mon, 02 Mar 2026 00:00:00 +0000

Context

The standard way to train a machine learning model is to give it an annotated dataset: data paired with class labels. Annotators build this dataset after training on annotation guidelines, the rules that define how to assign the classes. They label the data with classes from a given class set. The model learns from the labels alone, without the guidelines, so it can learn any mapping from data to classes that fits the given data.

Objective

This proposal gives the annotation guidelines directly to the model. During training, the model receives the guidelines and unlabeled data, but no class labels.

The two settings this proposal compares. In standard training, only the annotators learn from the guidelines; in guidelines-only training, the model receives the guidelines and no class labels.

A Corpus for Sentence-Level Subjectivity Detection on English News Articles

Wed, 01 May 2024 00:00:00 +0000

Research setting

Subjectivity detection decides whether a sentence states facts or carries the opinion of its author. The paper defines a sentence as subjective when its content is based on or influenced by personal feelings, tastes, or opinions, and as objective otherwise. It supports tasks such as sentiment analysis, bias detection, and fact-checking, where opinions must be separated from facts that can be checked. Training a classifier for this task needs a corpus: a collection of sentences labeled by human annotators who follow written annotation guidelines.

The task on two example sentences from the annotation guidelines (Table 8 of the paper). A model reads one sentence at a time and returns OBJ (objective) or SUBJ (subjective).

Two terms recur on this page:

Inter-annotator agreement (IAA) measures how consistently annotators label the same sentences. The paper uses Krippendorff’s alpha, where 1 means perfect agreement and 0 means agreement no better than chance.
Macro-F1 is the F1 score (the balance of precision and recall) computed for each class and then averaged, so objective and subjective sentences count equally. It ranges from 0 to 1, and higher is better.

Motivation

Perceiving subjectivity is hard even for experts. It depends on interpretation, background knowledge, and personal bias. The standard way to build corpora spots subjective words from a lexicon, a fixed list of opinion words for one language and domain. Moving such resources to another language needs extra tools such as machine translation. Guideline-based alternatives exist, but they struggle with ambiguous sentences and annotator bias.

Approach	No lexicons or machine translation	Disagreements discussed and resolved	Sentence level	Hand-labeled corpora in two languages
Lexicon-based corpus creation (Riloff and Wiebe, 2003; Wiebe and Riloff, 2005)	no	not discussed	yes	no
Evidentiality-based annotation (Wiebe et al., 1999b)	not discussed	certainty rating instead	yes	no
Multilingual detection through translation (Banea et al.)	no	not discussed	not discussed	no
This paper (NewsSD-ENG) This work	yes	yes	yes	yes

How the paper positions itself against the related work it discusses (Sections 1 and 6 of the paper).

Objective. Write annotation guidelines for sentence-level subjectivity that do not depend on language-specific cues. Use them to build an English news corpus, and test whether they carry over to Italian.

Approach

The authors follow the prescriptive paradigm: annotators share one written belief about what counts as subjective, and every disagreement leads to a discussion and a clearer rule. Seven annotators with near-native English and a linguistics or computing background label the sentences. The articles come from 8 British news outlets and cover controversial political affairs, such as law, civil rights, and economics. The guidelines start from those of SubjectivITA, an earlier Italian news corpus, and are refined in two pilot studies.

Starting guidelines Adapted from the Italian SubjectivITA guidelines.
First pilot 270 sentences, IAA 0.40, edge cases discussed.
Second pilot With article context, IAA 0.38. Without it, 0.53.
Final guidelines Isolated sentences, five subjective cases, clear objective cases.

How the guidelines were built (Section 2 of the paper).

A sentence is subjective when its author expresses an explicit opinion, uses sarcasm or irony, exhorts, uses discriminating or downgrading words, or uses rhetorical figures to convey an opinion. Opinions of a third party, such as quotes, are objective. The author’s own stated emotions are objective too, since the author is a reliable source about their own feelings. Annotators see each sentence alone, because article context made them label ambiguous sentences as subjective more often and increased their workload.

Two annotators Each sentence is labeled independently by two annotators.
Discussion The pair discusses ambiguous sentences to agree.
Third annotator Breaks the tie in the remaining cases.

The three-stage annotation of the final corpus (Section 3 of the paper).

To test whether the guidelines carry over to other languages, the authors re-annotate quotes and emotions in SubjectivITA with the same criteria, creating NewsSD-ITA. They then train classifiers in three settings:

Setting	Training data	Test data
Monolingual	One language (en→en or it→it)	Same language
Multilingual	English and Italian together (en+it)	English or Italian
Cross-lingual	One language only	The other language, with no training examples in it

Results

The authors first measure corpus quality with IAA, and then compare classifiers by macro-F1 on the test sets of NewsSD-ENG (English) and NewsSD-ITA (Italian), averaged over three seed runs. Two classifiers use tf-idf word features: a support vector machine (SVM) and a logistic regressor (LR). Two are multilingual transformers: M-BERT and M-SBERT. A random and a majority-class baseline complete the comparison.

NewsSD-ENG holds 1,049 sentences from 23 articles: 638 objective and 411 subjective.

0.83IAA after the pair discussion, up from 0.51 after independent labeling
-0.21average IAA of a lexicon-based tool (TextBlob) with the annotators
0.80macro-F1 of M-BERT on English when trained on English and Italian, against 0.75 on English alone

Macro-F1 on the English test set (higher is better)

M-SBERT, monolingual 0.69
M-SBERT, multilingual 0.71
M-SBERT, cross-lingual 0.67
M-BERT, monolingual 0.75
M-BERT, multilingual 0.80
M-BERT, cross-lingual 0.60

Macro-F1 of the two transformers on the NewsSD-ENG test set in each setting. Source: Table 4 of the paper (average over three seed runs).

Show as table

System	Macro-F1 on the English test set
M-SBERT, monolingual	0.69
M-SBERT, multilingual	0.71
M-SBERT, cross-lingual	0.67
M-BERT, monolingual	0.75
M-BERT, multilingual	0.80
M-BERT, cross-lingual	0.60

Model	Setting	English macro-F1	English SUBJ F1	Italian macro-F1	Italian SUBJ F1
Majority baseline	Monolingual	0.33	0.00	0.42	0.00
Random baseline	Monolingual	0.50	0.50	0.47	0.36
SVM	Monolingual	0.44	0.24	0.59	0.34
LR	Monolingual	0.55	0.48	0.60	0.42
M-SBERT	Monolingual	0.69	0.69	0.69	0.56
M-BERT	Monolingual	0.75	0.71	0.74	0.59
SVM	Multilingual	0.49	0.34	0.60	0.35
LR	Multilingual	0.64	0.65	0.61	0.42
M-SBERT	Multilingual	0.71	0.76	0.69	0.56
M-BERT	Multilingual	0.80	0.80	0.77	0.66
M-SBERT	Cross-lingual	0.67	0.74	0.66	0.49
M-BERT	Cross-lingual	0.60	0.46	0.65	0.46

Macro-F1 and F1 on the subjective class (SUBJ). Source: Table 4 of the paper.

The annotation reaches near-perfect agreement after discussion, and a third annotator was needed for fewer than 10% of the sentences. The lexicon-based tool disagrees with the annotators, which suggests that lexicons do not suit this task. Training on English and Italian together never lowers performance and helps M-BERT most: macro-F1 rises by 5 points on English and 3 points on Italian. In the cross-lingual setting, M-SBERT stays close to its monolingual scores, while M-BERT loses performance.

Takeaways

A careful corpus for English news.NewsSD-ENG labels 1,049 news sentences with guidelines refined through pilot studies and discussion, reaching an IAA of 0.83.
The guidelines travel across languages.They rely on no lexicon or translation, and the matching Italian corpus can be added to training without any loss.
Mixing languages helps.Multilingual training gives the best macro-F1 on both test sets, 0.80 on English and 0.77 on Italian with M-BERT.

On the Definition of Prescriptive Annotation Guidelines for Language-Agnostic Subjectivity Detection

Sun, 01 Jan 2023 00:00:00 +0000