Let Guidelines Guide You: A Prescriptive Guideline-Centered Data Annotation Methodology

Fri, 18 Sep 2026 00:00:00 +0000

Research setting

Content warning: this page contains an example of offensive content from the paper.

Many datasets for text classification are labeled by human annotators who follow written annotation guidelines. A guideline is one textual criterion, such as “The entry derogates women by comparing them to non-human entities”. A class set is the list of labels of a task, such as sexist and non-sexist. In the prescriptive paradigm, all annotators must apply one shared set of guidelines, and the standard annotation methodology (SAM) asks them to record only a class label for each text.

(a) A guideline set and a class set for sexism detection. (b) In SAM, the annotator knows how guidelines map to classes and records only the label. (c) In GCAM, the annotator sees only the guidelines and records which ones apply; a fixed mapping then gives the label (panels a to c of Figure 1 of the paper).

Motivation

When only the label is recorded, nobody knows which guideline the annotator used, or whether they used one at all. Agreement between annotators is then measured on labels only, which can hide disagreement on the guidelines. The labels are also tied to one class set, so a new formulation of the task needs a new round of human annotation. Finally, models never see the guidelines, so it is unclear whether they learn the task that the guidelines describe.

Annotation method	Records the guidelines used for each text	Annotators do not see the class set	Reuse with a new class set without new annotation	Annotators report one kind of annotation only
Standard annotation, SAM (Rottger et al., 2022)	no	no	no	yes
Guidelines plus labels (Jikeli et al., 2023)	yes	no	no	no
This paper (GCAM) This work	yes	yes	yes	yes

How GCAM relates to the annotation methods discussed in the paper (Sections 1, 2 and 4).

Objective. Define an annotation methodology in which annotators report the guidelines that apply to each text instead of a class label, and compare it with standard annotation in a human study and in experiments with machine learning models.

Approach

The Guideline-Centered Annotation Methodology (GCAM) splits annotation into two stages. First, annotators see only the guidelines and select the subset of guidelines that applies to each text. They never see the class set, so their choice must rest on the guidelines. Second, a class grounding function, fixed by design and hidden from annotators, maps the selected guidelines to class labels. Swapping this function gives labels for another class set from the same human annotation.

1. Annotate with guidelines

Input Data
Annotator Human work
Selected Data
Grounding
Labels

The input is a text and the guideline set, with no class set. The annotator reports the selected guidelines that apply. “Women are evil” gets guideline g1.

2. Ground to classes

Input
Annotator
Selected Data
Grounding Fixed
Labels Data

The class grounding function maps each selected guideline to a class, with no annotator involved. Guideline g1 maps to sexist.

3. Reuse for another task

Input
Annotator
Selected Data
Grounding Swapped
Labels Data

A different grounding function maps the same guidelines to another class set, for example animosity, threats, and derogation. No new human annotation is needed.

Swapped for a new class set Human work Fixed by design, not shown to annotators Data

The two stages of GCAM and the reuse of its annotations (Section 3 and Figure 1 of the paper).

The paper tests GCAM in two ways. In a human study, four annotators label 200 news sentences as subjective or objective with 12 guidelines, split in two batches of 100: one pair uses SAM and the other uses GCAM. In model experiments, encoder models and three open LLMs (Mistral, Llama, Phi) are trained or prompted to predict labels or guidelines on two datasets that record both: ToS-100 (unfair clauses in Terms of Service, five unfairness categories) and EDOS (online sexism).

Results

The human study measures inter-annotator agreement with Krippendorff’s alpha, on labels for both pairs and on selected guidelines for the GCAM pair, and the annotation time per batch. The model experiments report macro-F1, the average F1 score over classes, on labels and on guidelines, as the mean of 10 seed runs for encoders.

66 of 115guideline disagreements between GCAM annotators that still agree on the label
~150 minper batch of 100 sentences with GCAM, against ~60 min with SAM
80.14macro-F1 on ToS-A for the guideline-trained encoder, against 71.61 with labels only

Krippendorff’s alpha, as reported in the paper (higher is better)

SAM, class labels 57.90
GCAM, class labels This work 50.97
GCAM, guidelines This work 33.93

Inter-annotator agreement on all 200 sentences of the human study (Appendix B of the paper).

Show as table

System	Krippendorff's alpha, as reported in the paper
SAM, class labels	57.90
GCAM, class labels (this work)	50.97
GCAM, guidelines (this work)	33.93

Agreement on labels is comparable for the two methods (54.00 and 62.00 for SAM, 48.18 and 54.00 for GCAM, on the two batches). Agreement on guidelines is much lower (29.28 and 38.74), so agreeing on a broad set of guidelines is notably harder. GCAM makes this visible: 66 of the 115 disagreements (about 57%) agree on the label while reporting different guidelines, so a high label agreement can hide different reasons. The price is time, since reporting guidelines takes longer than reporting labels.

Dataset	Binary (SAM), labels	Fine-grained (GCAM), labels	Text-based (GCAM), labels	Fine-grained (GCAM), guidelines	Text-based (GCAM), guidelines
ToS-A	71.61	65.31	80.14 ^{∗, ∗∗}	18.33	37.49 ^∗∗
ToS-CH	87.96	83.61	90.23 ^{ns, ∗∗}	34.58	44.92 ^∗∗
ToS-CR	79.02	67.85	78.14 ^{ns, ∗∗}	17.37	27.21 ^∗∗
ToS-LTD	75.87	73.70	80.39 ^{∗∗, ∗∗}	19.87	21.49 ^ns
ToS-TER	86.86	78.80	84.57 ^{ns, ∗}	10.16	20.04 ^∗∗
EDOS	81.74	71.67	79.87 ^{∗, ∗∗}	27.09	22.97 ^∗∗

Macro-F1 of encoder models (LegalBERT on ToS-100, RoBERTaHate on EDOS), mean of 10 seed runs (Table 2 of the paper, left part). Binary predicts labels, Fine-grained treats each guideline as a class, Text-based reads each guideline text next to the input. Best values in bold, as in the paper. Superscripts give the Wilcoxon significance of Text-based against Binary and Fine-grained on labels, and against Fine-grained on guidelines: ns for not significant, ∗ for p ≤ 0.05, ∗∗ for p ≤ 0.01.

On labels, the Text-based encoder significantly outperforms the label-only Binary model on ToS-A and ToS-LTD, shows no significant difference on ToS-CH, ToS-CR, and ToS-TER, and is significantly below it on EDOS (79.87 against 81.74). It significantly outperforms Fine-grained on labels on every dataset, which shows the value of the guideline text. On guidelines, it significantly outperforms Fine-grained on four of the five ToS targets, shows no significant difference on ToS-LTD, and is significantly below it on EDOS (22.97 against 27.09). Guidelines also refine error analysis: for the Text-based model on EDOS, 423 test predictions have the right label for the wrong guideline, a case SAM counts as correct (Figure 2 of the paper). Prompted zero-shot, the LLMs lose much of their performance when asked for guidelines, and in EDOS 60 to 71% of their predictions get both guideline and label wrong (Section 6.3).

Takeaways

Record the reason, not only the label.When annotators report guidelines, one can check whether they followed them and see why they disagree, even when their labels match.
Annotate once, reuse for many tasks.Labels come from a separate fixed mapping, so the same guideline annotations produce labels for several class sets without new human work.
Guidelines help models and their evaluation.Encoders trained with guideline text perform as well as label-only training, and guideline errors separate right answers from right answers for the wrong reason.

On the Definition of Prescriptive Annotation Guidelines for Language-Agnostic Subjectivity Detection

Sun, 01 Jan 2023 00:00:00 +0000