Untangling Hate Speech Definitions: A Semantic Componential Analysis Across Cultures and Domains

Tue, 01 Apr 2025 00:00:00 +0000

Research setting

Hate speech detection is the task of deciding whether a text, such as a social media post, contains hate speech. Datasets and models for this task rest on a definition of hate speech. No definition is universally accepted, and what counts as hate depends on culture. The paper reads each definition as a set of definitional components: the elements that a definition uses to say what hate speech is. Components fall under three parent categories: the target (who is attacked), the intent or purpose, and the act or means.

Hate speech denotes speech intended to degradeIntent/purpose groups of people on the basis of their race,Target gender,Target age,Target color.Target

A hate speech definition from the paper and the components that annotators mark in it (Figure 1 of the paper).

Definitions come from several domains, the contexts where they emerge: laws, Wikipedia, online dictionaries, research papers, and the conduct policies of online platforms. A large language model (LLM) can detect hate speech zero-shot: it receives a definition and a text in the prompt and answers without any task-specific training.

Motivation

Two definitions with different components can give a text two different labels. Suppose only one of them lists political opinion as a target. A post that attacks a political group is then hate under the first and not hate under the second. Most NLP research works on English data, so the cultural side of definitions is often overlooked. Prior work compares labels of harmful language or proposes criteria for writing definitions. None of it breaks hate speech definitions themselves into components.

Approach	Studies hate speech definitions	Breaks definitions into components	Compares definitions across domains and cultures	Tests the effect of definitions on LLMs
Formal concept analysis and conceptual primitives (Kamphuis and Sarbo, 1998; Cambria et al., 2016)	no	yes	no	no
Harmful language categories (Fortuna et al., 2020)	no	no	no	no
Review of abusive, offensive, toxic, and uncivil (Pachinger et al., 2023)	no	no	partially	no
Modular hate speech criteria (Khurana et al., 2022)	yes	partially	no	no
This paper (SCA and HateDefCon) This work	yes	yes	yes	yes

How the paper relates to the prior work it discusses (Section 2).

Objective. Build a dataset of hate speech definitions annotated with their components, compare the definitions across domains and cultures, and test how the definition in the prompt changes LLM hate speech detection.

Approach

Semantic Componential Analysis (SCA) is a technique from linguistics that describes a meaning by the components that are present or absent in it. The paper applies SCA to 493 hate speech definitions from more than 100 cultures and builds the HateDefCon dataset. Keyword extraction proposes an initial list of components. Three annotators then mark each component of each definition as present or absent, with an average agreement (Cohen’s kappa) of 0.64. The result is the component hierarchy shown at the top of this page.

The HateDefCon pipeline. Definitions come from five domains, keyword extraction proposes components, and annotators mark the components present in each definition (Figure 1 of the paper).

Domain	Definitions	Cultures
Platform policies	278	Each culture appears once per platform
Laws	116	Each culture appears once
Wikipedia	49	Each culture appears once
Research papers	29	English 13, German 3, Arabic 2, Indonesian 2, the rest once
Dictionaries	21	English 7, Italian 5, the rest once
Total	493

Definitions and their cultures per domain in HateDefCon. Source: Table 1 of the paper.

To test the effect of definitions, the paper prompts three open LLMs (Llama3, Mistral2, and Phi3-mini) with a definition and a post, and asks for a yes or no answer.

Definition One definition from HateDefCon, swapped between runs.
Post A social media post to classify.
LLM Llama3, Mistral2, or Phi3-mini, temperature zero.
Answer Yes or no, compared with the gold label.

The zero-shot experiment: only the definition in the prompt changes between runs (Sections 6 and 7 and Appendix G of the paper).

Results

The definition analysis compares domains, cultures, and components. The LLM experiment uses 500 posts from the Gab Hate Corpus, 250 hate and 250 not hate. It measures the F1 score of the binary task, from 0 to 1, where higher is better. It compares three definitions. Dghc is the definition used to annotate the corpus. Dwiki is a long and detailed definition from the Macedonian Wikipedia. Ddict is a short general definition from the Merriam-Webster dictionary.

17 of 30research papers borrow their definition from other papers, laws, or platform policies
0.78F1 of Llama3 with Dwiki, against 0.67 with Dghc
5%of posts that Llama3 refuses to answer with Ddict, against 1% with Dwiki

Model	Dghc	Dwiki	Ddict
Llama3	0.67	0.78	0.70
Mistral2	0.56	0.53	0.58
Phi3-mini	0.60	0.58	0.58

F1 score for binary hate speech detection with each definition in the prompt. Source: Table 3 of the paper.

Definitions vary widely in their components, and the most frequent target components are religion, race, ethnicity, and nationality. Only laws refer to a specific culture. Platform policies repeat one text translated into many languages, and Wikipedia often relies on the Cambridge Dictionary. The LLMs react to the definition in different ways: Llama3 gains with the detailed Dwiki, while Mistral2 shows the reverse tendency. With Dwiki, Llama3 also finds more attacks on individuals and on political views, which the other two definitions miss.

The paper also prompts Llama3 with definitions tied to Levantine Arabic, a language spoken in Syria, Jordan, and Israel among other countries.

F1 score of Llama3 (higher is better)

Levantine dataset 0.32
Israel law 0.74
Syria law 0.74
Jordan law 0.67

Llama3 F1 score with the definition of a Levantine hate speech dataset and with the hate speech laws of three countries where Levantine Arabic is spoken. Source: Section 7 of the paper.

Show as table

System	F1 score of Llama3
Levantine dataset	0.32
Israel law	0.74
Syria law	0.74
Jordan law	0.67

The Syrian law names only one target component, race, and still gives the best score together with the Israeli law. The paper concludes that the models carry their own biases and rely on their internal knowledge as well as on the definition.

Takeaways