Promoting the Responsible Development of Speech Datasets for Mental Health and Neurological Disorders Research

Wed, 01 Jan 2025 00:00:00 +0000

Research setting

Mental health and neurological disorders (MHND) include depression, Alzheimer’s, bipolar disorder, and Parkinson’s. AI systems can help with MHND by detecting signs of a disorder in the way a person speaks. These systems learn from speech datasets: recordings of participants, each with a label for the disorder. The kind of speaking task that participants perform is called the discourse genre, for example a clinical interview, a reading aloud, or a speech task. Each dataset also has a source of content: personal interaction with participants, crowdsourcing through online platforms, or media already available online.

How a speech dataset for MHND is built, with DAIC-WoZ, the case study of Section 2 of the paper, as the example. Schema drawn from the paper’s description.

Motivation

Machine learning research focuses on models more than on how data are collected, yet biased or poorly documented data can make a system unreliable. In MHND, the data come from people with a disorder, who may share sensitive content when they speak freely. The paper shows the risk on DAIC-WoZ, one of the most used depression datasets. Its classes are imbalanced (65% negative, 35% positive). Its paper does not state how the PHQ-8 score maps to the binary label. There is also no way to report or fix its known errors.

Related work	Targets MHND	Speech data	Dataset guidance
Mental health ethics (Fadda et al., 2022, and others)	yes	no	partially
General datasheets and checklists (Gebru et al., 2021, and others)	no	no	yes
Healthcare datasets (Rostamzadeh et al., 2022)	partially	no	yes
Speech technology (Papakyriakopoulos et al., 2023, and others)	no	yes	yes
Aphasia speech data (Westerhout and Monachesi, 2006)	partially	yes	partially
This paper This work	yes	yes	yes

How the paper relates to the related work it discusses (Section 3 of the paper).

Objective. Define what a responsible speech dataset for MHND should offer, turn it into an actionable checklist, and use the checklist to review the existing datasets.

Approach

The paper first lists seven desiderata, the properties that a responsible MHND speech dataset should have. It then turns each desideratum into checklist items, labeled C1 to C7, that state what a dataset paper should report. It applies the checklist to 36 peer-reviewed papers, each releasing a public speech dataset recorded from people with an MHND. The analysis groups the datasets by target issue (the disorder), by discourse genre, and by source of content.

Case study Issues found in the DAIC-WoZ depression dataset.
Desiderata Seven properties of a responsible MHND speech dataset.
This work Checklist Items C1 to C7 that a dataset paper should report.
Survey 36 dataset papers checked, by disorder, genre, and source.

The method of the paper, from the case study to the survey (Sections 2, 4, 5, and 6).

Desideratum	What the checklist asks the paper to report
C1 Informed consent	Consent details shared with participants
C2 Data storage and security	Storage and security measures
C3 Data privacy	Anonymization and removal of personal information
C4 Accountability	Approval by an ethical board
C5 Fairness, bias, and diversity	Speakers by class, age, gender/sex, education, language, and ethnicity; bias awareness; diversity collection measures
C6 Data quality, validation, and maintainability	Recording settings; versioning and issue reporting; domain experts in the loop; diagnostic criteria
C7 Discourse genre	The genres performed and why they were chosen

The seven desiderata and their checklist items (Sections 4 and 5.2 of the paper). C5 and C6 also have items for Parkinson’s and Alzheimer’s datasets only.

Results

The survey counts how many of the 36 dataset papers report each checklist item. The papers cover eight target issues, with depression the most frequent (17 datasets).

20 / 36papers report informed consent (C1)
9 / 36papers report storage and security measures (C2)
2 / 36papers report the ethnicity of the speakers (C5-f)

Papers that report the item (higher is better)

C1 Informed consent 20
C2 Storage and security 9
C3 Privacy 10
C4 Ethical approval 21
C5-a Speakers by class 28
C5-b Age 25
C5-c Gender/sex 23
C5-d Education 12
C5-e Language 24
C5-f Ethnicity 2
C5-h Bias awareness 13
C5-i Diversity measures 7
C6-a Recording settings 12
C6-b Maintainability 8
C6-c Domain experts 17
C6-d Diagnostic criteria 8
C7-a Discourse genre 28
C7-b Genre motivation 17

Papers that report each checklist item, out of 36. Source: Table 1 of the paper. Items C5-g (Parkinson’s only) and C6-e (Alzheimer’s only) are left out.

Show as table

System	Papers that report the item
C1 Informed consent	20
C2 Storage and security	9
C3 Privacy	10
C4 Ethical approval	21
C5-a Speakers by class	28
C5-b Age	25
C5-c Gender/sex	23
C5-d Education	12
C5-e Language	24
C5-f Ethnicity	2
C5-h Bias awareness	13
C5-i Diversity measures	7
C6-a Recording settings	12
C6-b Maintainability	8
C6-c Domain experts	17
C6-d Diagnostic criteria	8
C7-a Discourse genre	28
C7-b Genre motivation	17

Item	Crowdsourcing (5 datasets)	Online media (2 datasets)
C1 Informed consent	3	0
C2 Storage and security	3	0
C4 Ethical approval	3	0
C5-i Diversity measures	5	0
C6-a Recording settings	0	0

Datasets that report each item, by source of content. Source: Table 4 of the paper.

Most papers describe their speakers and the discourse genre, so the community is aware that speaker distributions matter. About half do not report informed consent or ethical approval. Storage, privacy, and maintainability are reported even less. The gaps also depend on the source of content. All crowdsourced datasets report diversity measures, but none reports recording settings. The datasets built from online media report no consent, security, or ethical approval.

Takeaways