Disruptive situation detection on public transport through speech emotion recognition

Mon, 01 Jan 2024 00:00:00 +0000

Research setting

A disruptive situation is an emotionally charged event, such as people fighting or screaming, and detecting it in time on public transport can make passengers safer. The paper treats detection as speech emotion recognition (SER), where a model labels a short speech recording as disruptive (D) or non-disruptive (ND) from the emotion it expresses. The model reads two kinds of audio features. Mel-Frequency Cepstral Coefficients (MFCCs) summarize the sound spectrum frame by frame, and Root-Mean-Square Energy (RMSE) measures loudness. Training and test data come from four acted English datasets (RAVDESS, TESS, SAVEE, and CREMA-D), in which actors speak with a given emotion.

Disruptive situation detection framed as binary speech emotion recognition. Schema drawn from the paper’s definitions (Section 4.1.2); the emotion-to-class mapping is the paper’s.

Motivation

The main earlier work on public transport framed the problem as detecting screams and shouted speech. Yet disruption is not always loud: a passenger can be threatened quietly or be paralyzed by fear. A real system also has to work in a noisy train, with several people speaking at once, and for speakers of any gender. Lastly, many SER studies split data at random. The same actor can then appear in training and test, which makes results look better than they are.

Approach	Public transport setting	Emotion-based detection	Actor- and gender-independent splits	Domain-specific noise
Scream and shouted speech detection (Laffitte et al., 2016)	yes	no	not discussed	not discussed
SER with random data splits (Padi et al., 2020; de Pinto et al., 2020; Patel et al., 2022)	no	yes	no	not discussed
SER with leave-one-speaker-out or gender splits (Bitouk et al., 2010; Fu et al., 2020; Zhu et al., 2017)	no	yes	partially	not discussed
SER with white-noise augmentation (An and Ruan, 2021)	no	yes	not discussed	partially
This paper This work	yes	yes	yes	yes

How the paper relates to the related work it discusses (Sections 1, 2, and 4.1 of the paper).

Objective. Test whether disruptive situation detection can be framed as speech emotion recognition, with a model that is independent of speaker and gender, robust to noise, and robust to multiple overlapping voices.

Approach

The paper prepares the four datasets so that the evaluation is fair, then trains two lightweight classifiers on them. Each actor appears in only one split, and splits balance male and female actors when the dataset allows it. To mimic a train, the training audio is mixed with three real noise recordings. They capture the inside of a train, a freight train with squeaky wheels, and a small crowd of children playing. The two classifiers are a support vector machine (SVM) and a small convolutional neural network (CNN), both cheap enough for edge devices. The paper tests 64 combinations of 4 feature sets, 2 classifiers, and 8 training sets (each dataset with and without noise).

Two classes Map each emotion to disruptive or non-disruptive.
Fair splits Each actor in one split, genders balanced.
Train noise Overlay train or crowd noise, 2 dB below the voice.
Features 5-second clips at 16 kHz, 12 or 25 MFCCs, optional RMSE.
Classifier SVM or CNN, also combined in an ensemble.

The method of the paper, from raw recordings to a disruptive or non-disruptive label (Sections 4.1 and 4.2).

Results

The main metric is the F1 score on the disruptive class, which is high only when the model finds disruptive recordings without raising many false alarms. Models are tested on the clean test split of each dataset. RAVDESS, the reference dataset with 12 actresses and 12 actors, also gets noisy, single-gender, and overlapping-voice test sets.

0.91F1 (D) of the best CNN on the clean RAVDESS test set
0.87F1 (D) on noisy RAVDESS when the CNN is trained with noise
0.92F1 (D) of an ensemble that averages three models’ probabilities

Dataset	Best CNN F1 (D)	Best SVM F1 (D)
RAVDESS	0.91 (+0.21)	0.75 (+0.05)
TESS	0.77 (+0.04)	0.73 (+0.00)
SAVEE	0.67 (+0.00)	0.62 (-0.05)
CREMA-D	0.92 (+0.03)	0.89 (+0.00)

Best F1 score on the disruptive class on each clean test set, with the gain over a majority baseline, which always predicts the most frequent class, in parentheses. Source: Table 4 of the paper.

F1 on the disruptive class, noisy RAVDESS test set (higher is better)

CNN 0.73
CNN + noise augmentation 0.87
CNN + RMSE 0.76
SVM 0.70
SVM + noise augmentation 0.67
SVM + RMSE + noise augmentation 0.66

F1 (D) on the noise-corrupted RAVDESS test set for models trained on RAVDESS, with 25 MFCCs. Source: Table 6 of the paper.

Show as table

System	F1 on the disruptive class, noisy RAVDESS test set
CNN	0.73
CNN + noise augmentation	0.87
CNN + RMSE	0.76
SVM	0.70
SVM + noise augmentation	0.67
SVM + RMSE + noise augmentation	0.66

CNNs give the best results on all four datasets. On RAVDESS they beat the majority baseline by 0.21 F1 (D), while TESS, SAVEE, and CREMA-D remain hard. On clean test data, noise augmentation has almost no effect. On the noisy test set, it limits the CNN loss to 4 F1 points, against up to 17 points without it. The best RAVDESS CNN scores 0.94 on male-only and 0.89 on female-only test sets (Table 5), which suggests gender independence. With 2, 5, and 10 overlapping voices, RAVDESS models keep an F1 (D) of 0.89, 0.83, and 0.92, against 0.91 on single files (Table 7).

Takeaways