TRI-DEP: A Trimodal Comparative Study for Depression Detection Using Speech, Text, and EEG

Wed, 01 Oct 2025 00:00:00 +0000

Research setting

Automatic depression detection trains a model to tell people with Major Depressive Disorder (MDD) apart from healthy controls (HC), to support clinical decisions and telemedicine. A clinical session offers several signals at once, because human expression spans speech, language, and neural activity. This study uses the public MODMA dataset, which pairs a 5-minute resting-state EEG recording with the audio of a structured clinical interview for each person. The authors keep the 38 subjects who have both recordings.

Three signals per subject Each signal captures a different side of the person.
- EEG Electrical brain activity at rest, 128 channels
- Speech Audio of 29 interview answers
- Text Automatic transcripts of those answers
Classifier Learns depression cues from the signals.
Label MDD or healthy control.

The task: three signals from one person go in, and one label (MDD or healthy control) comes out.

Terms used on this page:

Handcrafted features are measurements designed by experts, such as spectral power for EEG or MFCCs (a standard description of the sound spectrum) and prosody for speech.
Pretrained embeddings are vectors produced by large models trained beforehand on other data, such as HuBERT for speech, MacBERT for text, and the brain-signal models LaBraM and CBraMod for EEG.
Late fusion trains one model per signal and combines only their final predictions.
Subject-level split means that all recordings of a person go either to training or to testing, never to both.
Macro-F1 is the F1 score (the balance of precision and recall) computed for each class and then averaged, so both classes count equally. It ranges from 0 to 1, and higher is better.

Motivation

Most prior systems combine only two signals, and most of them leave out text. Few compare handcrafted features with pretrained embeddings, and fusion strategies are rarely explored. Many studies also do not say how they split the data, and a segment-level split can place recordings of the same person in both training and test sets, which inflates scores.

Study	EEG + speech + text	Handcrafted vs pretrained features	Several fusion strategies	Clear subject-level splits
EEG and speech with image models (Yousufi et al.; Qayyum et al.)	no	not discussed	not discussed	no
Speech and text comparative analysis (Daly et al.)	no	not discussed	yes	not discussed
Trimodal GAT-CNN-MPNet (He et al., 2024)	yes	no	basic only	no
This paper (TRI-DEP) This work	yes	yes	yes	yes

How the paper positions itself against the related work it discusses (Introduction and Section 2.1 of the paper).

Objective. Build a reproducible benchmark for depression detection from EEG, speech, and text. Find out which features, models, and fusion strategies work best, and what EEG adds.

Approach

MODMA has no transcripts, so the authors transcribe the interviews automatically with WhisperX. For each signal, they compare handcrafted features with pretrained embeddings and train several small neural classifiers on top. They keep the best feature and classifier pair per signal and combine the three predictions with weighted averaging, Bayesian fusion, or majority voting. Fusion weights are chosen by grid search. Every experiment uses the same stratified 5-fold subject-level splits, which the authors release.

1. Extract features

Signals Data
Features Data
Classifier
Prediction
Fusion

Each signal is cut into segments and turned into handcrafted features or pretrained embeddings.

2. Train one classifier per signal

Signals Data
Features Data
Classifier Trained
Prediction Data
Fusion

A separate classifier (CNN, LSTM, or GRU based) learns to predict the label from each signal’s features.

3. Fuse the best predictions

Signals Data
Features Data
Classifier Frozen
Prediction Data
Fusion Optimized

The best classifier per signal is kept fixed, and its predictions are combined with fusion weights found by grid search.

Weights set by grid search Trained Frozen Data

The benchmark protocol, run on the same five subject-level folds.

Inputs EEG, speech, and WhisperX text.
Per-signal models Best features and classifier for each signal.
- EEG CBraMod and a CNN
- Speech HuBERT and a CNN with a BiGRU
- Text MacBERT and an LSTM
Late fusion Combines the three outputs.

The final framework: the best model for each signal, fused at the decision level.

Results

The authors measure macro-F1 on the 38 MODMA subjects, as the mean and standard deviation over the 5 subject-level folds. As baselines, they re-run two published EEG and speech image models (ViT and DenseNet-121) on the same splits, so these scores are not directly comparable to the original papers.

0.864macro-F1 when fusing EEG, speech, and text
0.809macro-F1 of the best single signal (speech)
0.586macro-F1 of the best re-run baseline

Macro-F1 (mean over 5 folds) (higher is better)

ViT baseline (EEG + speech) 0.560 ± 0.190
DenseNet-121 baseline (EEG + speech) 0.586 ± 0.240
EEG only (CBraMod + CNN) This work 0.446 ± 0.106
Text only (MacBERT + LSTM) This work 0.784 ± 0.114
Speech only (HuBERT) This work 0.809 ± 0.081
Speech + text, weighted averaging This work 0.839 ± 0.059
EEG + speech + text, weighted averaging This work 0.864 ± 0.095

Best configuration per setting, macro-F1 averaged over 5 subject-level folds; whiskers show one standard deviation. Source: Tables 3 and 4 of the paper.

Show as table

System	Macro-F1 (mean over 5 folds)
ViT baseline (EEG + speech)	0.560 ± 0.190
DenseNet-121 baseline (EEG + speech)	0.586 ± 0.240
EEG only (CBraMod + CNN) (this work)	0.446 ± 0.106
Text only (MacBERT + LSTM) (this work)	0.784 ± 0.114
Speech only (HuBERT) (this work)	0.809 ± 0.081
Speech + text, weighted averaging (this work)	0.839 ± 0.059
EEG + speech + text, weighted averaging (this work)	0.864 ± 0.095

Signal	Best handcrafted features	Best pretrained embeddings
EEG	0.418 (statistical, spectral, entropy)	0.446 (CBraMod)
Speech	0.655 (prosody + MFCCs)	0.809 (Chinese HuBERT)

Macro-F1, mean over 5 folds (Table 3 of the paper). Text was tested with pretrained embeddings only.

Speech is the strongest single signal, text comes close, and EEG alone stays below 0.5 macro-F1. Pretrained embeddings beat handcrafted features, most clearly for speech. Fusing all three signals gives the best score, 0.864 macro-F1 with both weighted averaging and Bayesian fusion, about 5.5 points above speech alone. With only 38 subjects and 5 folds, no pairwise difference is statistically significant at α = 0.05, so the gains show a consistent direction rather than a proven effect.

Takeaways