Investigating the effectiveness of explainability methods in Parkinson’s detection from speech

Wed, 01 Jan 2025 00:00:00 +0000

Research setting

Parkinson’s disease (PD) is a progressive disorder of the nervous system, and speech problems often appear in its early stages. Speech analysis is non-invasive and cheap, so models that detect PD from a voice recording are a promising aid to diagnosis. These models are accurate, but they do not say why they label a person as PD or healthy control (HC). This paper studies post-hoc explanation methods, which leave a trained detector unchanged and return a saliency map. A saliency map gives a score to each cell of the spectrogram, a time and frequency picture of the sound, and so shows which parts drove the label.

The task. The detector stays fixed, and an explanation method marks the spectrogram cells behind its decision. Illustrative schema drawn for this page from the definitions in Section III of the paper.

Terms used on this page:

Gradient-based methods build the map from how the detector’s output changes with each input cell. The paper tests Saliency, SmoothGrad, Integrated Gradients, Guided GradCAM, Guided Backprop, and Gradient SHAP.
Faithfulness means that the map agrees with the detector: keeping the marked cells should keep the decision, and removing them should change it.
IoU (intersection over union) measures how much two saliency maps overlap, from 0 (no shared cells) to 1 (identical maps).

Motivation

Most work on speech-based PD detection reports accuracy and pays little attention to interpretability. Some studies show GradCAM or EigenCAM maps, but they do not check them with numbers. Studies that use SHAP explain features such as MFCCs (a compact summary of the sound spectrum), which do not map directly to what a clinician hears.

Study	Explains the decision	Quantitative check of the explanations	Several methods compared
CNNs on spectrograms (Sonawane and Sharma, 2021; Rios-Urrego et al., 2022)	no	no	no
GradCAM or EigenCAM maps for PD detection	yes	no	not discussed
SHAP on MFCC features (Maffia et al., 2023)	yes	not discussed	no
This paper This work	yes	yes	yes

How the paper positions itself against the PD detection work it discusses (Section II of the paper).

Objective. Evaluate several mainstream explanation methods for a speech-based PD detector in a systematic way, and find out whether their saliency maps highlight speech features that are specific to PD.

Approach

The detector is HuBERT, a speech model pre-trained without labels and fine-tuned to tell PD from HC. HuBERT reads the raw waveform, so the authors compute the explanations on the spectrogram and turn it back into a waveform before it enters the detector. Each saliency map is then scored with faithfulness metrics, alone and combined with a second map by intersection or union. Then a new classifier, CNN14, learns to separate PD from HC using only the saliency maps. Selective metrics scale its accuracy and F1 score by one minus the mean value of the map, so a map that marks the whole input scores lower.

1. Explain the detector

Speech Data
HuBERT detector Frozen
Saliency maps Data
Metrics
CNN14 classifier

HuBERT is trained on the PD task and then kept fixed. Six explanation methods compute a saliency map for each test recording.

2. Score the maps

Speech Data
HuBERT detector Frozen
Saliency maps Data
Metrics Data
CNN14 classifier

Metrics check whether masking the input with each map keeps or removes the decision of the fixed detector. Pairs of maps are also combined by intersection and union.

3. Learn from the maps

Speech
HuBERT detector
Saliency maps Data
Metrics
CNN14 classifier Trained

CNN14 is trained on the saliency maps alone, to test how much PD information they carry.

Trained Frozen Data

The evaluation protocol, from the detector to the classifier trained on its explanations (Section III of the paper).

Results

The data is s-PC-GITA, Spanish recordings of 50 people with PD and 50 healthy controls, with results averaged over 10 folds. The HuBERT detector reaches 81.32% accuracy and 81.03% F1 score. Average Increase (AI) is the share of samples where the detector grows more confident on the explanation, and Average Decrease (AD) is the drop in confidence once the input is masked. The other faithfulness metrics are Average Gain (AG), Faithfulness (FF), and input fidelity (Fid-In), while Sparseness (SPS) and Complexity (COMP) measure how concise a map is.

0.89accuracy of CNN14 trained on Integrated Gradients maps
0.82accuracy of CNN14 trained on the original spectrograms
ρ = -0.92correlation between AD and IoU for intersected maps

Method	AI ↑	AD ↓	FF ↑	Fid-In ↑	SPS ↑	COMP ↓
Saliency	74.66	1.80	0.004	82.82	0.69	11.98
SmoothGrad	75.10	1.85	0.004	81.89	0.50	12.51
Guided GradCAM	64.99	5.24	0.001	82.85	0.83	10.69
Guided Backprop	75.21	1.62	0.005	82.67	0.70	11.94
Integrated Gradients	78.83	1.22	0.013	81.93	0.77	11.69
Gradient SHAP	76.55	2.26	0.004	81.93	0.69	11.99

Mean over 10 folds on s-PC-GITA; arrows show the better direction and bold marks the best value (Table I of the paper, which also reports AG and standard deviations).

Accuracy of CNN14 (PD vs HC) (higher is better)

Original spectrograms 0.82 ± 0.12
Guided GradCAM maps 0.78 ± 0.09
Gradient SHAP maps 0.84 ± 0.13
SmoothGrad maps 0.86 ± 0.11
Saliency maps 0.87 ± 0.12
Guided Backprop maps 0.87 ± 0.10
Integrated Gradients maps 0.89 ± 0.08

Accuracy of the CNN14 classifier trained on each method’s saliency maps, against training on the original spectrograms; whiskers show one standard deviation. Source: Table III of the paper.

Show as table

System	Accuracy of CNN14 (PD vs HC)
Original spectrograms	0.82 ± 0.12
Guided GradCAM maps	0.78 ± 0.09
Gradient SHAP maps	0.84 ± 0.13
SmoothGrad maps	0.86 ± 0.11
Saliency maps	0.87 ± 0.12
Guided Backprop maps	0.87 ± 0.10
Integrated Gradients maps	0.89 ± 0.08

All six methods give comparable faithfulness, and the large spread across folds follows the spread of the detector itself. Integrated Gradients is best on AI, AD, AG, and FF, while Guided GradCAM gives the most concise maps. When two maps are intersected, a larger overlap goes with better scores: FF rises with IoU (ρ = 0.89) and AD falls (ρ = -0.92). In most cases, a classifier trained on the maps beats one trained on the original spectrograms, and Integrated Gradients also has the highest selective accuracy, 0.88. Still, the maps tend to focus on high-frequency regions and are not easy for people to interpret.

Takeaways