LMAC-TD: Producing Time Domain Explanations for Audio Classifiers

Wed, 01 Jan 2025 00:00:00 +0000

Research setting

Audio classifiers are neural networks that assign a class to a sound clip, but their decision mechanisms remain opaque. A post-hoc explanation method leaves the trained classifier unchanged and adds a separate interpreter that points to the parts of the input behind the decision. For audio, a listenable explanation is itself a sound: the part of the clip that drove the decision, which a person can play and hear. A waveform is the raw sequence of audio samples, and a spectrogram is a time and frequency picture of the sound. This paper produces the explanation in the time domain, that is, directly as a waveform instead of as a spectrogram.

The task. The classifier stays fixed, and the interpreter returns an explanation i(t) plus the masked-out rest of the clip. Schema drawn for this page from the definitions in Section II of the paper.

Motivation

Earlier listenable methods such as SLIME and AudioLIME score regions or separated sources of the input. L2I and L-MAC (Listenable Maps for Audio Classifiers) train a decoder instead, and L-MAC drops the pre-trained NMF dictionary (a fixed set of spectral patterns) that L2I needs. L-MAC gives faithful explanations, but it masks the magnitude STFT, a spectrogram without phase. To turn the masked spectrogram back into sound, it reuses the phase of the input mixture, and this causes artifacts.

Method	Listenable explanation	Trained decoder	No pre-trained NMF dictionary	Waveform output without the input phase
SLIME (Mishra et al., 2017)	yes	no	n/a	not discussed
AudioLIME (Haunschmid et al., 2020)	yes	no	n/a	not discussed
L2I (Parekh et al., 2022)	yes	yes	no	not discussed
L-MAC (Paissan et al., 2024)	yes	yes	yes	no
This paper (LMAC-TD) This work	yes	yes	yes	yes

How the paper positions LMAC-TD against the listenable explanation methods it discusses (Section I of the paper).

Objective. Improve the audio quality of L-MAC explanations by generating them directly in the time domain, without losing faithfulness to the classifier.

Approach

LMAC-TD adds parts of SepFormer, a transformer model for time-domain source separation (splitting a recording into its sources), to the L-MAC interpreter. A UNet decoder maps the classifier representations, and a SepFormer encoder maps the input waveform, to two latent maps of the same shape. A weight α between 0 and 1 mixes the two maps, and the SepFormer MaskNet turns the mix into a mask over the encoded input. The SepFormer decoder turns the masked input into the explanation waveform, and the inverse mask gives the masked-out rest. A masking loss then trains the interpreter so that the frozen classifier keeps its decision on the explanation and loses it on the masked-out rest.

1. Read the classifier

Input audio Data
Classifier Frozen
UNet decoder
SepFormer
i(t)

The audio goes through the frozen classifier, a CNN14 network on a Mel spectrogram, and the interpreter keeps its last four representations.

2. Build the mask

Input audio Data
Classifier Frozen
UNet decoder Trained
SepFormer Trained
i(t)

The UNet decoder maps the representations to the SepFormer latent space, mixes them with the encoded input through α, and the MaskNet outputs a mask.

3. Decode the waveform

Input audio
Classifier
UNet decoder Trained
SepFormer Trained
i(t) Data

The SepFormer decoder turns the masked encoding into the explanation i(t), with no phase borrowed from the input.

4. Check against the classifier

Input audio
Classifier Frozen
UNet decoder Trained
SepFormer Trained
i(t) Data

The frozen classifier scores the explanation and the masked-out rest. The masking loss, with an ℓ1 penalty on the spectrogram of i(t) against trivial solutions, updates only the interpreter.

Trained Frozen Data

What is trained and what stays frozen in LMAC-TD (Section II and Figure 1 of the paper).

Results

The classifier is a CNN14 fine-tuned on ESC50, a dataset of environmental sounds, with added noise. Faithfulness means how well the explanation follows this classifier. It is measured in domain on ESC50, and out of domain on mixtures of two ESC50 clips and on clips with white noise or speech added. Average Increase (AI) is the rise in classifier confidence on the explanation, and Average Decrease (AD) is the confidence drop once the explanation is removed. Faithfulness (FF) and input fidelity (Fid-In) are two further scores from earlier work, and for all of them except AD higher is better. In a user study, 19 participants rated nine audio samples from 1 to 100 for match with the predicted class and for audio quality. The average rating is the Mean Opinion Score (MOS).

69.75AI in domain for LMAC-TD (α = 0.75), against 36.25 for L-MAC
0.91Fid-In in domain for LMAC-TD (α = 0.75), against 0.42 for L-MAC
19participants in the listening study

Average Increase (AI), in domain on ESC50 (higher is better)

Saliency 0.00
SHAP 0.00
L2I 1.63
GradCAM 8.50
L-MAC with fine-tuning 32.37
L-MAC 36.25
LMAC-TD, α = 0 This work 46.50
LMAC-TD, α = 1 This work 66.00
LMAC-TD, α = 0.75 This work 69.75

In-domain Average Increase on ESC50 for a selection of the methods in Table I of the paper, higher is better.

Show as table

System	Average Increase (AI), in domain on ESC50
Saliency	0.00
SHAP	0.00
L2I	1.63
GradCAM	8.50
L-MAC with fine-tuning	32.37
L-MAC	36.25
LMAC-TD, α = 0 (this work)	46.50
LMAC-TD, α = 1 (this work)	66.00
LMAC-TD, α = 0.75 (this work)	69.75

Setting	Method	AI ↑	AD ↓	FF ↑	Fid-In ↑
In domain	L-MAC	36.25	1.15	0.20	0.42
In domain	LMAC-TD, α = 0.75	69.75	2.10	0.42	0.91
Two-clip mixtures	L-MAC	60.63	4.82	0.39	0.81
Two-clip mixtures	LMAC-TD, α = 0.75	59.50	3.42	0.41	0.87
White noise added	L-MAC	83.62	1.50	0.33	0.86
White noise added	LMAC-TD, α = 0.75	63.50	3.06	0.35	0.83
Speech added	L-MAC	70.75	2.73	0.33	0.83
Speech added	LMAC-TD, α = 0.75	62.63	2.95	0.43	0.90

L-MAC against LMAC-TD with α = 0.75 (Tables I, II, and III of the paper). Arrows give the direction of improvement. The better value of each pair is in bold.

In domain, LMAC-TD with α = 0.75 beats L-MAC on every faithfulness metric except AD, which stays close. Out of domain, it keeps a higher FF in every setting, while L-MAC keeps a higher AI and, with added noise, a lower AD. A larger α, which gives more weight to the classifier representations, generally gives more faithful explanations. The complexity metric COMP (lower is better) is worse for LMAC-TD (10.53 against 4.71 in domain), which the authors relate to computing it on spectrograms of a waveform output. In the user study, LMAC-TD with α = 1 and α = 0.75 obtained the highest MOS, above L-MAC and L-MAC with fine-tuning, while α = 0 was comparable to L-MAC and still above L2I.

Takeaways