Speech Processing

Definition

Speech processing is the automatic analysis of spoken language from its audio signal, often together with a transcript of the words. The voice carries information that the words alone do not, such as tone and emphasis. This information can help tasks such as argument mining (Lippi and Torroni, 2016). We study what the audio adds to the text, how to explain the decisions of audio models, and how speech data are collected and documented.

Core concepts

Audio features

Audio models typically represent the audio signal with features before classification. Spectral features, such as Mel-frequency cepstral coefficients (MFCCs), describe how the energy of the signal is distributed over frequencies. Pretrained neural networks such as Wav2vec instead learn embeddings directly from the raw waveform.

AUDIOFEATURESMFCCsEmbedding

Transcripts

A speech recognizer turns the audio into text, so that text models can read it. Its errors carry over to later tasks: in one debate, claim detection on recognized text was no better than chance for one of the speakers.

Recognizerwho created 2 million jobs …

Modalities

A modality is a type of input, such as text or audio. A model can read the text only, listen to the audio only, or combine both, and comparing the three settings shows what each modality contributes.

TextAudioModel

Explanations

Neural audio models are typically black boxes. Post-hoc explanation methods, applied to a model after training, show which parts of the input the model relied on. Some produce a saliency map over the input, and others produce audio that a person can listen to.

EXPLANATIONRelied on

What does the audio add?

Many tasks on speech start from a transcript, a written version of what was said. A transcript keeps the words but loses how they were said, such as tone, rhythm, and emphasis. These paralinguistic cues can relate to the meaning of an argument. For example, an angry tone can signal an appeal to emotion (Mancini et al., 2024). The examples below compare models that read the transcript alone with models that also use the audio.

Claim detection decides whether a sentence states a claim, an opinion or thesis that the speaker asserts. Many speech recognizers write the words without punctuation, so a statement and a question with the same words give the same transcript. The pitch of the voice tells them apart.

1 What a listener hears

1. Said as a statement

we will cut taxes

Stress on cut, and the pitch falls at the end. Rate: 3.1 words per second.

2. Same words, asked as a question

we will cut taxes

No stress, and the pitch rises at the end. Rate: 3.0 words per second.

3. Other words, said firmly

good eveningpause
0.3 s all

Stress on evening, a short pause, and the pitch falls at the end. Rate: 2.0 words per second.

2 Answers

Is the sentence a claim?

Input123
AnnotatorsYesNoNo
Text onlyThe words, without punctuation Yes (correct) Yes (wrong) No (correct)
Audio onlyPitch, loudness, pauses, and rate Yes (correct) No (correct) Yes (wrong)
Text and audioBoth Yes (correct) No (correct) No (correct)
Illustrative example: the sentences, cue values, and answers are constructed. No recording or model output is shown. The task follows the claim detection setting of Lippi and Torroni (2016), who combine text and audio features of a televised political debate.

A fallacy is a deceptive, misleading, or generally invalid argument. An appeal to emotion seeks agreement by stirring feelings instead of giving reasons. A transcript keeps the words but loses the anger in the voice.

1 What a listener hears

1. Said calmly

this plan hurts families

Even loudness, and the pitch stays level. Rate: 2.8 words per second.

2. Same words, said with anger

this plan hurtspause
0.4 s families

Loud throughout, a pitch peak on hurts, and a pause for effect. Rate: 2.0 words per second.

3. Other words, said with anger

we vote on tuesday

Loud throughout, and a pitch peak on vote. Rate: 3.0 words per second.

2 Answers

Is the sentence an appeal to emotion?

Input123
AnnotatorsNoYesNo
Text onlyThe words, without punctuation No (correct) No (wrong) No (correct)
Audio onlyPitch, loudness, pauses, and rate No (correct) Yes (correct) Yes (wrong)
Text and audioBoth No (correct) Yes (correct) No (correct)
Illustrative example: the sentences, cue values, and answers are constructed. No recording or model output is shown. The task follows the fallacy classification setting of Mancini et al. (2024), who pair sentences of US presidential debates with their audio.

Depression detection predicts whether a person has depression from signals such as their speech. A transcript keeps what the person says, but not how flat or slow the voice is. The voice and the words can each carry signs that the other misses.

1 What a listener hears

1. Said with varied pitch

i slept well today

Varied pitch and a regular pace. Rate: 3.1 words per second.

2. Same words, said flat and slow

i slept wellpause
0.8 s today

Flat pitch, low loudness, a slow pace, and a long pause. Rate: 1.5 words per second.

3. Other words, said with varied pitch

nothing matters anymore

Varied pitch and a regular pace. Rate: 2.9 words per second.

2 Answers

Does the speaker show signs of depression?

Input123
Clinical labelNoYesYes
Text onlyThe words, without punctuation No (correct) No (wrong) Yes (correct)
Audio onlyPitch, loudness, pauses, and rate No (correct) Yes (correct) No (wrong)
Text and audioBoth No (correct) Yes (correct) Yes (correct)
Illustrative example: the sentences, cue values, and labels are constructed. The answers are not a diagnosis. No recording or model output is shown. The task follows the depression detection setting of Nurfidausi et al. (2025), who compare speech, text, and EEG.

Research areas

Political debates are spoken, so their arguments come with the voice of the speaker. Systems detect and classify argumentative sentences from the transcript, the audio, or both.

Claim detection
Given a sentence, the task is to say whether it contains a claim. A claim is the conclusion of an argument, a statement that asserts an opinion or a thesis.
Argumentative sentence detection
Given a sentence, the task is to say whether it contains an argument.
Argumentative component classification
Given an argumentative sentence, the task is to say whether it contains a claim or a premise. A premise is a reason offered for a claim.
Argumentative relation classification
Given a pair of sentences, the task is to say whether the first supports the second, attacks it, or neither.
Fallacy detection and classification
Given a sentence of a debate, the task is to say whether it contains a fallacy. A second task assigns the category of the fallacy, such as ad hominem or appeal to emotion.

Multimodal models combine an encoding of the text with an encoding of the audio. Comparing them with text-only and audio-only models shows what each modality contributes.

Audio encoding
Given an audio segment, the task is to represent it for a classifier, either with spectral features such as MFCCs or with embeddings from a pretrained network such as Wav2vec.
Modality fusion
Given an encoding of the text and an encoding of the audio of the same sentence, the task is to combine them into one input for a classifier.
Benchmarking
Given several datasets and models, the task is to compare them with the same data loading and the same evaluation, so that results are comparable.

Conditions such as depression and Parkinson's disease can change how a person speaks. Models that detect them depend on datasets whose collection must protect the people who speak.

Depression detection
Given recordings of a person, possibly with other signals such as electroencephalography (EEG), a recording of brain activity, the task is to predict whether the person has major depressive disorder.
Parkinson's disease detection
Given a speech recording, the task is to distinguish people with Parkinson's disease from healthy controls.
Dataset documentation
Given a speech dataset for mental health, the task is to check whether its paper reports informed consent, privacy, accountability, and the other items of a checklist.

Audio classifiers rarely show why they decide. Explanation methods mark the parts of the input that a decision depends on, and their quality must be measured.

Post-hoc explanation
Given a trained audio classifier and an input, the task is to produce an explanation of the decision. The explanation can be a saliency map, which scores each part of the input, or an audio signal that a person can listen to.
Faithfulness evaluation
Given an explanation, the task is to measure whether it reflects what the model actually used for its decision.

Speech models also serve safety on public transport and music retrieval. These settings bring noise, overlapping voices, and limited hardware.

Speech emotion recognition
Given a speech recording, the task is to recognize the emotion of the speaker, such as anger or calm, as one of a set of discrete classes.
Disruptive situation detection
Given a recording from a public space, the task is to detect disruptive situations, emotionally charged events such as people fighting or screaming.
Lyrics matching
Given the audio of a song, the task is to find songs with similar lyrics in a collection, working from the audio rather than from written lyrics.

Our focus

Arguments in speech

We study whether the voice of a speaker helps a machine find and classify arguments. Our data come from televised political debates.

Current

Built from the task definition in Section 3 of the paper. A sentence from a political debate comes in one of three input modes: text only, audio only, or text with its aligned audio, and systems may also use the previous sentences of the debate as context. Fallacy detection (AFD) decides whether the sentence contains a fallacy. A fallacious sentence then goes to fallacy classification (AFC), which picks one of six fallacy types.DEBATE SENTENCE, THREE INPUT MODESText onlyAudio onlyText + audioprevious sentences as contextDetection (AFD)fallacy or notif fallaciousClassification (AFC)one of six typesFallacy type

Overview of MM-ArgFallacy2025 on Multimodal Argumentative Fallacy Detection and Classification in Political Debates

Workshop on Argument Mining (ArgMining) · 2025

MM-ArgFallacy2025 benchmarks fallacy detection and classification in U.S. presidential debates from text, audio, or both, and finds that text-based systems still score highest.

Past work

  • Multimodal Fallacy Classification in Political Debates
  • Argument Mining from Speech: Detecting Claims in Political Debates
Read more

Text and audio together

We compare models that read, listen, or do both, and we release tools that make these comparisons reproducible.

Past work

  • MAMKit: A Comprehensive Multimodal Argument Mining Toolkit
  • Multimodal Argument Mining: A Case Study in Political Debates
Read more

Speech and health

We study speech as a signal of mental health and neurological disorders, and how speech datasets in this domain are collected and documented.

Current

After Figure 1 of the paper. Each subject gives three signals: resting-state EEG, the audio of 29 interview answers, and text transcribed from that audio with WhisperX. Each signal has its own pipeline: CBraMod embeddings with a CNN for EEG, HuBERT embeddings with a CNN, BiGRU, and attention encoder and a CNN classifier for speech, and MacBERT embeddings with an LSTM for text. The three predictions are combined by late fusion, with weighted averaging or Bayesian fusion, into one label: major depressive disorder or healthy control. Over 5 subject-level folds, trimodal fusion reaches a macro-F1 of 0.864, against 0.809 for speech alone; the paper reports that fusion does not yield a statistically significant improvement over the best unimodal model at this sample size (Section 4).THREE SIGNALS PER SUBJECTEEGresting stateSpeech29 answersTextvia WhisperXCBraMod+ CNNHuBERT+ BiGRU, CNNMacBERT+ LSTMLate fusionweighted, BayesianMDD or controlMacro-F1 0.864 fused vs 0.809, not significant

TRI-DEP: A Trimodal Comparative Study for Depression Detection Using Speech, Text, and EEG

Preprint · 2025

A fair, reproducible benchmark of EEG, speech, and interview text for depression detection, showing that fusing all three signals works best.

Past work

  • Promoting the Responsible Development of Speech Datasets for Mental Health and Neurological Disorders Research
Read more

Explaining audio models

We study how to explain the decisions of audio classifiers in a form that people can check, including by listening.

Current

The input audio goes to a frozen classifier, a CNN14 network, which predicts a class. The LMAC-TD interpreter reads the classifier representations and the waveform, and decodes the explanation directly as a waveform: a listenable clip on which the frozen classifier keeps its decision, while it loses it on the masked-out rest.EXPLAINING AN AUDIO CLASSIFIERInput audio x(t)Classifierfrozen CNN14classInterpretertrained; reads the classifier and the audioExplanation i(t)Masked-out rest

LMAC-TD: Producing Time Domain Explanations for Audio Classifiers

IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · 2025

LMAC-TD explains an audio classifier with a listenable clip decoded directly as a waveform, which improves perceived audio quality without losing faithfulness.

Past work

  • Investigating the effectiveness of explainability methods in Parkinson’s detection from speech
Read more

Speech in applications

We apply speech technologies to safety on public transport and to music retrieval.

Current

After Figure 1 of the paper. Stage 1, feature extraction: the song audio is cut into 30-second chunks and a frozen Whisper model turns it into decoder latents, without a transcript. Stage 2, feature adaptation: a transformer encoder trained with a contrastive loss maps the latents to one song embedding. The embedding ranks the other songs so that versions of the query come first. On LyricCovers2.0, WEALY reaches MAP 0.692, against 0.573 for the best transcription baseline.1. FEATURE EXTRACTIONSong audio30 s chunksWhisperfrozenlatents2. FEATURE ADAPTATIONTransformer encodertrained, contrastive lossSong embeddingRanked listversions firstMAP on LYC: 0.692, best transcript baseline 0.573

Leveraging Whisper Embeddings for Audio-based Lyrics Matching

IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · 2026

WEALY matches songs by their lyrics straight from the audio, using Whisper speech embeddings, and offers a reproducible baseline across three public datasets.

Past work

  • Disruptive situation detection on public transport through speech emotion recognition
Read more

Arguments in speech

Argument mining extracts arguments and their relations from natural language. In a political debate the arguments are spoken, so a model can use the transcript, the audio, or both. A claim is the conclusion of an argument, and a fallacy is a deceptive, misleading, or generally invalid argument. We ask whether audio features improve the detection of claims and the classification of fallacies.

Current

Past work

Text and audio together

A multimodal model combines several modalities, for example by joining an encoding of the text and an encoding of the audio before a classifier. Comparing text-only, audio-only, and text-audio settings shows what each modality adds. We ask how to encode the audio and how to fuse it with the text.

Past work

Speech and health

Mental health and neurological disorders, such as depression and Parkinson’s disease, can change how a person speaks. Speech datasets in this domain come from vulnerable people, and their limitations and biases affect how far the models built on them can be trusted. We ask how to document these datasets responsibly and which modalities help detection.

Current

Past work

Explaining audio models

Neural networks are typically black boxes, whose decision mechanisms are opaque. Post-hoc explanation methods explain a model after training, such as a saliency map that marks the parts of the input a model relied on. An explanation is faithful when it reflects what the model actually used. We ask how to make explanations faithful and useful for people, including domain experts.

Current

Past work

Speech in applications

Outside the lab, speech comes with noise, several voices, and limited hardware. Speech emotion recognition labels a recording with an emotion, such as anger or calm. Lyrics matching finds songs whose lyrics are similar in theme, meaning, or structure. We ask how speech models hold up in these settings.

Current

Past work