Leveraging Whisper Embeddings for Audio-based Lyrics Matching

Fri, 01 May 2026 00:00:00 +0000

Research setting

Audio-based lyrics matching finds songs whose lyrics are similar, using only the song audio. It can support copyright protection, music discovery, and songwriters who want to avoid unintentional plagiarism. Datasets of matched lyrics are scarce, so the paper evaluates on musical version identification (MVI). In MVI, a system receives a song and must find its other versions, such as covers, in a large collection. Versions of one song usually share their lyrics, so they give similarity labels without curated lyric texts.

Query song Raw audio only, no lyrics text.
Song embedding One vector that describes the lyrics.
Compare with collection Similarity with every candidate song.
Ranked list Versions of the query should come first.

The retrieval task used for evaluation: a query song goes in, and a ranked list of candidate songs comes out.

Terms used on this page:

Whisper is a large speech recognition model that turns speech into text. Its decoder writes the text one word piece at a time.
Whisper embeddings (or latents) are the internal vectors of the decoder just before it picks each word piece. They describe what is being sung without writing it down.
Contrastive learning trains a model to place versions of the same song close together and different songs far apart.
Mean average precision (MAP) scores a ranked list by how high the true versions of each query appear, averaged over all queries. It ranges from 0 to 1, and higher is better.

Motivation

Lyrics text is often unavailable, incomplete, or restricted by copyright, and transcription of singing makes errors. Early lyrics-based systems depend on transcripts or external lyrics databases. Recent systems use Whisper inside complex pipelines that combine source separation, transcriptions, and several embeddings. These pipelines offer limited transparency and reproducibility. Prior work is also evaluated on limited datasets, so the field has no robust benchmark.

Approach	Works without transcripts	Transparent and reproducible	Several datasets
Text-based (Knees et al., 2005; Patra et al., 2017)	no	not discussed	not discussed
Transcript or lyrics database (Correya et al., 2018; Vaglio et al., 2021)	no	not discussed	not discussed
Whisper-based (Balluff et al., 2024; Du et al., 2024)	no	no	not discussed
This paper (WEALY) This work	yes	yes	yes

How the paper positions itself against the related work it discusses (Introduction and Section 2 of the paper).

Objective. Build a fully reproducible, end-to-end pipeline that matches songs by their lyrics directly from audio, and set transparent baselines for it on several public datasets.

Approach

WEALY (Whisper Embeddings for Audio-based LYrics matching) works in two stages. First, a frozen Whisper model listens to the song in 30-second chunks and keeps its decoder embeddings, without producing a transcript. Second, a transformer encoder reads a random window of 1500 of these embeddings, pools them over time, and projects them to one 512-dimensional song embedding. Only the encoder is trained, with a contrastive loss on pairs of versions of the same song. At test time, each song is cut into overlapping windows, and two songs are as similar as their best-matching pair of windows.

1. Feature extraction

Song audio Data
Whisper Frozen
Latents Data
Encoder
Output

The song is resampled to 16 kHz mono and cut to at most 5 minutes. Frozen Whisper decodes it chunk by chunk, and the decoder states are kept as lyrics-aware latents.

2. Feature adaptation

Song audio Data
Whisper Frozen
Latents Data
Encoder Trained
Output Data

A transformer encoder with 4 blocks turns a window of latents into the output, one song embedding. It learns with a contrastive loss that pulls versions of the same song together.

Trained Frozen Data

The two stages of WEALY: what is frozen and what is trained (Section 3 and Figure 1 of the paper).

The paper also tests a multimodal extension. It adds the WEALY distance between two songs to the distance from CLEWS, an existing audio-content model for version identification, so lyrics and sound are combined at the distance level.

Results

The authors measure MAP for version identification on three public datasets: DiscogsVI-YT (DVI), SHS100k-v2 (SHS), and LyricCovers2.0 (LYC). LYC covers 80 languages. They compare WEALY with baselines that transcribe the lyrics with Whisper and then compare the texts, and with raw Whisper embeddings averaged over time without training.

0.692MAP of WEALY on LYC
0.573MAP of the best transcription baseline on LYC
0.912MAP on SHS when WEALY is fused with CLEWS

Method	DVI	SHS	LYC
Random	0.001	0.003	0.002
TF-IDF on transcripts, cosine	0.272	0.503	0.537
TF-IDF on transcripts, Lucene	0.242	0.457	0.486
Sentence-BERT on transcripts, cosine	0.294	0.508	0.573
Sentence-BERT on transcripts, trained transformer	N/A	0.480	0.516
Averaged Whisper embeddings, no training	0.166	0.297	0.322
WEALY (this work)	0.328	0.640	0.692
Oracle upper bound (versions with valid lyrics)	0.967	0.956	0.954

MAP on the test sets (Table 1 of the paper). The oracle gives a perfect match to every version with usable lyrics, so it bounds what a lyrics-only method can reach.

WEALY beats every transcription-based baseline on all three datasets. Averaged Whisper embeddings without training score lowest, so the trained encoder is needed. In the ablation on SHS, the contrastive loss matters most: a triplet loss drops MAP from 0.640 to 0.548. Forcing Whisper to decode every song as English drops MAP to 0.578, so the multilingual cues in the embeddings help retrieval.

MAP on SHS (higher is better)

ByteCover1/2 (audio content) 0.813 ± 0.006
ByteCover3.5 (audio content) 0.857
CLEWS (audio content) 0.876 ± 0.005
WEALY + CLEWS (lyrics and audio) This work 0.912 ± 0.004

Version identification on SHS, MAP on the test set; whiskers show the reported ± value. Source: Table 3 of the paper.

Show as table

System	MAP on SHS
ByteCover1/2 (audio content)	0.813 ± 0.006
ByteCover3.5 (audio content)	0.857
CLEWS (audio content)	0.876 ± 0.005
WEALY + CLEWS (lyrics and audio) (this work)	0.912 ± 0.004

Alone, WEALY stays below the strongest audio-content models for version identification. Added to CLEWS with a simple sum of distances, it gives the best MAP, so lyrics and sound carry complementary information.

Takeaways