Knowledge Distillation

TWOLAR: A TWO-Step LLM-Augmented Distillation Method for Passage Reranking
TWOLAR distills the passage ranking skill of ChatGPT into a Flan-T5 reranker, which matches far larger LLM rerankers on standard search benchmarks.
Davide Baldelli, Junfeng Jiang, Akiko Aizawa, Paolo Torroni
Cite DOI
Fast Vocabulary Transfer for Language Model Compression
Fast Vocabulary Transfer moves a language model to an in-domain tokenizer, so the model runs faster and can shrink, with little loss in F1.
Leonidas Gee, Andrea Zugarini, Leonardo Rigutini, Paolo Torroni
Cite DOI URL