On the use of Prompting for Fine-Tuning Neural models for Speech Processing