Paralinguistic acoustic feature to natural language prompt conversion using hybrid transformer-LLaMA architecture

초록

Speaking-style captioning aims to generate natural language descriptions of paralinguistic characteristics in speech, such as pitch, speaking speed, and vocal energy. While recent approaches leverage large pre-trained speech encoders and language models, the interaction between heterogeneous speech representations and their impact on style caption generation remain insufficiently explored. In this work, we conduct an empirical study analyzing the effects of combining self-supervised and supervised speech representations within an end-to-end captioning framework. We integrate a HuBERT-based waveform encoder and a Whisper-based spectrogram encoder, whose complementary representations are fused via cross-attention and aggregated through a learnable post encoder before being decoded by a QLoRA-adapted LLaMA model. Our results on PromptTTS, DreamVoice, and TextrolSpeech demonstrate that representation types contribute unevenly across speaking-style attributes. The dual-encoder configuration achieves the strongest overall performance in most settings, and the post encoder improves the overall trend across attribute and captioning metrics, although a small CIDEr fluctuation is observed on DreamVoice. Compared with both SECap and zero-shot Qwen2-Audio baselines, our method achieves stronger caption quality and more reliable attribute-level interpretability. These results demonstrate that representation complementarity and post-aggregation play a critical role in producing semantically grounded and stylistically expressive speaking-style captions.

키워드