거대 언어 모델 기반 선호 데이터 자동 구축을 통한 페어와이즈 공감 대화 생성

초록

Empathetic dialogue generation, which aims to produce responses that appropriately reflect users’ emotions and situations, has become an increasingly important research topic with the growing demand for emotionally interactive conversational agents. Nevertheless, existing supervised approaches mainly rely on single-reference imitation, which often results in safe yet repetitive responses and limits the model’s ability to learn relative differences in empathy across candidate responses. Furthermore, the construction of high-quality empathetic preference data is both expensive and labor-intensive. To overcome these challenges, this study proposes a pairwise-learning-based framework for empathetic dialogue generation that exploits preference signals obtained from large language models (LLMs). The proposed framework consists of three stages. First, multiple candidate responses are generated for each dialogue context from the EmpatheticDialogues dataset using an LLM, and the candidate pool is refined through an iterative branching strategy incorporating Maximal Marginal Relevance (MMR), Jaccard-similarity-based scoring, and candidate replenishment. Second, an evaluator LLM scores the candidates according to empathy, fluency, appropriateness, and reliability, from which Best/Worst preference pairs are constructed. Third, a T5-base generator is optimized through weighted supervised fine-tuning and subsequent Simple Preference Optimization (SimPO), and final responses are selected via reranking and multi-prompt-based inference. Experimental results show that the proposed method outperforms existing sentiment-based, knowledge-based, and reinforcement-learning-based empathetic dialogue generation methods in fluency (PPL), diversity (Distinct-1/2), and empathy alignment (Emp-F1). These results indicate that automatically constructed LLM-based preference data provide an effective and data-efficient mechanism for improving empathy alignment in dialogue generation models.

키워드