Development of an LLM-based CPX Practicing Chatbot for Korean Medicine Education: Implementation of Automated Scoring and Feedback Generation Framework

초록

Objectives: This study aimed to develop an AI-based CPX (Clinical Performance Examination) practicing chatbot for Korean medicine education, implementing automated quantitative scoring and qualitative feedback systems to enhance individualized learning experiences. Methods: Building upon a previously developed CPX practicing chatbot, we integrated a quantitative scoring system and a qualitative text feedback system using Large Language Models (LLMs). Scoring prompts and feedback prompts were designed based on standardized CPX scenarios. We compared the performance of OpenAI's GPT-4 and Anthropic's Claude-3.5-Sonnet models in terms of scoring accuracy, feedback quality, consistency, latency, and fluency. Three sample chat histories representing varying levels of student performance were created for evaluation. Results: Claude-3.5-Sonnet demonstrated perfect accuracy in quantitative scoring across all samples and faster execution times (average 43.2 seconds) compared to GPT-4. In qualitative feedback generation, GPT-4 provided more specific and actionable feedback, while Claude-3.5-Sonnet produced more natural and supportive language. By combining the strengths of both models, we implemented an optimized system that first uses GPT-4 for detailed feedback generation and then refines the language with Claude-3.5-Sonnet, resulting in accurate scoring and high-quality feedback. Conclusions: The study successfully developed an LLM-based CPX practicing chatbot that offers automated scoring and individualized feedback for Korean medicine education. The optimized system enhances the learning experience by providing accurate assessments and specific, supportive feedback, addressing limitations in current CPX education practices and resource constraints.

키워드