ExplainCPE: A Free-text Explanation Benchmark of Chinese Pharmacist Examination
Dongfang Li, Jindi Yu, Baotian Hu, Zhenran Xu, Min Zhang · 2023
In the field of Large Language Models (LLMs), researchers are increasingly exploring their effectiveness across a wide range of tasks.However, a critical area that requires further investigation is the interpretability of these models, particularly the ability to generate rational explanations for their decisions.Most existing explanation datasets are limited to the English language and the general domain, which leads to a scarcity of linguistic diversity and a lack of resources in specialized domains, such as medical.To mitigate this, we propose ExplainCPE, a challenging medical dataset consisting of over 7K problems from Chinese Pharmacist Examination, specifically tailored to assess the model-generated explanations.From the overall results, only GPT-4 passes the pharmacist examination with a 75.7% accuracy, while other models like ChatGPT fail.Further detailed analysis of LLM-generated explanations reveals the limitations of LLMs in understanding medical text and executing computational reasoning.With the increasing importance of AI safety and trustworthiness, ExplainCPE takes a step towards improving and evaluating the interpretability of LLMs in the medical domain.The dataset is available at https: //github.com/HITsz-TMG/ExplainCPE. QuestionThe patient, female, 65 years old, received the second cycle of chemotherapy for breast cancer.Which antiemetic drugs are prohibited?Options A.Metoclopramide.✓ B.Ondansetron.C.Dexamethasone.D.Diphenhydramine.E.Aprepitant.Gold Explanation This question examines antiemetic drugs.Optional antiemetic drugs are mainly dopamine receptor blockers (such as metoclopramide), 5-HT3 receptor blockers (such as ondansetron, tropisetron, and palonosetron, etc.) , corticosteroids (such as dexamethasone), anticholinergics and antihistamines (such as diphenhydramine), and NK-1 receptor blockers (such as aprepitant), etc. Special emphasis should be placed on the prohibition of metoclopramide as an antiemetic in patients with breast cancer.Therefore, choose A for this question.GPT-4 Response Answer: A. Metoclopramide.Analysis: This question examines the contraindications of antiemetic drugs.Metoclopramide is a dopamine D2 receptor antagonist, and its anti-emetic effect is achieved by blocking central and peripheral 5-HT3 receptors and dopamine D2 receptors.Elderly patients over 65 years of age are contraindicated for metoclopramide, because the drug may increase central nervous system adverse reactions in the elderly population, such as drowsiness, ataxia, and akathisia.Other antiemetics, such as ondansetron, dexamethasone, diphenhydramine, and aprepitant, are safe in older patients.Therefore, the correct answer is A. ChatGPT ResponseAnswer: B. Ondansetron.Analysis: Ondansetron is a 5-HT3 receptor antagonist, mainly used to prevent nausea and vomiting caused by chemotherapy for malignant tumors.However, ondansetron is contraindicated in patients with hepatic insufficiency or pregnancy, because these patients have reduced ability to metabolize and clear drugs, which may easily cause drug accumulation and cause side effects.For the specific situation of the patient, it should be selected according to the clinical judgment of the doctor.But in this case, ondansetron is contraindicated, so the correct answer is B.