Is ChatGPT a Financial Expert? Evaluating Language Models on Financial Natural Language Processing
Yue Guo, Zian Xu, Yi Ping Yang · 2023
The emergence of Large Language Models (LLMs), such as ChatGPT, has revolutionized general natural language preprocessing (NLP) tasks.However, their expertise in the financial domain lacks a comprehensive evaluation.To assess the ability of LLMs to solve financial NLP tasks, we present FinLMEval, a framework for Financial Language Model Evaluation, comprising nine datasets designed to evaluate the performance of language models.This study compares the performance of encoder-only language models and the decoderonly language models.Our findings reveal that while some decoder-only LLMs demonstrate notable performance across most financial tasks via zero-shot prompting, they generally lag behind the fine-tuned expert models, especially when dealing with proprietary datasets.We hope this study provides foundation evaluations for continuing efforts to build more advanced LLMs in the financial domain.