Investigating the Impact of Linguistic Errors of Prompts on LLM Accuracy
Praneeth Vadlapati · ESP Journal of Engineering & Technology Advancements · 2023
Large Language Models (LLMs), such as GPT-3.5, have demonstrated exceptional abilities in processing text by understanding and generating text. Language models are trained on diverse data, which is typically free of linguistic errors such as spelling and grammatical mistakes. However, in real-world scenarios, the queries from the users commonly include linguistic errors. This research systematically examines the robustness of LLM in handling uncommon linguistic errors. The study utilizes original error-free text, text with errors in spelling, and text with errors in grammar. The study analyzes accuracy across multiple types of tasks such as quantitative reasoning, text manipulation, and linguistic tasks to test across various scenarios and evaluate whether the resilience of the model to linguistic errors varies across multiple types of tasks. In addition, the study explores the vulnerabilities of generating harmful text using jailbreaking through adversarial prompts that include grammatical errors. The results underscore the necessity of handling linguistic errors and implementing advanced mechanisms to mitigate threats from adversarial inputs. This study contributes to the research on investigating the reliability and robustness of AI systems in real-world applications. The source code is available at github.com/Pro-GenAI/PromptSpell.