Wealth of Nations, Wealth of Data: How GDP Shapes Diverse Large Language Models like ChatGPT : Interviewing Assorted Open Source Generative AI Models
Alex Kaplunovich · 2023
Generative large language models (such as ChatGPT) are increasingly influencing various aspects of our lives, partly due to their training on vast datasets that encompassing big data paradigms and range of topics. "Intervista," an award-winning Italian film by Federico Fellini, focuses on his interview with a Japanese TV crew. Inspired by this, we conducted interviews with a diverse set of open-source and OpenAI models to explore various political, economic, and cultural aspects of life, evaluating LLM performance. We also examined whether a correlation exists between a country’s GDP per capita and the quality of the model’s answers. To this end, we utilized a Huggingface model leaderboard to select appropriate models and deployed them in an AWS SageMaker GPU environment. The identical questions were posed about nearly 200 countries, and the responses were analyzed to verify their accuracy and correlation with Gross Domestic Product (GDP). We were amazed by the diversity, quantity, and quality of existing pretrained open-source LLMs. Our journey provided insights into model selection, inference pipeline automation, GPU configuration, generated texts benchmarking, and systematic evaluation of model quality. Overall, leading LLMs performed well, providing reasonable responses for many countries. However, we discovered that the depth and detail of the answers were influenced by a country’s GDP per capita, with higher-income nations receiving more accurate responses.