Harnessing AI for Business Insight
Alex P. Miller, Kartik Hosanagar, Ramayya Krishnan · Oxford University Press eBooks · 2025
Abstract Large language models (LLMs) have demonstrated significant potential in handling unstructured, natural language data. However, their adaptation to complex business settings remains a challenging endeavor. We explore the challenges and research opportunities associated with deploying LLMs for document and knowledge summarization in various business applications. We evaluate current paradigms for evaluation and highlight their inability to fully capture the multi-dimensional considerations in business settings, including relevance, provenance, and factuality of the output. The chapter emphasizes the need for a paradigm shift in evaluation approaches to better align with the nuanced needs of business applications. Key considerations include expanding the dimensionality of automated evaluation metrics, incorporating human-computer interaction factors, and addressing domain-specific needs. A case study on summarizing user-generated content from a product announcement video on social media is presented to illustrate these challenges and associated research opportunities.