Towards Controllability, Efficiency, and Trustworthiness of Text Generation Systems
Shuyang Cao · Deep Blue (University of Michigan) · 2025
As reliance on automated text generation continues to grow, understanding and overcoming the limitations of current systems are critical. This thesis addresses the challenges of controllability, efficiency, and trustworthiness in text generation systems, which can enhance their practical applicability across diverse domains. The first part of this work focuses on controllability, where text outputs must meet specific user needs or constraints. We introduce a type-controlled question generation system that employs exemplar templates and optionally generated fine-grained templates to achieve improved control over target attributes. The template-based control signals significantly improve the system’s ability to produce questions of required types, compared to systems that directly apply constraints with type labels. Second, we investigate efficient training methods in the context of long-document summarization, where efficiency issues are most pronounced. For incorporating document structure information that commonly exists in long documents, we design learnable biases representing section-level relations. These biases add minimal overhead to model training, while improving models’ structure understanding, leading to summaries of high quality. We further consider a divide-and-conquer framework to enable model training on long sequences under strict resource constraints. Summarizing divided chunks separately, the GPU memory requirement is significantly reduced. Additionally, our framework is equipped with memory mechanisms and global salient content to establish connections between chunks, achieving performance comparable to resource-demanding systems. To evaluate models with increasing context length enabled by improved efficiency, we develop an adaptive evaluation benchmark that can flexibly adjust input context length and task difficulty. Our evaluation benchmark curates diverse synthetic tasks targeting varying levels of model capabilities. By requiring complex model capabilities, our benchmark presents sufficient challenges to current and potential future models. Notably, these synthetic tasks share the same contexts within each domain, thus mitigating the confounding effects of input context variations and allowing for more controlled comparisons of model capabilities to understand model behaviors. Lastly, we address the issue of factual inaccuracies in generated summaries. Our proposed contrastive learning framework trains models to differentiate correct outputs from erroneous ones that are automatically collected with various carefully designed strategies. Models trained with our framework consistently deliver more trustworthy outputs. Besides trustworthy models, we also explore verifiable generation with fine-grained citations that enhances user confidence in the reliability of generated content. The generated citations to source documents are attached to specific spans of the outputs, allowing for quick and precise verification of generated content. We collect a new dataset to felicitate the study of this generation paradigm. By ensuring outputs are not only contextually relevant but also factually accurate and verifiable, real-world applications of automatic text generation systems can be expanded with excellent user trust.