Handwritten Document Interaction using LlaMA2: Leveraging Scale Space Techniques for Word Segmentation and Deep Learning for OCR
Aakarsh Mishra, Nancy Saxena, Utkarsh Mishra, Ansh Pandey, Shashank Kumar Singh, K. Vijayaprabakaran · 2025
This paper introduces an integrated framework that enables Large Language Models to interact with handwritten documents by converting them into machine readable text. The methodology combines the "Scale Space Technique for Word Segmentation" with advanced document processing and large language model (LLM) integration to allow real-time interaction with handwritten content. Handwritten documents, often challenging for Optical Character Recognition (OCR) systems, are first processed through a robust word segmentation algorithm that extracts and transforms handwritten text into a typed format using CNNs and RNNs with an effective accuracy of 91%. This document is then uploaded into an NLP system using a combination of LlaMA2, Hugging Face embeddings, and FAISS similarity search, enabling efficient interaction and query-based responses. This system facilitates seamless interaction with historical, degraded, or complex handwritten documents, making them accessible through modern NLP models for a variety of applications including research, education, and data analysis. The paper showcases this solution's effectiveness in translating handwritten notes into actionable digital insights, allowing users to query and interact with handwritten text as easily as with typed documents.