BERNN: A Transformer-BiLSTM Hybrid Model for Cross-Domain Short Text Classification in Agricultural Expert Systems
Xueyong Li, Menghao Zhang, Xiaojuan Guo, Jiaxin Zhang, Jiaxia Sun, Xianqin Yun, Liyuan Zheng, Wenyue Zhao, Lican Li, Haohao Zhang · Symmetry · 2025
With the advancement of artificial intelligence, Agricultural Expert Systems (AESs) show great potential in enhancing agricultural management efficiency and resource utilization. Accurate extraction of semantic features from agricultural short texts is fundamental to enabling key functions such as intelligent question answering, semantic retrieval, and decision support. However, existing single-structure deep neural networks struggle to capture the hierarchical linguistic patterns and contextual dependencies inherent in domain-specific texts. To address this limitation, we propose a hybrid deep learning model—Bidirectional Encoder Recurrent Neural Network (BERNN)—which combines a domain-specific pre-trained Transformer encoder (AgQsBERT) with a Bidirectional Long Short-Term Memory (BiLSTM) network. AgQsBERT generates contextualized word embeddings by leveraging domain-specific pretraining, effectively capturing the semantics of agricultural terminology. These embeddings are then passed to the BiLSTM, which models sequential dependencies in both directions, enhancing the model’s understanding of contextual flow and word disambiguation. Importantly, the bidirectional nature of the BiLSTM introduces a form of architectural symmetry, allowing the model to process input in both forward and backward directions. This symmetric design enables balanced context modeling, which improves the understanding of fragmented and ambiguous phrases frequently encountered in agricultural texts. The synergy between semantic abstraction from AgQsBERT and symmetric contextual modeling from BiLSTM significantly enhances the expressiveness and generalizability of the model. Evaluated on a self-constructed agricultural question dataset with 110,647 annotated samples, BERNN achieved a classification accuracy of 97.19%, surpassing the baseline by 3.2%. Cross-domain validation on the Tsinghua News dataset further demonstrates its robust generalization capability. This architecture provides a powerful foundation for intelligent agricultural question-answering systems, semantic retrieval, and decision support within smart agriculture applications.