Fine-Tuned BERT with Attention-Based Bi-GRU-CapsNet Framework for Joint Intent Recognition and Slot Filing
Nahida Shafi, Manzoor Ahmad Chachoo · 2023
In the field of natural language processing (NLP), the two most prominent research areas are slot tagging and intent recognition. Modern joint learning strategies examine the link between slot-tag identification and intent classification, utilising the shared knowledge between the two tasks for their collective advantage.However, methods that combine various variants of pre-trained Bidirectional Encoder Representations from Transformers [BERT] models with attention based capsule networks for joint slot-tag identification and intent detection have not been fully explored.This study proposes a multi-stage framework trained on different versions of BERT models $(\mathrm{B}\mathrm{E}\mathrm{R}\mathrm{T}_{\mathrm{b}\mathrm{a}\mathrm{s}\mathrm{e}}$ and $\mathrm{B}\mathrm{E}\mathrm{R}\mathrm{T}_{\mathrm{l}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}})$ with Bidirectional Gated Recurrent Unit [Bi-GRU] and self attention mechanism as intent detection decoder to capture the underlying information and to discover the explicit links.While the capsule network, in accordance with the dynamic routing algorithm, acts as a slot filler decoder in predicting intents and slots and representing the semantic and syntactic relationships. The experimental findings demonstrate that the proposed approach enhances semantic frame accuracy at the sentence level, outperforming various baseline methodologies by a significant margin with a 1.2% improvement in the intent Flscore and 3.24% in the slot Fl-score, relative to the previous state-of-the-art models on the SNIPS datasets.