BioNART: A Biomedical Non-AutoRegressive Transformer for Natural Language Generation
Masaki Asada, Makoto Miwa · 2023
We propose a novel Biomedical domainspecific Non-AutoRegressive Transformer model for natural language generation: BioNART.Our BioNART is based on an encoder-decoder model, and both encoder and decoder are compatible with widely used BERT architecture, which allows benefiting from publicly available pre-trained biomedical language model checkpoints.We performed additional pre-training and finetuned BioNART on biomedical summarization and doctor-patient dialogue tasks in English.Experimental results show that our BioNART achieves about 94% of the ROUGE score to the pre-trained autoregressive model while realizing an 18 times faster inference speed on the iCliniq dataset.