Order-Aware, Parameter-Efficient Abstract Discourse Tagging on PubMed RCT-20k
Suryanshu Anand · 2025
Sentence-level discourse classification of PubMed abstracts was investigated, with each sentence labeled as Background, Objective, Methods, Results, or Conclusions. PubMed-BERT was employed as the sentence encoder, and a comparison was conducted between a standard sentence-only baseline and a lightweight order-aware model. In the latter, a small BiLSTM head was added over abstract-level sentence embeddings while keeping the encoder frozen, providing a parameter-efficient setup. Due to Apple-Silicon compute constraints, training was performed on reduced data, and evaluation was carried out on the full PubMed 20k RCT test split (29,578 sentences). The sentence only PubMedBERT achieved a macro-F1 score of 76.8 and an accuracy of 84.6%, requiring approximately 109.5M trainable parameters. In contrast, the order-aware BiLSTM with the frozen encoder utilised only 2.10M parameters but attained lower performance, with 67.6 macro-F1 and 81.4% accuracy. Per-class analysis revealed that the Objective label was the most challenging (51.5 F1), whereas Methods and Results achieved F1 scores above 90. Although sequence modelling was incorporated, the frozen order-aware head underperformed the fully fine-tuned baseline in this compute-limited regime. These findings suggest that while sequence modeling provides value, greater benefits may be realized with additional training or modest encoder adaptation (e.g., LoRA), which is recommended for future work.