DeBERTa-Based SMILES Encoders for ADMET-Aware Drug Design
Jong Hyeon Lim, Myounwoo Kim, Youngmahn Han, Jin Yong Lee · ChemRxiv · 2025
Multi-modal drug discovery frameworks increasingly rely on robust encoders capable of representing chemical structures alongside other data modalities. In this study, we fine-tuned a DeBERTa-based SMILES encoder to improve its predictive capacity for 22 ADMET (Absorption, Distribution, Metabolism, Excretion, and Toxicity) endpoints while maintaining strong comprehension of molecular syntax. Starting from a pretrained ZINC-based DeBERTa checkpoint, we trained on a 300K PubChem-ADMET dataset using a multi-label regression scheme with a focal MAE loss to stabilize learning across diverse properties. Our encoder achieved top-10 rankings on 16 TDC benchmark tasks, including notable improvements of 14–30% on critical endpoints such as bioavailability and CYP2C9-substrate. Compared with BERT- and RoBERTa-based molecular encoders, our approach preserved significantly higher MLM accuracy (>89%) over training, indicating robust retention of chemical language understanding. Additional analysis using an ADMET path length metric revealed that DeBERTa produced more disentangled latent representations, underscoring its suitability for property-specific molecular manipulation. These results demonstrate that a disentangled, ADMET-aware DeBERTa encoder can serve as a powerful component for future multi-modal pipelines in AI-driven drug design, effectively balancing structural fluency with predictive specialization.