Reinforcement learning and GAN-augmented autoencoders for H&E-Stained bone biopsy image analysis
B. S. Vandana, Usha Desai, Sathyavathi R. Alva · Intelligence-Based Medicine · 2026
The histopathological image approach plays a predominant role in computer-assisted medical diagnosis; however, variations in staining conditions, non-availability of sufficient labeled datasets, and high-dimensional image features remain major challenges for effective classification. The existing approaches often focus on individual techniques for preprocessing, augmentation, or feature extraction however which lacks an effective integrated approach. To overcome this research gap, this study proposes a unified framework that integrates Reinforcement Learning (RL), Generative Adversarial Networks (GANs), and Autoencoder-based feature extraction for improved efficient histopathological image analysis. Experiments were carried using a labeled dataset containing 3543 hematoxylin and eosin (H&E)-stained bone biopsy images of normal, benign, and malignant classes from The Cancer Imaging Archive (TCIA) open-source dataset. An RL-based preprocessing technique applied that enhances a complete image in a dynamic manner based on quality of an image. The image quality metric was evaluated, achieving an average Structural Similarity Index Matric (SSIM) of 0.9255 and an average Peak Signal-to-Noise Ratio (PSNR) of 37.71 dB. In addition, to overcome the limitations of small datasets, GAN-based augmentation applied, which generates synthetic images corresponding to 20% of original dataset. The generated images were tested using Fréchet Inception Distance (FID) and Inception Score (IS). The developed autoencoder architecture extracts the compact latent feature representation that captures important morphological patterns in histopathological images, using classifiers including Support Vector Machine (SVM), Random Forest (RF), and Deep Neural Network (DNN). Among these models, SVM achieved the highest performance with 92.5% accuracy and an F1-macro score of 0.8876, while RF and DNN attained accuracies of 88.06% and 87.59%, respectively. Result validates, the proposed RL–GAN–Autoencoder framework provides image quality, improves the dataset diversity, and enables effective feature learning for histopathological image classification. This approach has the potential for the sustainable computer-assisted diagnosis by providing a trust-based supportable framework of intelligent analysis of medical images.