DeepBSS++: A Hybrid Framework Integrating Transfer Learning, Spatiotemporal Transformers, and Graph Neural Networks for Blind Source Separation of Electroencephalogram Signals
Sam Ansari, Abir Jaafar Hussain, Sohaib Majzoub, E'qab R. Almajali, Anwar Jarndal, Khawla A. Alnajjar, Talal Bonny, Soliman Awad Mahmoud · IEEE Access · 2026
Blind source separation (BSS) in electroencephalogram (EEG) signals remains one of the fundamental and complex challenges of computational neuroscience and biological signal processing; a challenge that stems from the multilayered interaction of nonlinear phenomena, inherent instability, extensive physiological noise, and deep spatiotemporal dependencies in brain neural networks. This work introduces a deep, multi-modal, and integrated architecture called DeepBSS++, which relies on transfer learning, spatiotemporal transformers, and graph neural networks (GNNs) to provide advanced capabilities in simultaneously modeling temporal dynamics and complex spatial structures of EEG. The DeepBSS++ architecture leverages the synergy of three main modules: temporal convolutional networks (TCNs) to decode short- and medium-term temporal patterns, Transformer encoder blocks to extract long-term dependencies and model inter-channel relationships, and GNNs to represent the spatial topology of electrodes and neurophysiological interactions. In addition, the use of knowledge transfer has enabled the model to achieve stable convergence and rich representations, even under limited data conditions. Experimental evaluations on three datasets, Synthetic EEG, brain-computer interface (BCI) Competition IV, and PhysioNet, illustrate that DeepBSS++ demonstrates consistently stronger overall separation and reconstruction performance than the evaluated classical and deep-learning baselines under the defined experimental protocol. Furthermore, DeepBSS++ records scale-invariant signal-to-noise ratio (SI-SNR) values of 4.87 dB, 4.55 dB, and 5.34 dB in the inter-dataset evaluation on three valid datasets: PhysioNet, BCI-IV, and Synthetic, respectively, and consistently outperformed Transformer, U-Net, DPRNN-TasNet, Deep-ICA, fast independent component analysis (FastICA), and principal component analysis (PCA). These results further indicate that the proposed framework exhibits stable performance and robustness to data variations and noise in the evaluated datasets and protocols. The ablation analysis further confirms that the integrated configuration of DeepBSS++ provides better separation performance than its single-component variants, highlighting the complementary contribution of the GNN, TCN, and Transformer modules. The model achieves inference times of 3.5–4.8 ms across the evaluated datasets, suggesting its potential applicability to latency-sensitive EEG applications, including BCIs and real-time neural monitoring.