PrivacyPreserveNet: A Multilevel Privacy-Preserving Framework for Multimodal LLMs via Gradient Clipping and Attention Noise
Yunfei Guo, Yiming Yu · 2025
The deployment of multimodal large language models introduces heightened risks of privacy leakage, especially when training involves sensitive text, image, and audio data. Existing solutions typically apply differential privacy or gradient clipping individually, but these lack cohesion and often compromise model utility. This paper proposes PrivacyPreserveNet, a novel framework built on Llama-7B that integrates Differential Privacy-enhanced Pretraining, Privacy-Aware Gradient Clipping, and a Noise-Injected Attention module to enforce privacy at multiple levels of the learning process. PrivacyPreserveNet introduces noise into both model gradients and attention distributions, ensuring comprehensive protection against data leakage without sacrificing performance. The framework also incorporates composite regularization and visualization-based robustness assessments to enhance model stability. Experimental validation confirms that PrivacyPreserveNet achieves a superior balance between privacy guarantees and task performance, establishing a practical path forward for secure multimodal model training.