Enhancing Data Governance and Security in Spark Scala Applications on Google Cloud Platform
Senior Data Engineer, Walmart, USA, Mahendra Babu Iragala · International Journal of Innovative Research in Science Engineering and Technology · 2025
Data governance and security are critical concerns for organizations leveraging cloud platforms like Google Cloud Platform (GCP) to process large datasets using Spark Scala applications. This paper explores key aspects of enhancing data governance and security in this context. We discuss challenges, best practices, and techniques for implementing secure and compliant big data solutions on GCP. Topics covered include data encryption, access control, auditing, and regulatory compliance. The proposed methodology leverages GCP features like Customer-Managed Encryption Keys (CMEK), Identity and Access Management (IAM), Cloud Audit Logs, and compliance tools. Experimental results demonstrate the effectiveness of the approach in securing a sample Spark Scala application processing sensitive financial data. The techniques presented equip practitioners with actionable insights to build secure and governed big data applications on GCP.