Implementing Quality of Service and Confidentiality for Batch Processing Applications
Igor Ataide, Gabriel Vinha, Clenimar Souza, Andrey Brito · 2018
With a massive volume of information generated from many kinds of sources and for many different purposes comes the need for processing systems that help digest these data. One approach for efficient processing of data is to batch process larger sets of data. For example, periodically, recent data can be analyzed and filtered, having as a result actions, updated models, or cleaned versions of the data uploaded to databases. Increasingly often, such applications have strong requirements regarding data security, quality of service, and scalability. While scalability is typically addressed by leveraging the abundant resources and fair pricing of cloud computing techniques, using a shared environment triggers resistance from developers regarding data security and quality of service. In this work, we present a system that combines a novel technology for data protection, Intel SGX, with a popular, open source tool for managing interactive and data processing applications, Kubernetes, to provide a batch processing system that is easy to use while still providing quality of service and strong confidentiality guarantees. Our evaluation shows that independent tasks that consume data from scalable storages can be implemented in a controllable, secure fashion, without added complexity or considerable performance or resource overheads.