Partial Update: Efficient Materialized View Maintenance in a Distributed Graph Database
SungJu Cho, Roman Averbukh, Yanwei Zhang, Andrew Carter, Jane Alam Jan · 2018
In order to efficiently serve a high volume of queries, LinkedIn's distributed graph service caches computationally-expensive materialized views. Despite this, even basic view maintenance has been a scalability bottleneck. In this paper, we introduce Partial Update, which efficiently applies incremental view maintenance in the absence of deletions in a materialized view. Without expensive transaction handling, Partial Update guarantees eventual consistency across source of truth databases, materialized views and base relations stores. By design, its delta change computation logic only produces lightweight queries, reducing any need to augment the underlying data storage systems. This allows Partial Update to seamlessly integrate on top of traditional eventually consistent systems. In LinkedIn's production systems, Partial Update achieves up to a 50% reduction in both maintenance time and network bandwidth utilization. The improved view maintenance scalability allows us to improve the quality of cached data by a factor of 20. We consider that Partial Update is general enough to be applicable to the majority of select-project-join view maintenance strategies in eventually consistent distributed data stores.