A Preparing Approach to Manipulating Nested Data Structures
Jeffrey Myers, Yaser Mowafi · 2024
Processing nested data collections in large-scale distributed systems exhibits considerable challenges in query processing. Manipulating such data demands an extravagant number of operations, leading to extensive data duplication and imposing challenges in ensuring balanced distribution across partitions. This research proposes preparing flattening procedures for nested data structures. The work aims to alleviate the adverse implications of data duplication and information loss while addressing the irregularity of nesting structures. The efficacy of the proposed approach is assessed on question-answering datasets, comparing its performance against the Pandas Python package flattening implementation.