A Performance Optimization Scheme for Migrating Hive Data to Neo4j Database
Dan Liu, Ming Ming Li · 2018
As enterprise social networks are increasingly valued and applied by enterprises, the data capacity involved in social networking platforms is increasing and the relationship between data is becoming more and more complex, and traditional databases have struggled to deal with these large amounts of data and complex data. This paper presents a model of unstructured data which transforms the structured data into graph data, which can migrate the hive data to the neo4j, and make it easy for people to find the relationship between data and explore the potential value by visual data. When mass data migrated from Hive to Neo4j, there are two problems as follows: first, from the "dirty data" from hive migration, the Neo4j node GroupID is not unique; second, LOAD CSV is slow. This paper is divided into two parts to solve the above problems. When neo4j-import migration is used, the data obtained from Hive is cleaned first by "dirty data", and the problem of GroupID is not unique when the data is migrated to Neo4j. In LOAD CSV, the Cypher insertion needs to build a cable to solve the slow speed problem of LOAD CSV. The above scheme can ensure that noe4j-import tens of millions of data migration is completed at the minute level, and LOAD CSV 100 nodes are incremented at the second level.