Real-time big data warehousing and analysis framework
Abbas Raza Ali · 2018
Big Data technology is gradually becoming a dire need of large enterprises. These enterprises are producing large amount of offline and streaming data in both structured and unstructured forms on daily basis. Moreover, it is a challenging task to effectively extract useful insights from this type of complex data. On the other hand, the traditional systems are unable to efficiently manage transactional data history of more than a few months. This paper presents a framework to efficiently manage and effectively analyze massively large and complex datasets. The framework can be very effective for the communication industry which is bound by the regulators to manage significant history of their subscribers' call records, where every single action of a subscriber generates a packet containing more than 500 attributes. The analyses of transactional data allows the service providers to better understand their customers' behavior, for example, deep packet inspection requires transactional internet usage data to explain the data usage behaviour of the subscribers. On the contrary, relational database systems limit the service providers to only maintain semantic level call history which is aggregated at subscriber-level. The framework addresses the mentioned challenges by leveraging Big Data technology which optimally manages and allows deep analysis of large and complex datasets. The framework has been applied on a communication service provider producing massive amount of streaming data in binary format. It has been used to offload the existing Intelligent Network Mediation and Relational DataWarehouse of the service provider on Big Data technology. The service provider has 50+ million subscriber-base with yearly growth of 7-10%. The end-to-end CDR processing takes not more than 10 minutes which involves binary to ASCII decoding of call detail records, stitching of various interrogations against a call (transformations) and aggregations of transformed call records of a subscriber.