Performance enhancement of apache APEX
Sushama Annaso Shirke, Tushar Gosavi, Anurag Kumar Mishra, Vishal Vir Singh, Akshay Srivastava · 2017 International conference of Electronics, Communication and Aerospace Technology (ICECA) · 2017
In a processing engine there is a continuous exchange of data between the computing nodes. The data that needs to be exchanged cannot be transferred in its normal form hence needs to be converted into normal bytes for faster execution. This is achieved by the process of Serialization. If there is any computation required to be performed on the data that is being transmitted we will have to de-serialize the data, perform the computation, reserialize it and transmit it again towards destination. This causes a lot of unnecessary overhead that can be avoided. This can be avoided by making use of the UnsafeRow Format developed by DataBricks. We can use this format to perform the required computation on the serialized data without the need to deserialize it. Thereby, increasing the overall performance of the apex framework.