A Framework For Effective Processing Of Jobs In Hadoop
Amarendra Kumar Mohanty, P. Ranjana · International Journal of Engineering & Technology · 2018
The main challenges in oozie based scheduling is high computing, high CPU usage and resource intensive. This leads to resource contention in production because it was not load balanced optimally.The objective of the proposed New Sql Server built Java based (NSSJ) Scheduling is to overcome some of the current challenges in the existing oozie based scheduling. It stores all the inventory information on SQL Server environment. SQL Server is preferred over Hbase, because at any given point of time, there were multiple threads hitting same inventory table to ensure transaction level processing. One can run or kill or put on hold any number of deamons or jobs at any point of time. This gives complete flexibility to the end user to load balance based on the number of jobs. It has auto restart feature when a task or job fails. It will try to attempt for one re-run, if it fails second time, it will put the job in abandoned state.Thus the proposed NSSJ scheduling load balances the resource optimally during production.