Centralized log management for pepper
V. Deepak Kumar, Ruchir Shah, Aby Z. Philip · 2011
Pepper provides a scalable and flexible platform for processing high volume small feeds. Pepper hosts workflows as web application on data nodes in Hadoop. As a result, it provides application level scalability, self-healing, capacity planning and isolation among other benefits. The logs generated by these workflows on request execution are located in individual Hadoop slave boxes and hence aggregating and making them viewable in a centralized way is a big challenge. In this poster, we present log aggregation and management mechanism deployed in Pepper system. It provides high availability of logs, zero loss of logs and accessibility of logs from dashboard on single click.