Scalable Data Reporting Platform for A/B Tests
Deepak Kumar Vasthimal, Pavan Kumar Srirama, Arun Kumar Akkinapalli · 2019
A/B Test data platform enables users, such as a Product Manager or Product Owner, to answer important questions such as "How will this new product feature benefit ?" or "Does this new page layout improve user engagement and increase GMV (Gross Merchandise Value)?" Testing allows business units to conceptually explore new ideas with respect to page content/style, search algorithms, product features, etc. which can vary from subtle to radical variations. Such test variations can be easily targeted to segments of the total customer population based on desired percentage/ramp up and contextual criteria (geographic, system or app specific) providing a level of assurance before launching to a broader audience. A/B Test data platform runs hundreds of live experiments that are responsible for processing over hundreds of terabytes of reporting data contained in millions of files in Hadoop data platform and data warehouse tables using Hadoop infrastructure, consuming thousands of computing resources. The entire report generation process generates around ~300 metrics of different categories for each experiment. Report generation process takes around 10-12 hours daily to enable our millions of customers to experience small and large innovations to enable them to buy and sell products in various countries in different currencies and using different payment mechanisms in a better way every day. This paper describes the architecture and components of this scalable reporting infrastructure.