Storage solutions II---Analyzing all the data all the time
Bill Blake · 2006
A growing number of on-line merchants, wireless phone operators, internet search companies and companies mining credit card transactions to discover customer buying pattern insights have reached the multi-Petabyte scale of data in their operations. It has been recently estimated that all credit card transaction information for all card holders in North America for a ten year period amounts to slightly less than 200 Terabytes, and many Internet businesses have amassed Petabytes of clickstreams. All of these companies are pushing the limits of relational database technology to perform "business intelligence" analysis of the hundreds of billions of records created by their business processes to better understand their customers and gain business advantage. A further challenge to this analysis is the need to perform very complex ad hoc queries, often in the span of hours in the case of fraud detection, on multi-terabyte fact files without the ability to perform the indexing and normalization of data needed to speed up relational data bases used in on-line transaction processing.This presentation will explain the work at Netezza Corporation developing massively parallel systems; purpose built for tera-scale database analysis that can deliver dramatically higher rates of I/O bandwidth to a single table--arguably the single most important metric in data warehouses--than large SMP or cluster of SMP systems. In a nutshell, Netezza is developing systems with a high degree of processor and storage integration that is reminiscent of an active disk approach to bringing processing power as physically close as possible to where the database tables reside. This is a significant departure from the current orthodoxy of large scale systems built with highly virtualized network connected storage. Additional performance gains are achieved by performing many of the key algebraic set operations of the relational database, such as the projection of columns and restriction of rows, in FPGA hardware as the data records stream off the disk drives. Next, the important role of the database optimizer and planner in supporting effective parallelization of queries written in the declarative query language ANSI standard SQL will be outlined. Finally, a number of actual results, many showing a 10 to 50 times speedup over comparable systems, achieved in the customer deployment of systems will be demonstrated.