Towards Business Intelligence over Unified Structured and Unstructured Data Using XML
Zhen Hua, Vishu Krishnamurthy · InTech eBooks · 2012
IntroductionTraditional data warehousing has been very successful in helping business enterprises to make intelligent decisions through declarative analysis of large amount of structured data stored in a relational database.However, not all enterprise data naturally fit into a relational model.Within an enterprise, there are huge amount of unstructured data, such as document content, emails, spreadsheets, that do not have a fixed schema, or have a very sparse or loose schema that cannot be effectively modeled using relational model.Yet, like relational data, unstructured data record many useful facts that are equally essential and important to be analyzed by businesses to make intelligent decisions.In this chapter, we propose an XML-enabled RDBMS that uses XML as the underlying logical data model to uniformly represent both well-structured relational data, semi-structured and unstructured data in building an enterprise data warehouse that is able to store and analyze any data regardless of existence of schema or not.We show how XQuery used in SQL/XML as a declarative language to do data query, analysis and transformation over both structured data and unstructured content in the data warehouse.We present the rationale for using XML as the logical data model for unified data warehouse query, XML extended inverted text index to integrate structured data query and context aware full text search for unstructured content so as to support efficient data analysis over large volume of structured and unstructured data.We argue that the technical approach of using XML to unify both structured and unstructured data in a warehouse has the potential to push business intelligence over all enterprise data to a new era. Concept of a data warehouseInductive reasoning refers to human arriving at a conclusion based on their observations.The inductive reasoning is a bottom up process where a general conclusion is reached from many instances observed and analyzed.(Myers,1986) Data Warehouse and decision support capabilities in modern database management system (DBMS) reflect the human inductive reasoning process.Data Warehouse (DWH) and decision support system (DSS), typically based on an RDBMS, involve extraction of operational data from business activities, transformation of the operational data, and loading of the results conforming to a fixed relational data model into a DWH store.Sophisticated data transformation, analysis, and mining can then be applied to a DWH to derive useful conclusions that assist businesses in www.intechopen.comBusiness Intelligence -Solution for Business Development 56 making intelligent decisions.Such evidence based decision-making process achieved through DWH is generally accepted as standard business intelligence practices in Enterprises.To achieve the goal of business intelligence, the design of DWH in DBMS must address the following requirements that are different from operational data store in Online Transaction Processing (OLTP) environment: Data Heterogeneity, Data Extraction and Batch Loading, Large Data Volumes, and Declarative Ad-Hoc Query Performance.Data Heterogeneity: since DWH store loads data from different operational store, therefore, it is likely that data may not be as homogeneous as operational store is.That is, data may not have well-defined common schema or may not have schema at all.The general trend is that unstructured data content and semi-structured data are more common than wellstructured data to process and to query.Data Extraction and Batch Load: building decision support system involves extraction of data from various operational stores and bulk loading of them into a central DWH store.This is known as ETL (ExTract Load) process.Data transformation is applied during ETL process to convert data from different operational stores into canonical form.To handle large data volume, tables can be partitioned and managed by several data server instances in a clustered environment.Query can use table partition criteria as selection qualification to work on different partitions of data.Data in DWH is typically partitioned based on certain criteria, such as timestamp based range partition criteria or hash partition based on record key or hybrid combination of the two.Such partition scheme facilitates life cycle management of data and enables query parallelism.Large Data Volume: Given the fast growing of memory core, it is reasonable to assume that operational data are able to all fit in memory such that in memory database processing becomes very attractive to overcome the gap between disk I/O speed and CPU speed.However, the amount of data for DWH store shall never be assumed to fit in main memory.Therefore, DWH design must take into consideration of selecting data layout to be disk I/O friendly.For example, design favoring small number of sequential large I/O requests generally delivers better performance than that of large number of random small I/O requests.Consequently, DWH design usually lays out data in a way to be optimized for large number of read requests instead of laying out data to be optimized for a large number of random frequent data modification requests.Declarative Ad-hoc Query Performance: declarative query is an attractive property for DBMS so that users can declaratively specify what they want to ask instead of procedurally programming the system on how to obtain the answer.Declarative query language processing with superior performance is critical for the success of DBMS.For operational store, the supported operations over data are usually pre-determined, therefore, the data query and modification requests have deterministic patterns.Operational store query is typically point query using id lookup that selects small amount of data using simple query criteria.However for DWH environment, the query requests are ad-hoc and exploring in nature.The query pattern is less predictive than that of operational store.DWH Query typically involves processing large amount of data to get summarized report to facilitate decision-making process or to mine data to derive insightful conclusion based on statistical analysis.So DWH query can be long running compared with short running point query in operational store.www.intechopen.