Quality Control: How to Analyze and Verify Financial Data
Michelle Duan, As Sas · 2010
®programmers dealing with massive financial data from a variety of data vendors, one major challenge we are facing is effective and efficient quality control. With data sources ranging from plain text files to SQL databases, from a few megabytes to several hundred gigabytes, it is critical for us to make sure the data we provide for our end users is consistent, reliable and up-to-date. This paper will provide an overview of techniques for analyzing and validating financial data products, including checking for uniqueness of key data items such as Company IDs, verifying if missing values or duplicate data entries were introduced during the converting process, comparing and confirming that any data addition, removal or changes between vintages are within normal range, etc. Different data steps and procedures will be used.