Metadata Management for Large Statistical Databases
John McCarthy · eScholarship (California Digital Library) · 1982
Data description or metadat.apresents a significant database management challenge, particularly for scientific and statistical databases.Ideally, we would lllc.e to access and manipulate data and metadata using the same DBMS tools, but there are few systems that even begin to provide such integrated capabilities.This paper outlines a framework.for more integrated metadata management by synthesizing ideas from statistical analysis, bibliographic retrieval; data dictionary, and database ,management systems.Drawing on experience and examples from a' large statistical database project, the paper.discusses and analyzes:• general tYpes and uses of data about data • special types of nletadahi for statistical databases • metadata struct_ure and characteristics • principles,al'}p r~q~ir~ments for metadata management 1 •.lntr9du~tionAs databases continue-to grow in number, size, and complexity,.database management researchers, •system implementors, and users have recognized a need for more detailed data description or metadat.a to provide systematic information for e_nd-users, database administrators, appliCation programs,• and database management sOftware.They have also noted the desirability of Integrating metadata facilitfes; such as data dictionaries, with datlibas•a management systems [CODD82, CURT81, MEYE81 ].An "active," integrated data dictionary could provide information such as attribute names and characteristics, security requirements, etc., for the DBMS as well as for us•ers •and application programs.The DBMS could in turn manage meta data by treating the data dictionary as a database: Scientific• and statistical databases [TEIT77, CHAN81, BORA82, SHOS82] •share this need for integrated metadata management.In addition, they require metadata that are not yet found even in specialized data dictionary systems, They also need manipulation and analysis routines .that autqmatlcaUy .use and produce self-Selfdescribing data files facilitate use of data with external statistical analysis prcigrams.They are essential in an Interactive environment where the output of any routine can immediately serve as input to another.At present, few data management systems provide even rudimentary Integrated facilities for metadata manage-81811t.Even fewer provide facilities for statistical metadata.A:s a' result, most scientifiC and statistical users -' •..