SIPping from the data firehose
George H. John, Brian Lent · 1997
When mining large databases, the data extraction problem and the interface between the database and data mining algorithm become important issues. Rather than giving a mining algorithm full access to a database (by extracting to a flat file or other directly-accessible data structure), we propose the SQL Inter-face Protocol (SIP), which is a framework for inter-action between a mining algorithm and a database. The data continues to reside entirely within the data-base management system (DBMS), but the query in-terface to the database gives the data mining algorithm sufficient information to discover the same patterns it would have found with direct access to the data. This model of interaction brines several advantages; for ex-----ample, it allows a mining algorithm to be parallelized automatically just by using a parallelized DBMS to answer queries. We show how two families of mining algorithms may be implemented as “SIPpers, ” and we discuss related work in databases that should further enhance performance in the future.