Big data clustering validity
Mania Tlili, Tarek M. Hamdani · 2014
Nowadays we communicate in a digital universe. In fact the amount of data (structured and unstructured) is exploding. That's what we call Big Data. The voluminous data are in the most of cases noisy and overlapping, their clustering makes critical challenges. In addition validating resulting partitions is a serious problem. In this paper we present a new fuzzy validity index able to interpret the best partition of Big Data clustering. Called Fuzzy Validity Index with Noise-Overlap Separation (FVINOS), this new technique provides sufficient interpretation of the properties of the Big Data by detecting the overall geometric structure within and between clusters. The main contribution of FVINOS is to define a crisp and fuzzy clustering validation taking in account the structure of Big Data sets.