Application of Spark parallelization technology in architectural text classification
Xiang Zhang, Bingfeng Wu, Lili Dong, Na Ye · Journal of Computational Methods in Sciences and Engineering · 2018
There is a lot of Chinese information on the Internet, including many architectural texts. In the field of building information, how to effectively find and organize these architectural texts has become a hot research topic. In this paper, the parallel optimizing of Deep Belief Network (DBN) implem ented by Spark is proposed to classify the architectural texts. Firstly, the architectural texts preprocessing is automatically parallelized by spark technology. Secondly, the Word2Vec tool is used to represent the architectural texts and construct the feature dictionary. Thirdly the architectural text classification model is constructed by the Deep Belief Network. Finally, architectural text classification is parallelized by the Spark cluster. From experimental results we can see that the training time of parallel DBN is shortened to a quarter of the original, and gets less errors and better performance than that of the traditional shallow learning method.