The Micro-services architecture-based Text Analysis Engine
Swaswati Dutta, Parvej Reja Saleh, Srivathshan Ks · 2020
As the hype for social media and various other web applications increases, the need for modern scalable data management has become a sheer need to cope with the huge bulks of data. In the domain of text analytics, the data collected from various social media needs to be collected, cleaned, processed and visualized for providing various insights. This paper presents anarchitecture relying on the microservice approach for creating a data management backend for text analysis. The microservices-based data caching, data processing, data analysis, and data visualization methods can be applied to enhance the available data for providing efficient management services to the users. As of 2018, the micro-blogging site Twitter averaged at 321 million monthly active users[1]. Twitter provides a strong emphasis on real-time information. Information relating to geolocation and entities such as author id, author name/id, source, the reaction of people towards an event, etc can be extracted, stored and further analyzed by various Big Data Tools. This work aims at applying machine learning while adopting microservice approach for analyzing events from the Twitter platform in real-time based on keywords and geolocation, and finally, propose a user-friendly visualization based on the data. The tweets are stored and fetched using the Elasticsearch search engine. Indexing and standardizing of the Elasticsearch framework are used for large scale text mining. The results obtained from the query-based search engine are finally visualized using the powerful d3.js library.