Effective Application of the Big Data Techniques in the Internet Resource Processing System
Damian Jankowski, Przemysław Ścibiorek, Sebastian Szwaczyk, Zbigniew Piotrowski, Paweł Kaczmarek · 2024
Browsing online resources is crucial for detecting activities like criminal or anomalous behavior, but it involves analyzing vast amounts of data. Big data tools are essential for this task. This paper presents AISearcher, an automated system for searching classified content, including data from social networking sites. It uses specialist tools like Cassandra, Storm, and Kafka for processing and storage. The paper discusses implementation techniques for handling large datasets with the NoSQL Cassandra database, supporting the detection of specific content. Three research scenarios compare the effectiveness of different architectural approaches, with a detailed description of the big data processing components.