Comparison of performance of the popular approaches to implementing parallel crawlers
Andriy Sultanov, Maksym Protsyk, Maksym Kuzyshyn, Daria Omelkina, Vyacheslav Shevchuk, Oleg Farenyuk · 2021 IEEE 16th International Conference on Computer Sciences and Information Technologies (CSIT) · 2021
This research aims to compare the efficiency of the several popular parallel programming approaches used for an I/O-heavy workload: process pools, thread pools, and asynchronous I/O using the M:N model. As a model task, parallel Web crawlers were used, implemented using the C++ language. Their performance was measured using a wall-time approach. We tackled the inherent noise of this method using randomization and statistical hypothesis testing. The results of the experiment showed that all approaches are plausible for crawlers, though the epoll-based one is substantially faster than the two others and the thread-based is faster than the process-based.