The investigation of the possibility of automated collection of information in the hidden segment of the Internet
Andrey I. Levin, Igor A. Voronov · 2018
The principles of the work of search engines on the Internet have been studied for a long time, while the search on the hidden segment of the Internet remains poorly understood. The technologies for site search in hidden segment of the Internet are different from the Internet search technologies and, the methods used on the Internet are not applicable often in the hidden segment of the Internet. In this connection, this paper considers the basic principles of the work of search robot on the hidden segment of the Internet, namely, the architecture was developed and the module for downloading web pages into the data storage was implemented. In addition, the possibility of using classical algorithms, as well as their effectiveness in the hidden Internet compared to the Internet, was explored.