Monsters at the gate: When softbots visit web search engines
Bernard J. Jim Jansen, Amanda H. Spink, Jan Ole Pedersen · International Conference on Internet Computing · 2003
There has been little research investigating how Web agents search for information. We focused on three research questions: (1) How do agents interact with Web search engines? (2). What type of information are Web agents looking for and in what quantity?, and (3) What is the duration and frequency of Web agent interactions The queries examined for this study were submitted to Alta Vista on 8 September 2002, spanning a 24-hour period. Findings include: (1) when agents interact with Web search engines they use simpler queries than those submitted by human searchers, (2) Web agents are searching for a wide variety of information, with 60% of the terms used being unique, and (3) agent – Web search engine interaction is typically over several hours with multiple instances of interaction per second. Implications for Web agents and search engines are discussed.