Bots Make Up Majority Of Website Traffic

Non-human traffic on the Internet rose by 21 percent in the nine months between March and December 2013, with 61.5 percent generated by non-human entities, a third of which is malicious.

Cloud-based security and acceleration service provider Incapsula found that only 38.5 percent of website traffic was from actual living and breathing human beings, down from 49 percent human users a year ago.

Bot Traffic

According to Incapsula, 31 percent of this is from bots and search engines (up 55 percent from a year earlier); 5 percent is data scrappers; 4.5 percent is from hacking tools (down 10 percent); 0.5 percent from spammers (down a staggering 75 percent), and 20.5 percent is from other impersonators such as marketing intelligence gathering tools (up 8 percent).

Incapsula said that it had compiled the data after it observed 1.45 billion website visits, which occurred over a 90 day period. The data was apparently collected from a group of 20,000 sites on Incapsula’s network, with the traffic spread geographically to cover all of the world’s 249 countries.

“Compared to the previous report from 2012, we see a 21 percent growth in total bot traffic, which now represents 61.5 percent of website visitors,” blogged Incapsula’s Igal Zeifman. “The bulk of that growth is attributed to increased visits by good bots (ie certified agents of legitimate software, such as search engines) whose presence increased from 20 percent to 31 percent in 2013.”

Zeifman said that a study of the user-agent data, showed two plausible explanations of this growth. Firstly is it is down to the evolution of Web-based services, which the emergence of new online services, which introduces new bot types into the pool. “For instance, we see newly established SEO oriented services that crawl a site at a rate of 30-50 daily visits or more,” Zeifman wrote.

The second possible explanation for this growth is down to the increased activity of existing bots. According to Zeifman, visitation patterns of some good bots (for example search engine type crawlers) consist of re-occurring cycles. “In some cases we see that these cycles are getting shorter and shorter to allow higher sampling rates, which also results in additional bot traffic,” he wrote.

Spam Decline

The good news from the study is that spam has seen a dramatic decline in the past year, which decreased from two percent in 2012 to 0.5 percent in 2013. Zeifman believes the most plausible explanation for this steep decrease is Google’s anti-spam campaign.

“Based on our figures, it looks like Google was able to discourage link spamming practices, causing a 75 percent decrease in automated link spamming activity,” wrote Zeifman.

But Incapsula also warns that 31 percent of bots are still malicious, although this figure has not changed much over the past year.

The Incapsula study presents an interesting insight into what makes the up most of the traffic currently travelling over the World Wide Web. Some studies have predicted that by 2017, Internet traffic worldwide will triple, thanks to more connected devices and faster networks.

What do you know about Internet security? Find out with our quiz!