Featured Research

from universities, journals, and other organizations

Google Favored Over Other Search Engines By Webmasters

Date:
November 16, 2007
Source:
Penn State
Summary:
Website policy makers who use robots.txt files as gatekeepers to specify what is open and what is off limits to Web crawlers have a preference for Google over other search engines, say researchers whose study of more than 7,500 Websites revealed Google's advantage. "With the preference, Google can index some information which other search engines can't," according to one of the researchers.

Web site policy makers who use robots.txt files as gatekeepers to specify what is open and what is off limits to Web crawlers favor Google over other search engines, say Penn State researchers whose study of more than 7,500 Web sites revealed Google's advantage.

That finding was surprising, said C. Lee Giles, the David Reese Professor of Information Sciences and Technology who led the research team which developed a new search engine--BotSeer--for the study.

"We expected that robots.txt files would treat all search engines equally or maybe disfavor certain obnoxious bots, so we were surprised to discover a strong correlation between the robots favored and the search engines' market share," said Giles of Penn State's College of Information Sciences and Technology (IST).

Robots.txt files are not an official standard, but by informal agreement, they regulate Web crawlers -- also known as "spiders" and "bots" -- which mine the Web 24/7 for everything from the latest news to e-mail addresses. Web policy makers use the files found in a Web site's directory to restrict crawler access to non-public information. Robots.txt files also are used to reduce server load which can result in denial of service and shut down Web sites. But some Web policy makers and administrators are writing robots.txt files which are not uniformly blocking access.

Instead, those robots.txt files give access to Google, Yahoo and MSN while restricting other search engines, the researchers learned.

As an example, some U.S. government sites favor Google's bot--Googlebot--followed by Yahoo and MSN, according to the researchers.

While the study doesn't include explanations for why Web policy makers have opted to favor Google, the researchers know the choice was made consciously. Not using a robots.txt file gives all robots equal access to a Web site.

"Robots.txt files are written by Web policy makers and administrators who have to intentionally specify Google as the favored search engine," Giles said.

Not every site has a robots.txt file although the number is growing. Of the 7,500 sites analyzed by the researchers, about four in 10 had a robots.txt file--up from less than 1 in 10 in 1996.

That growth, which the researchers anticipate will continue, was one reason for the study.

The researchers didn't know what they would find when they set BotSeer on the loose to look at and index the content of the robots.txt files of the Web sites which spanned several market segments including government, newspaper, university and Fortune 1000 companies.

"Our intent was exploratory--to see if there was anything interesting," Councill said. Consumers with a soft spot for Google aren't affected by the bias. But consumers who prefer other search engines may be at a disadvantage.

"With the preference, Google can index some information which other search engines can't," Giles said.

This finding is described in a paper, "Determining Bias to Search Engines from Robots.txt," given at the recent 2007 IEEE/WIC/ACM International Conference on Web Intelligence in Silicon Valley. Besides Giles, the authors include Yang Sun and Ziming Zhuang, IST graduate students, and Isaac Councill, an IST post-doctoral scholar.


Story Source:

The above story is based on materials provided by Penn State. Note: Materials may be edited for content and length.


Cite This Page:

Penn State. "Google Favored Over Other Search Engines By Webmasters." ScienceDaily. ScienceDaily, 16 November 2007. <www.sciencedaily.com/releases/2007/11/071115122502.htm>.
Penn State. (2007, November 16). Google Favored Over Other Search Engines By Webmasters. ScienceDaily. Retrieved August 21, 2014 from www.sciencedaily.com/releases/2007/11/071115122502.htm
Penn State. "Google Favored Over Other Search Engines By Webmasters." ScienceDaily. www.sciencedaily.com/releases/2007/11/071115122502.htm (accessed August 21, 2014).

Share This




More Computers & Math News

Thursday, August 21, 2014

Featured Research

from universities, journals, and other organizations


Featured Videos

from AP, Reuters, AFP, and other news services

Families Can Now Ask Twitter To Remove Photos Of Deceased

Families Can Now Ask Twitter To Remove Photos Of Deceased

Newsy (Aug. 20, 2014) In the wake of a high-profile harassment case, Twitter says family members can ask for photos of dying or dead relatives to be taken down. Video provided by Newsy
Powered by NewsLook.com
Ballmer Leaves Microsoft's Board, Has Advice For Nadella

Ballmer Leaves Microsoft's Board, Has Advice For Nadella

Newsy (Aug. 19, 2014) In a letter to Microsoft CEO Satya Nadella, Ballmer said he's leaving the board of directors and offered tips on how the company can be successful. Video provided by Newsy
Powered by NewsLook.com
What Google Can Gain From Special Accounts For Children

What Google Can Gain From Special Accounts For Children

Newsy (Aug. 19, 2014) Google will reportedly offer official accounts for children younger than 13 years old. Video provided by Newsy
Powered by NewsLook.com
Breakingviews: Ebola's Economic Impact Could Eclipse SARS

Breakingviews: Ebola's Economic Impact Could Eclipse SARS

Reuters - Business Video Online (Aug. 18, 2014) The virus ravaging Africa has yet to spread elsewhere. Yet Asia’s SARS crisis in 2003 showed how changes to behaviour can hurt the economy more than the actual disease, says Breakingviews' Una Galani. Video provided by Reuters
Powered by NewsLook.com

Search ScienceDaily

Number of stories in archives: 140,361

Find with keyword(s):
Enter a keyword or phrase to search ScienceDaily for related topics and research stories.

Save/Print:
Share:

Breaking News:
from the past week

In Other News

... from NewsDaily.com

Science News

Health News

Environment News

Technology News



Save/Print:
Share:

Free Subscriptions


Get the latest science news with ScienceDaily's free email newsletters, updated daily and weekly. Or view hourly updated newsfeeds in your RSS reader:

Get Social & Mobile


Keep up to date with the latest news from ScienceDaily via social networks and mobile apps:

Have Feedback?


Tell us what you think of ScienceDaily -- we welcome both positive and negative comments. Have any problems using the site? Questions?
Mobile: iPhone Android Web
Follow: Facebook Twitter Google+
Subscribe: RSS Feeds Email Newsletters
Latest Headlines Health & Medicine Mind & Brain Space & Time Matter & Energy Computers & Math Plants & Animals Earth & Climate Fossils & Ruins