Reference Article

from Wikipedia, the free encyclopedia

Web crawler

A web crawler (also known as a web spider or web robot) is a program or automated script which browses the World Wide Web in a methodical, automated manner.

This process is called Web crawling or spidering.

Many legitimate sites, in particular search engines, use spidering as a means of providing up-to-date data.

Web crawlers are mainly used to create a copy of all the visited pages for later processing by a search engine, that will index the downloaded pages to provide fast searches.

Crawlers can also be used for automating maintenance tasks on a Web site, such as checking links or validating HTML code.

Also, crawlers can be used to gather specific types of information from Web pages, such as harvesting e-mail addresses (usually for spam).

Note: This article excerpts material from the Wikipedia article "Web crawler", which is released under the GNU Free Documentation License.

See the following related content on ScienceDaily:


Related Videos

last updated on 2014-09-30 at 7:27 pm EDT

Quadriga: Worldwide Web Free Internet in Danger?

Quadriga: Worldwide Web Free Internet in Danger?

Deutsche Welle (Dec. 6, 2012) Up to now, the Internet has enabled millions of people to access information and express their opinion. Now, China and Russia are pushing for states to have a greater say over the World Wide Web at the UN telecom's conference in Dubai. Could this mean an end to the Internet as we know it? UN member states are set to review Internet regulations at the conference, which kicked off in the United Arab Emirates on Monday. User data has been administered by private organizations up to now. But Moscow has put forward a proposal saying member states should have equal rights to manage the Internet. Data protectionists and human rights campaigners fear greater curbs on cyber freedoms in authoritarian countries. And Europe is pressing for a new Internet tax to target the Web's largest content providers, including Google, Facebook and Apple. Under the proposals, the companies would have to pay a fee to send data along telecom operators' networks outside the US.
Powered by NewsLook.com
Google to Boost the Search Result Rankings of Encrypted Web Sites

Google to Boost the Search Result Rankings of Encrypted Web Sites

TheStreet (Aug. 7, 2014) Google says it's starting to use HTTPS encryption as a ranking signal as part of its effort to make the Internet safer. Google said for now, it's a lightweight signal, affecting fewer than 1% of global queries and carrying less weight than other signals like high-quality content. The company said it's giving webmasters time to switch to HTTPS because in the future, it may decide to strengthen the ranking signal. Websites using HTTP may start to slip down in Google's search results if they don't switch over to more secure encryption. Google said it encourages all Web sites to switch from HTTP to HTTPS to keep Web users more secure. Video provided by TheStreet
Powered by NewsLook.com
Guy Kawasaki: Photo Sharing Will Take Over the Internet

Guy Kawasaki: Photo Sharing Will Take Over the Internet

FORA.tv (Oct. 30, 2013) Guy Kawasaki: Photo Sharing Will Take Over the Internet The 3 Percent Conference - The Intercontinental Hotel Sponsored by Rosetta Collaborative consumption. Appcessories. Responsive web design. Wearable devices. Which trends and technologies will captivate consumers over the next few years? Hear from two of marketing's most revered trend-watchers as they compare notes, make predictions, and challenge attendees to think ahead about things that may not even be on your radar yet.
Powered by NewsLook.com
Japan Mobile Game Giants Ring Up Profits but for How Long?

Japan Mobile Game Giants Ring Up Profits but for How Long?

Reuters (Nov. 13, 2013) The rules have changed - Japan's mobile gamers are moving to apps to get their daily fix, leaving the old web-based gaming platforms behind. And some companies are starting to struggle as a result.
Powered by NewsLook.com

Related Stories


Share This



Search ScienceDaily

Number of stories in archives: 140,361

Find with keyword(s):
Enter a keyword or phrase to search ScienceDaily for related topics and research stories.

Save/Print:
Share:

Breaking News:
from the past week

In Other News

... from NewsDaily.com

Science News

Health News

Environment News

Technology News



Save/Print:
Share:

Free Subscriptions


Get the latest science news with ScienceDaily's free email newsletters, updated daily and weekly. Or view hourly updated newsfeeds in your RSS reader:

Get Social & Mobile


Keep up to date with the latest news from ScienceDaily via social networks and mobile apps:

Have Feedback?


Tell us what you think of ScienceDaily -- we welcome both positive and negative comments. Have any problems using the site? Questions?
Mobile: iPhone Android Web
Follow: Facebook Twitter Google+
Subscribe: RSS Feeds Email Newsletters
Latest Headlines Health & Medicine Mind & Brain Space & Time Matter & Energy Computers & Math Plants & Animals Earth & Climate Fossils & Ruins