Reliable Web-crawling for estimating node indegree in a Web-graph

Dr. Viv Cothey Post-Doctoral Researcher School of Computing and IT Wolverhampton University Wolverhampton, UK

DEPARTMENT SEMINAR DATE: Monday, August 30, 2004 TIME: 3:30pm PLACE: Kinesiology 246 *** Everyone is welcome ***

Abstract

The principles of Web-crawling appear well known but practice receives little attention. Yet understanding the practice of Web-crawling becomes important when the object of study is the resulting Web-graph and its structural properties. Web-crawling can be regarded as a sampling procedure where different selection methods can be adopted. For example in "content-crawling" the goal is to sample the Web's content in order to support Web content discovery. However it can be seen that the heuristics used by content-crawlers do not reliably represent the Web's link structure.

Different Web-crawling heuristics have been used to construct an alternative Web link-crawler that is designed to reliably sample the Web's link structure. This work is part of the European Union WISER project and is a work in progress.

In this presentation I discuss:

• the experience of building and operating a research Web-crawler and in particular some issues associated with sharing crawl data
• some initial findings from analyses of the link structure within the EU and some open questions and future directions for "Web-graph" research.
• some reliability issues associated with Web-crawling

About the speaker

Viv is a Postdoctoral research fellow in the Statistical Cybermetrics Research Group in the School of Computing and IT, University of Wolverhampton. He completed his Ph.D. from the University of Bristol in 2002 in Web information strategies.

[an error occurred while processing this directive]