Focused Crawling

Farag, Mohamed Magdy Gharib; Khan, Mohammed Saquib Akmal; Mishra, Gaurav; Ganesh, Prasad Krishnamurthi

Focused Crawling

dc.contributor.author	Farag, Mohamed Magdy Gharib	en
dc.contributor.author	Khan, Mohammed Saquib Akmal	en
dc.contributor.author	Mishra, Gaurav	en
dc.contributor.author	Ganesh, Prasad Krishnamurthi	en
dc.date.accessioned	2012-12-12T01:35:11Z	en
dc.date.available	2012-12-12T01:35:11Z	en
dc.date.issued	2012-12-11	en
dc.description	The Crisis, Tragedy, and Recovery network (CTRnet, see external link: http://www.ctrnet.net) project makes use of general purpose crawlers, like Heritrix (see the list of like packages on p. 12 of 'Lucene in Action').However, these crawlers are strongly influenced by the quality of the seeds used, as well as other configuration details that govern the crawl.Focused crawlers typically use extra information, related to the topic of the crawl, to decide which links to follow from any page being examined. Thus, they may be able to reduce noise, and increase precision, though this may reduce recall.Focused crawling about events is particularly challenging. This project aims to explore this problem, design and implement a prototype that will improve upon existing solutions, and demonstrate its effectiveness with regard to CTRnet efforts.	en
dc.description.abstract	Finding information on WWW is difficult and challenging task because of the extremely large volume of the WWW. Search engine can be used to facilitate this task, but it is still difficult to cover all the webpages on the WWW and also to provide good results for all types of users and in all contexts. Focused crawling concept has been developed to overcome these difficulties. There are several approaches for developing a focused crawler. Classification-based approaches use classifiers in relevance estimation. Semantic-based approaches use ontologies for domain or topic representation and in relevance estimation. Link analysis approaches use text and link structure information in relevance estimation. The main differences between these approaches are: what policy is taken for crawling, how to represent the topic of interest, and how to estimate the relevance of webpages visited during crawling. We present in this report a modular architecture for focused crawling. We separated the design of the main components of focused crawling into modules to facilitate the exchange and integration of different modules. We will present here a classification-based focused crawler prototype based on our modular architecture.	en
dc.identifier.uri	http://hdl.handle.net/10919/19085	en
dc.language.iso	en_US	en
dc.rights	In Copyright	en
dc.rights.uri	http://rightsstatements.org/vocab/InC/1.0/	en
dc.subject	Focused Crawler	en
dc.subject	Crawler	en
dc.subject	Naive Bayes Classifier	en
dc.subject	Support Vector Machine Classifier	en
dc.title	Focused Crawling	en
dc.type	Technical report	en
dc.type	Working paper	en