Abstract:
In a focused cralwing system, multi-crawlers crawl parallelly Web and download Web pages. It is one of hotspot researches for a search engine how the diferent focused crawlers avoid to visit the same URLs and download efficiently Web pages related to the search topic. In order to rapidly accomplish the crawling tasks of the system for the specific topic, and embody fully every Web crawler's ability, the author considers that these history visited Web pages (URLs) of every focused crawler reflect their backgroup knowledge. On the basis of cralwing independently, collaborating togather and competing with each other for Web crawlers of the system, the paper proposes the novel understanding, cooperating and competing strategy of concept context grap. It includes four aspects as follows: constructing the mathematical model of backgrounp knowledge of every Web crawler based on hierarchy concept context graph, according to the semantic characteristics-concepts of Web pages and their semantic relationships among the concepts; studying the understanding method and model among Web crawlers based on hierarchy concept context graph; studying and implementing the cooprtating, competing model among Web crawlers of the same group managing by a F-Agent; studying and implementing the cooprtating, competing model among Web crawlers of the diferent group managing by F-Agents.