銆銆Nutch鏄竴涓紑婧愮殑Web鎼滅儲寮曟搸銆?
涓昏鍒嗕負涓や釜閮ㄥ垎錛氱埇铏玞rawler鍜屾煡璇earcher錛屼袱鑰呬箣闂寸殑鎺ュ彛鏄儲寮曘?
JDK1.6
Tomcat6.0
cygwin
nutch1.0
涓嬭澆鍦板潃錛?a >http://www.sun.com
瀹夎鐩綍錛欳:\Java\jdk1.6.0_14
淇敼鐜鍙橀噺錛?/p>
JAVA_HOME= C:\Java\jdk1.6.0_14
CLASSPATH= C:\Java\jdk1.6.0_14\lib\dt.jar; C:\Java\jdk1.6.0_14\lib\tool.jar
PATH= %JAVA_HOME%\bin
嫻嬭瘯錛?/p>
Java -version
涓嬭澆鍦板潃錛?
http://tomcat.apache.org/download-60.cgi?Preferred=http%3A%2F%2Fapache.freelamp.com
瀹夎鐩綍錛欳:\tomcat6
瀹夎嫻嬭瘯錛?
a) 鍚姩tomcat錛氬湪DOS涓嬶紝cd C:\tomcat6\bin\startup.bat
b) 嫻忚鍣ㄥ湴鍧鏍忚緭鍏ワ細
鍑虹幇tomcat涓婚〉鍒欐垚鍔?
鏇存敼C:\tomcat6\conf \tomcat-users.xml鍐呭
<tomcat-users>
銆銆<role rolename="manager"/>
銆銆<user username="tomcat" password="tomcat" roles="manager"/>
</tomcat-users>
涓嬭澆鍦板潃錛?/p>
http://www.cygwin.cn/
浣跨敤鍘熷洜錛?
榪愯Nutch鑷甫鐨勮剼鏈懡浠ら渶瑕丩inux鐨勭幆澧冿紝浣跨敤cygwin鏉ユā鎷熻鐜銆俢ygwin鏄湪windows騫沖彴涓婅繍琛岀殑unix妯℃嫙鐜銆?
瀹夎鐩綍錛欳:\cygwin
娉ㄦ剰錛?/strong>鍦ㄩ夋嫨涓嬭澆绔欑偣闇瑕佽緭鍏ュ涓嬬綉鍧錛屽茍鍦ㄦ渶鍚庨夋嫨鏈湴紓佺洏瀹夎錛?
嫻嬭瘯錛?/p>
榪涘叆cygwin
涓嬭澆鍦板潃錛?/p>
鐗堟湰錛歯utch-1.0
瑙e帇鍚庡埌錛欳:\nutch-1.0
璁劇疆Nutch鐨勭幆澧冨彉閲忥細
NUTCH_JAVA_HOME= C:\Java\jdk1.6.0_14
淇敼鐜鍙橀噺PATH 錛?/p>
PATH= %JAVA_HOME%\bin; % NUTCH_JAVA_HOME %\bin
鍦–:\nutch-1.0涓嬪緩绔媢rl.txt鏂囦歡鏉ュ埗瀹氱埇鍘誨垪琛?
鍦╰xt鏂囦歡涓啓鍏ラ渶瑕佺埇鍙栫殑緗戠珯鍦板潃http://dblp.lab/
嫻嬭瘯錛?
寮鍚疌ygwin
cd cygdriver/c/nutch-1.0
bin/nutch
鑻ュ嚭鐜拌嫢騫插懡浠わ紝鍒欒鏄嶯utch閰嶇疆鎴愬姛
# accept hosts in MY.DOMAIN.NAME
+^http://dblp.lab/
<configuration>
銆銆<property>
銆銆<name>http.agent.name</name>
銆銆<value>my nutch agent</value>
銆銆</property>
銆銆<property>
銆銆<name>http.agent.version</name>
銆銆<value>1.0</value>
銆銆</property>
</configuration>
cd /cygdrive/c/nutch-1.0
Bin/nutch crawl url.txt -dir crawled -depth 3 - threads 4 >&crawl.log
銆銆 鍏朵腑錛歞ir鏄寚瀹氱埇鍙栧唴瀹規墍瀛樻斁鐨勭洰褰曪紝depth琛ㄧず浠ヨ鐖彇緗戠珯欏剁駭緗戝潃涓鴻搗鐐圭殑鐖娣卞害錛宼hreads鎸囧畾騫跺彂鐨勭嚎紼嬫暟
http://localhost:8080/nutch-1.0/
鏇存敼c:\tomcat6\webapps\nutch-1.0\WEB-INF\classes\nutch-site.xml 錛屽皢鍐呭鏇存敼涓虹儲寮曠敓鎴愮殑鐩綍
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="nutch-conf.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<nutch-conf>
<property>
銆銆<name>searcher.dir</name>
銆銆<value>C:\nutch-1.0\crawled</value>
</property>
</nutch-conf>
淇敼鏂囦歡C:\tomcat6\conf\server.xml
<Connector port="8080" maxThreads="150" minSpareThreads="25"
maxSpareThreads="75" enableLookups="false" redirectPort="8443"
acceptCount="100" connectionTimeout="20000" disableUploadTimeout="true"
URIEncoding="UTF-8" useBodyEncodingForURI="true" protocol="HTTP/1.1" />
http://localhost:8080/nutch-1.0/
錛坆y 鐜嬫搗鏄庯級
灝嗙敤鎴烽渶姹傝繘琛屽垎瑙o細灝介噺瑕嗙洊鐢ㄦ埛鐨勮瘝姹囧綋鐒舵槸濂斤紝浣嗘槸闅忎箣浼氫駭鐢熼棶棰橈紝璇嶅簱鐨勮噧鑲垮寲鍜岃綆楃殑浠d環鍔犲ぇ錛屽氨瑕佸墧闄ゆ帀涓嶅繀瑕佺殑璇嶆潯淇℃伅浜嗭紝鍙垜浠庝箞鐭ラ亾鍝簺鎵嶆槸鐢ㄦ埛闇瑕佺殑鍛紝閫氳繃鑾峰彇鐢ㄦ埛杈撳叆鐜鎵懼埌絎﹀悎鐢ㄦ埛闇姹傜殑鏈灝忚瘝姹囷紝鎵嶆槸鐜嬮亾銆傝繖鏍瘋瘝搴撴瀯鎴愬熀鏈垎涓虹郴緇熷叡鏈夎瘝搴?杈呭姪璇嶅簱錛屽竷灞鍚堢悊錛屽叾涓悗鑰呮槸鍔ㄦ佸姞杞藉拰鏇存柊鐨勶紝寰堟櫤鑳藉寲銆?
涓у寲錛氭彁楂樼敤鎴蜂綋楠岀涓嶅紑涓у寲錛岄傚悎澶у鐨勬湁鍙兘閫傚悎鑷繁錛岄傚悎鑷繁鐨勪笉涓瀹氶傚悎澶у錛屼釜浣撲箣闂村瓨鍦ㄥ樊寮傦紝鎹曟崏鐢ㄦ埛鐨勪嬌鐢ㄤ範鎯紝鎵懼埌涓ч儴鍒嗗緢閲嶈錛岀敤鎴瘋瘝搴擄紝涓ц瘝絳夋妧鏈熀鏈В鍐抽儴鍒嗛棶棰樸傚紩鐢ㄥ師鏂囩殑璇濓細鈥滆褰曠敤鎴瘋緭鍏ョ殑瀛楄瘝錛屽涔犳煇涓敤鎴烽傜敤鐨勮瘝姹囷紱閫氳繃涓孌墊椂闂村湴涓嶆柇瀛︿範錛屽皢璇嶅簱閫愭笎鏀舵暃鍒版煇涓敤鎴瘋緭鍏ョ殑鏈浣崇姸鎬侊紝浠庤屽艦鎴愮敤鎴瘋瘝搴撱傞氳繃鍔ㄦ佽皟棰戯紝浣垮叾閫愭笎閫傚簲璇ョ敤鎴鳳紝褰撶敤鎴鋒槸鍞竴鏃訛紝寰堝ソ鍦版彁鍗囨晥鐜団濓紝寰堝埌浣嶃?
絳栫暐闂錛氬鎼滅嫍杈撳叆娉曟湁寰堝鐢ㄦ埛錛屽埄鐢ㄧ敤鎴風殑淇℃伅錛屽弽榪囨潵鍙堝洖棣堢粰鐢ㄦ埛錛岃繖鏍峰弽澶嶈繃紼嬶紝鏈緇堣杈懼埌涓涓敹鏁涘鉤琛$殑鐘舵併?
璧勬簮鐨勯噸瑕佹э細鎼滅嫍瀹濊吹鐨勪笢瑗挎槸璇枡搴擄紝鍚勭鎯蟲硶瀹炴柦鐨勫厛鍐蟲潯浠躲?
璇︾粏鐨凱PT鍦細http://d.onto.ac.cn/wipt/sogou.zip錛堟暀鑲茬綉鏈嶅姟鍣級
(by 闊╄壋鏅?