浣滆咃細forfuture1978 (from CSDN)
鎴戜滑閮界煡閬擄紝Lucene鏄竴涓紑鏀炬簮浠ｇ爜 鐨勫叏鏂囨绱㈠紩鎿庡伐鍏峰寘銆傞偅涔堝叏鏂囨绱㈠埌搴曟槸浠涔堬紵 榪欒浠庢垜浠敓媧諱腑鐨勬暟鎹璧楓?/span>
鎴戜滑鐢熸椿涓殑鏁版嵁鎬諱綋鍒嗕負涓ょ錛?span style="color: #0000ff">緇撴瀯鍖栨暟鎹?/strong> 鍜?span style="color: #0000ff">闈炵粨鏋勫寲鏁版嵁 銆?/p>
褰撶劧鏈夌殑鍦版柟榪樹細鎻愬埌絎笁縐嶏紝鍗婄粨鏋勫寲鏁版嵁錛屽XML錛孒TML絳夛紝褰撴牴鎹渶瑕佸彲鎸夌粨鏋勫寲鏁版嵁鏉ュ鐞嗭紝涔熷彲鎶藉彇鍑虹函鏂囨湰鎸夐潪緇撴瀯鍖栨暟鎹潵澶勭悊銆?/p>
闈炵粨鏋勫寲鏁版嵁鍙堜竴縐嶅彨娉曞彨鍏ㄦ枃鏁版嵁銆?/strong>
鎸夌収鏁版嵁鐨勫垎綾伙紝鎼滅儲涔熷垎涓轟袱縐嶏細 瀵歸潪緇撴瀯鍖栨暟鎹篃鍗沖鍏ㄦ枃鏁版嵁鐨勬悳绱富瑕佹湁涓ょ鏂規(guī)硶錛?/p>
涓縐嶆槸欏哄簭鎵弿娉?/strong> (Serial Scanning) 錛?/strong> 鎵璋撻『搴忔壂鎻忥紝姣斿瑕佹壘鍐呭鍖呭惈鏌愪竴涓瓧絎︿覆鐨勬枃浠訛紝灝辨槸涓涓枃妗ｄ竴涓枃妗ｇ殑鐪嬶紝瀵逛簬姣忎竴涓枃妗ｏ紝浠庡ご鐪嬪埌灝撅紝濡傛灉姝ゆ枃妗ｅ寘鍚瀛楃涓詫紝鍒欐鏂囨。涓烘垜浠鎵劇殑鏂囦歡錛屾帴鐫鐪嬩笅涓涓枃浠訛紝鐩村埌鎵弿瀹屾墍鏈夌殑鏂囦歡銆傚鍒╃敤windows鐨勬悳绱篃鍙互鎼滅儲鏂囦歡鍐呭錛屽彧鏄浉褰撶殑鎱€傚鏋滀綘鏈変竴涓?0G紜洏錛屽鏋滄兂鍦ㄤ笂闈㈡壘鍒頒竴涓唴瀹瑰寘鍚煇瀛楃涓茬殑鏂囦歡錛屼笉鑺變粬鍑犱釜灝忔椂錛屾曟槸鍋氫笉鍒般侺inux涓嬬殑grep鍛戒護涔熸槸榪欎竴縐嶆柟寮忋傚ぇ瀹跺彲鑳借寰楄繖縐嶆柟娉曟瘮杈冨師濮嬶紝浣嗗浜庡皬鏁版嵁閲忕殑鏂囦歡錛岃繖縐嶆柟娉曡繕鏄渶鐩存帴錛屾渶鏂逛究鐨勩備絾鏄浜庡ぇ閲忕殑鏂囦歡錛岃繖縐嶆柟娉曞氨寰堟參浜嗐?/p>
鏈変漢鍙兘浼氳錛屽闈炵粨鏋勫寲鏁版嵁欏哄簭鎵弿寰堟參錛屽緇撴瀯鍖栨暟鎹殑鎼滅儲鍗寸浉瀵硅緝蹇紙鐢變簬緇撴瀯鍖栨暟鎹湁涓瀹氱殑緇撴瀯鍙互閲囧彇涓瀹氱殑鎼滅儲綆楁硶鍔犲揩閫熷害錛夛紝閭ｄ箞鎶婃垜浠殑闈炵粨鏋勫寲鏁版嵁鎯沖姙娉曞紕寰楁湁涓瀹氱粨鏋勪笉灝辮浜嗗悧錛?/p>
榪欑鎯蟲硶寰堝ぉ鐒訛紝鍗存瀯鎴愪簡鍏ㄦ枃媯绱㈢殑鍩烘湰鎬濊礬錛屼篃鍗沖皢闈炵粨鏋勫寲鏁版嵁涓殑涓閮ㄥ垎淇℃伅鎻愬彇鍑烘潵錛岄噸鏂扮粍緇囷紝浣垮叾鍙樺緱鏈変竴瀹氱粨鏋勶紝鐒跺悗瀵規(guī)鏈変竴瀹氱粨鏋勭殑鏁版嵁榪涜鎼滅儲錛屼粠鑰岃揪鍒版悳绱㈢浉瀵硅緝蹇殑鐩殑銆?/p>
榪欓儴鍒嗕粠闈炵粨鏋勫寲鏁版嵁涓彁鍙栧嚭鐨勭劧鍚庨噸鏂扮粍緇囩殑淇℃伅錛屾垜浠О涔?strong>绱㈠紩 榪欑璇存硶姣旇緝鎶借薄錛屼婦鍑犱釜渚嬪瓙灝卞緢瀹規(guī)槗鏄庣櫧錛屾瘮濡傚瓧鍏革紝瀛楀吀鐨勬嫾闊寵〃鍜岄儴棣栨瀛楄〃灝辯浉褰撲簬瀛楀吀鐨勭儲寮曪紝瀵規(guī)瘡涓涓瓧鐨勮В閲婃槸闈炵粨鏋勫寲鐨勶紝濡傛灉瀛楀吀娌℃湁闊寵妭琛ㄥ拰閮ㄩ媯瀛楄〃錛屽湪鑼尗杈炴搗涓壘涓涓瓧鍙兘欏哄簭鎵弿銆傜劧鑰屽瓧鐨勬煇浜涗俊鎭彲浠ユ彁鍙栧嚭鏉ヨ繘琛岀粨鏋勫寲澶勭悊錛屾瘮濡傝闊籌紝灝辨瘮杈冪粨鏋勫寲錛屽垎澹版瘝鍜岄煹姣嶏紝鍒嗗埆鍙湁鍑犵鍙互涓涓鍒椾婦錛屼簬鏄皢璇婚煶鎷垮嚭鏉ユ寜涓瀹氱殑欏哄簭鎺掑垪錛屾瘡涓欏硅闊抽兘鎸囧悜姝ゅ瓧鐨勮緇嗚В閲婄殑欏墊暟銆傛垜浠悳绱㈡椂鎸夌粨鏋勫寲鐨勬嫾闊蟲悳鍒拌闊籌紝鐒跺悗鎸夊叾鎸囧悜鐨勯〉鏁幫紝渚垮彲鎵懼埌鎴戜滑鐨勯潪緇撴瀯鍖栨暟鎹斺斾篃鍗沖瀛楃殑瑙ｉ噴銆?/p>
榪欑鍏堝緩绔嬬儲寮曪紝鍐嶅绱㈠紩榪涜鎼滅儲鐨勮繃紼嬪氨鍙叏鏂囨绱?Full-text Search) 銆?/strong>
涓嬮潰榪欏箙鍥炬潵鑷奓ucene in action銆嬶紝浣嗗嵈涓嶄粎浠呮弿榪頒簡Lucene鐨勬绱㈣繃紼嬶紝鑰屾槸鎻忚堪浜嗗叏鏂囨绱㈢殑涓鑸繃紼嬨?img height="479" alt="" src="http://dl.javaeye.com/upload/picture/pic/55997/9512a770-0784-3231-9da5-608a46014b2d.png" width="544" /> 鍏ㄦ枃媯绱㈠ぇ浣撳垎涓や釜榪囩▼錛?span style="color: #0000ff">绱㈠紩鍒涘緩 (Indexing) 鍜?span style="color: #0000ff">鎼滅儲绱㈠紩 (Search) 銆?/p>
浜庢槸鍏ㄦ枃媯绱㈠氨瀛樺湪涓変釜閲嶈闂錛?/p>
1. 绱㈠紩閲岄潰絀剁珶瀛樹簺浠涔堬紵(Index) 2. 濡備綍鍒涘緩绱㈠紩錛?Indexing) 3. 濡備綍瀵圭儲寮曡繘琛屾悳绱紵(Search) 涓嬮潰鎴戜滑欏哄簭瀵規(guī)瘡涓釜闂榪涜鐮旂┒銆?/p>
涓銆?/span> 绱㈠紩閲岄潰絀剁珶瀛樹簺浠涔堬紵(Index) 棣栧厛鎴戜滑鏉ョ湅涓轟粈涔堥『搴忔壂鎻忕殑閫熷害鎱細 鍏跺疄鏄敱浜庢垜浠兂瑕佹悳绱㈢殑淇℃伅鍜岄潪緇撴瀯鍖栨暟鎹腑鎵瀛樺偍鐨勪俊鎭笉涓鑷撮犳垚鐨勩?/p>
闈炵粨鏋勫寲鏁版嵁涓墍瀛樺偍鐨勪俊鎭槸姣忎釜鏂囦歡鍖呭惈鍝簺瀛楃涓詫紝涔熷嵆宸茬煡鏂囦歡錛屾姹傚瓧絎︿覆鐩稿瀹規(guī)槗錛屼篃鍗蟲槸浠庢枃浠跺埌瀛楃涓茬殑鏄犲皠銆傝屾垜浠兂鎼滅儲鐨勪俊鎭槸鍝簺鏂囦歡鍖呭惈姝ゅ瓧絎︿覆錛屼篃鍗沖凡鐭ュ瓧絎︿覆錛屾姹傛枃浠訛紝涔熷嵆浠庡瓧絎︿覆鍒版枃浠剁殑鏄犲皠銆備袱鑰呮伆鎭扮浉鍙嶃備簬鏄鏋滅儲寮曟昏兘澶熶繚瀛樹粠瀛楃涓插埌鏂囦歡鐨勬槧灝勶紝鍒欎細澶уぇ鎻愰珮鎼滅儲閫熷害銆?/p>
鐢變簬浠庡瓧絎︿覆鍒版枃浠剁殑鏄犲皠鏄枃浠跺埌瀛楃涓叉槧灝勭殑鍙嶅悜榪囩▼錛屼簬鏄繚瀛樿繖縐嶄俊鎭殑绱㈠紩縐頒負鍙嶅悜绱㈠紩 銆?/p>
鍙嶅悜绱㈠紩鐨勬墍淇濆瓨鐨勪俊鎭竴鑸涓嬶細 鍋囪鎴戠殑鏂囨。闆嗗悎閲岄潰鏈?00綃囨枃妗ｏ紝涓轟簡鏂逛究琛ㄧず錛屾垜浠負鏂囨。緙栧彿浠?鍒?00錛屽緱鍒頒笅闈㈢殑緇撴瀯 宸﹁竟淇濆瓨鐨勬槸涓緋誨垪瀛楃涓詫紝縐頒負璇嶅吀 銆?/span> 姣忎釜瀛楃涓查兘鎸囧悜鍖呭惈姝ゅ瓧絎︿覆鐨勬枃妗?Document)閾捐〃錛屾鏂囨。閾捐〃縐頒負鍊掓帓琛?/strong> (Posting List) 銆?/p>
鏈変簡绱㈠紩錛屼究浣夸繚瀛樼殑淇℃伅鍜岃鎼滅儲鐨勪俊鎭竴鑷達紝鍙互澶уぇ鍔犲揩鎼滅儲鐨勯熷害銆?/p>
姣斿璇達紝鎴戜滑瑕佸鎵炬棦鍖呭惈瀛楃涓?#8220;lucene”鍙堝寘鍚瓧絎︿覆“solr”鐨勬枃妗ｏ紝鎴戜滑鍙渶瑕佷互涓嬪嚑姝ワ細 1. 鍙栧嚭鍖呭惈瀛楃涓?#8220;lucene”鐨勬枃妗ｉ摼琛ㄣ?/p>
2. 鍙栧嚭鍖呭惈瀛楃涓?#8220;solr”鐨勬枃妗ｉ摼琛ㄣ?/p>
3. 閫氳繃鍚堝茍閾捐〃錛屾壘鍑烘棦鍖呭惈“lucene”鍙堝寘鍚?#8220;solr”鐨勬枃浠躲?img height="55" alt="" src="http://hxraid.javaeye.com/upload/picture/pic/56001/7e04a727-6cdb-3434-a260-023bc163b497.jpg" width="544" /> 鐪嬪埌榪欎釜鍦版柟錛屾湁浜哄彲鑳戒細璇達紝鍏ㄦ枃媯绱㈢殑紜姞蹇簡鎼滅儲鐨勯熷害錛屼絾鏄浜嗙儲寮曠殑榪囩▼錛屼袱鑰呭姞璧鋒潵涓嶄竴瀹氭瘮欏哄簭鎵弿蹇灝戙傜殑紜紝鍔犱笂绱㈠紩鐨勮繃紼嬶紝鍏ㄦ枃媯绱笉涓瀹氭瘮欏哄簭鎵弿蹇紝灝ゅ叾鏄湪鏁版嵁閲忓皬鐨勬椂鍊欐洿鏄姝ゃ傝屽涓涓緢澶ч噺鐨勬暟鎹垱寤虹儲寮曚篃鏄竴涓緢鎱㈢殑榪囩▼銆?/p>
鐒惰屼袱鑰呰繕鏄湁鍖哄埆鐨勶紝欏哄簭鎵弿鏄瘡嬈￠兘瑕佹壂鎻忥紝鑰屽垱寤虹儲寮曠殑榪囩▼浠呬粎闇瑕佷竴嬈★紝浠ュ悗渚挎槸涓鍔蟲案閫哥殑浜嗭紝姣忔鎼滅儲錛屽垱寤虹儲寮曠殑榪囩▼涓嶅繀緇忚繃錛屼粎浠呮悳绱㈠垱寤哄ソ鐨勭儲寮曞氨鍙互浜嗐?/p>
榪欎篃鏄叏鏂囨悳绱㈢浉瀵逛簬欏哄簭鎵弿鐨勪紭鍔夸箣涓錛氫竴嬈＄儲寮曪紝澶氭浣跨敤銆?/strong> 浜屻?/span> 濡備綍鍒涘緩绱㈠紩錛?Indexing) 鍏ㄦ枃媯绱㈢殑绱㈠紩鍒涘緩榪囩▼涓鑸湁浠ヤ笅鍑犳錛?/p>
1銆佷竴浜涜绱㈠紩鐨勫師濮嬫枃妗?Document) 涓轟簡鏂逛究璇存槑绱㈠紩鍒涘緩榪囩▼錛岃繖閲岀壒鎰忕敤涓や釜鏂囦歡涓轟緥錛?/p>
鏂囦歡涓錛歋tudents should be allowed to go out with their friends, but not allowed to drink beer. 鏂囦歡浜岋細My friend Jerry went to school to see his students but found them drunk which is not allowed. 2銆佸皢鍘熷鏂囨。浼犵粰鍒嗚瘝緇勪歡(Tokenizer) 鍒嗚瘝緇勪歡(Tokenizer)浼氬仛浠ヤ笅鍑犱歡浜嬫儏( 姝よ繃紼嬬О涓篢okenize) 錛?/strong> 1. 灝嗘枃妗ｅ垎鎴愪竴涓竴涓崟鐙殑鍗曡瘝銆?/strong> 2. 鍘婚櫎鏍囩偣絎﹀彿銆?/strong> 3. 鍘婚櫎鍋滅敤璇?Stop word) 銆?/strong> 鎵璋?strong>鍋滅敤璇?Stop word)
鑻辮涓尯璇?Stop word)濡傦細“the”,“a”錛?#8220;this”絳夈?/p>
瀵逛簬姣忎竴縐嶈璦鐨勫垎璇嶇粍浠?Tokenizer)錛岄兘鏈変竴涓仠璇?stop word)闆嗗悎銆?/p>
緇忚繃鍒嗚瘝(Tokenizer) 鍚庡緱鍒扮殑緇撴灉縐頒負璇嶅厓(Token) 銆?/strong>
鍦ㄦ垜浠殑渚嬪瓙涓紝渚垮緱鍒頒互涓嬭瘝鍏?Token)錛?/p>
“Students”錛?#8220;allowed”錛?#8220;go”錛?#8220;their”錛?#8220;friends”錛?#8220;allowed”錛?#8220;drink”錛?#8220;beer”錛?#8220;My”錛?#8220;friend”錛?#8220;Jerry”錛?#8220;went”錛?#8220;school”錛?#8220;see”錛?#8220;his”錛?#8220;students”錛?#8220;found”錛?#8220;them”錛?#8220;drunk”錛?#8220;allowed”銆?/p>
3銆佸皢寰楀埌鐨勮瘝鍏?Token)浼犵粰璇█澶勭悊緇勪歡(Linguistic Processor)
璇█澶勭悊緇勪歡(linguistic processor)涓昏鏄寰楀埌鐨勮瘝鍏?Token)鍋氫竴浜涘悓璇█鐩稿叧鐨勫鐞嗐?/p>
瀵逛簬鑻辮錛岃璦澶勭悊緇勪歡(Linguistic Processor) 涓鑸仛浠ヤ笅鍑犵偣錛?/strong>
1. 鍙樹負灝忓啓(Lowercase) 銆?/strong>
2. 灝嗗崟璇嶇緝鍑忎負璇嶆牴褰㈠紡錛屽“cars ”鍒?#8220;car ”絳夈傝繖縐嶆搷浣滅О涓猴細stemming 銆?/strong>
3. 灝嗗崟璇嶈漿鍙樹負璇嶆牴褰㈠紡錛屽“drove ”鍒?#8220;drive ”絳夈傝繖縐嶆搷浣滅О涓猴細lemmatization 銆?/strong>
Stemming 鍜?lemmatization鐨勫紓鍚岋細
- 鐩稿悓涔嬪錛歋temming鍜宭emmatization閮借浣胯瘝姹囨垚涓鴻瘝鏍瑰艦寮忋?
- 涓よ呯殑鏂瑰紡涓嶅悓錛?
- Stemming閲囩敤鐨勬槸“緙╁噺”鐨勬柟寮忥細“cars”鍒?#8220;car”錛?#8220;driving”鍒?#8220;drive”銆?
- Lemmatization閲囩敤鐨勬槸“杞彉”鐨勬柟寮忥細“drove”鍒?#8220;drive”錛?#8220;driving”鍒?#8220;drive”銆?
- 涓よ呯殑綆楁硶涓嶅悓錛?
- Stemming涓昏鏄噰鍙栨煇縐嶅浐瀹氱殑綆楁硶鏉ュ仛榪欑緙╁噺錛屽鍘婚櫎“s”錛屽幓闄?#8220;ing”鍔?#8220;e”錛屽皢“ational”鍙樹負“ate”錛屽皢“tional”鍙樹負“tion”銆?
- Lemmatization涓昏鏄噰鐢ㄤ繚瀛樻煇縐嶅瓧鍏哥殑鏂瑰紡鍋氳繖縐嶈漿鍙樸傛瘮濡傚瓧鍏鎬腑鏈?#8220;driving”鍒?#8220;drive”錛?#8220;drove”鍒?#8220;drive”錛?#8220;am, is, are”鍒?#8220;be”鐨勬槧灝勶紝鍋氳漿鍙樻椂錛屽彧瑕佹煡瀛楀吀灝卞彲浠ヤ簡銆?
- Stemming鍜宭emmatization涓嶆槸浜掓枼鍏崇郴錛屾槸鏈変氦闆嗙殑錛屾湁鐨勮瘝鍒╃敤榪欎袱縐嶆柟寮忛兘鑳借揪鍒扮浉鍚岀殑杞崲銆?
璇█澶勭悊緇勪歡(linguistic processor)鐨勭粨鏋滅О涓鴻瘝(Term) 銆?/strong>
鍦ㄦ垜浠殑渚嬪瓙涓紝緇忚繃璇█澶勭悊錛屽緱鍒扮殑璇?Term)濡備笅錛?/p>
“student”錛?#8220;allow”錛?#8220;go”錛?#8220;their”錛?#8220;friend”錛?#8220;allow”錛?#8220;drink”錛?#8220;beer”錛?#8220;my”錛?#8220;friend”錛?#8220;jerry”錛?#8220;go”錛?#8220;school”錛?#8220;see”錛?#8220;his”錛?#8220;student”錛?#8220;find”錛?#8220;them”錛?#8220;drink”錛?#8220;allow”銆?/p>
涔熸鏄洜涓烘湁璇█澶勭悊鐨勬楠わ紝鎵嶈兘浣挎悳绱rove錛岃宒rive涔熻兘琚悳绱㈠嚭鏉ャ?/p>
4銆佸皢寰楀埌鐨勮瘝(Term)浼犵粰绱㈠紩緇勪歡(Indexer)
绱㈠紩緇勪歡涓昏鏄負浜嗗緱鍒頒笅闈㈣繖涓浘錛?img alt="" src="http://hxraid.javaeye.com/upload/picture/pic/56003/7ca0f9b1-e82a-35c1-a97e-e40c2d4a70c9.jpg" />
鍦ㄦ琛ㄤ腑錛屾湁鍑犱釜瀹氫箟錛?/p>
- Document Frequency 鍗蟲枃妗ｉ嬈★紝琛ㄧず鎬誨叡鏈夊灝戞枃浠跺寘鍚璇?Term)銆?
- Frequency 鍗寵瘝棰戠巼錛岃〃紺烘鏂囦歡涓寘鍚簡鍑犱釜姝よ瘝(Term)銆?
鎵浠ュ璇?Term) “allow”鏉ヨ錛屾誨叡鏈変袱綃囨枃妗ｅ寘鍚璇?Term)錛屼粠鑰岃瘝(Term)鍚庨潰鐨勬枃妗ｉ摼琛ㄦ誨叡鏈変袱欏癸紝絎竴欏硅〃紺哄寘鍚?#8220;allow”鐨勭涓綃囨枃妗ｏ紝鍗?1鍙鋒枃妗ｏ紝姝ゆ枃妗ｄ腑錛?#8220;allow”鍑虹幇浜?嬈★紝絎簩欏硅〃紺哄寘鍚?#8220;allow”鐨勭浜屼釜鏂囨。錛屾槸2鍙鋒枃妗ｏ紝姝ゆ枃妗ｄ腑錛?#8220;allow”鍑虹幇浜?嬈°?/p>
鍒版涓烘錛岀儲寮曞凡緇忓垱寤哄ソ浜嗭紝鎴戜滑鍙互閫氳繃瀹冨緢蹇殑鎵懼埌鎴戜滑鎯寵鐨勬枃妗ｃ?/p>
鑰屼笖鍦ㄦ榪囩▼涓紝鎴戜滑鎯婂枩鍦板彂鐜幫紝鎼滅儲“drive”錛?#8220;driving”錛?#8220;drove”錛?#8220;driven”涔熻兘澶熻鎼滃埌銆傚洜涓哄湪鎴戜滑鐨勭儲寮曚腑錛?#8220;driving”錛?#8220;drove”錛?#8220;driven”閮戒細緇忚繃璇█澶勭悊鑰屽彉鎴?#8220;drive”錛屽湪鎼滅儲鏃訛紝濡傛灉鎮(zhèn)ㄨ緭鍏?#8220;driving”錛岃緭鍏ョ殑鏌ヨ璇彞鍚屾牱緇忚繃鎴戜滑榪欓噷鐨勪竴鍒頒笁姝ワ紝浠庤屽彉涓烘煡璇?#8220;drive”錛屼粠鑰屽彲浠ユ悳绱㈠埌鎯寵鐨勬枃妗ｃ?/p>
涓夈?/span> 濡備綍瀵圭儲寮曡繘琛屾悳绱紵(Search)
鍒拌繖閲屼技涔庢垜浠彲浠ュ甯?#8220;鎴戜滑鎵懼埌鎯寵鐨勬枃妗ｄ簡”銆?/p>
鐒惰屼簨鎯呭茍娌℃湁緇撴潫錛屾壘鍒頒簡浠呬粎鏄叏鏂囨绱㈢殑涓涓柟闈€備笉鏄悧錛熷鏋滀粎浠呭彧鏈変竴涓垨鍗佷釜鏂囨。鍖呭惈鎴戜滑鏌ヨ鐨勫瓧絎︿覆錛屾垜浠殑紜壘鍒頒簡銆傜劧鑰屽鏋滅粨鏋滄湁涓鍗冧釜錛岀敋鑷蟲垚鍗冧笂涓囦釜鍛紵閭ｄ釜鍙堟槸鎮(zhèn)ㄦ渶鎯寵鐨勬枃浠跺憿錛?/p>
鎵撳紑Google鍚э紝姣斿璇存?zhèn)ㄦ儧_湪寰蔣鎵句喚宸ヤ綔錛屼簬鏄?zhèn)ㄨ緭鍏?#8220;Microsoft job”錛屾?zhèn)ㄥ嵈鍙戠幇鎬誨叡鏈?2600000涓粨鏋滆繑鍥炪傚ソ澶х殑鏁板瓧鍛錛岀獊鐒跺彂鐜版壘涓嶅埌鏄竴涓棶棰橈紝鎵懼埌鐨勫お澶氫篃鏄竴涓棶棰樸傚湪濡傛澶氱殑緇撴灉涓紝濡備綍灝嗘渶鐩稿叧鐨勬斁鍦ㄦ渶鍓嶉潰鍛紵
褰撶劧Google鍋氱殑寰堜笉閿欙紝鎮(zhèn)ㄤ竴涓嬪氨鎵懼埌浜唈obs at Microsoft銆傛兂璞′竴涓嬶紝濡傛灉鍓嶅嚑涓叏閮ㄦ槸“Microsoft does a good job at software industry…”灝嗘槸澶氫箞鍙曠殑浜嬫儏鍛銆?/p>
濡備綍鍍廏oogle涓鏍鳳紝鍦ㄦ垚鍗冧笂涓囩殑鎼滅儲緇撴灉涓紝鎵懼埌鍜屾煡璇㈣鍙ユ渶鐩稿叧鐨勫憿錛?/span>
濡備綍鍒ゆ柇鎼滅儲鍑虹殑鏂囨。鍜屾煡璇㈣鍙ョ殑鐩稿叧鎬у憿錛?/span>
榪欒鍥炲埌鎴戜滑絎笁涓棶棰橈細濡備綍瀵圭儲寮曡繘琛屾悳绱紵
鎼滅儲涓昏鍒嗕負浠ヤ笅鍑犳錛?/p>
1銆佺敤浜庤緭鍏ユ煡璇㈣鍙?/span>
鏌ヨ璇彞鍚屾垜浠櫘閫氱殑璇█涓鏍鳳紝涔熸槸鏈変竴瀹氳娉曠殑銆?/p>
涓嶅悓鐨勬煡璇㈣鍙ユ湁涓嶅悓鐨勮娉曪紝濡係QL璇彞灝辨湁涓瀹氱殑璇硶銆?/p>
鏌ヨ璇彞鐨勮娉曟牴鎹叏鏂囨绱㈢郴緇熺殑瀹炵幇鑰屼笉鍚屻傛渶鍩烘湰鐨勬湁姣斿錛欰ND, OR, NOT絳夈?/p>
涓句釜渚嬪瓙錛岀敤鎴瘋緭鍏ヨ鍙ワ細lucene AND learned NOT hadoop銆?/p>
璇存槑鐢ㄦ埛鎯蟲壘涓涓寘鍚玪ucene鍜宭earned鐒惰屼笉鍖呮嫭hadoop鐨勬枃妗ｃ?/p>
2銆佸鏌ヨ璇彞榪涜璇嶆硶鍒嗘瀽錛岃娉曞垎鏋愬強璇█澶勭悊
鐢變簬鏌ヨ璇彞鏈夎娉曪紝鍥犺屼篃瑕佽繘琛岃娉曞垎鏋愶紝璇硶鍒嗘瀽鍙婅璦澶勭悊銆?/p>
璇嶆硶鍒嗘瀽涓昏鐢ㄦ潵璇嗗埆鍗曡瘝鍜屽叧閿瓧銆?/strong>
濡備笂榪頒緥瀛愪腑錛岀粡榪囪瘝娉曞垎鏋愶紝寰楀埌鍗曡瘝鏈塴ucene錛宭earned錛宧adoop, 鍏抽敭瀛楁湁AND, NOT銆?/p>
濡傛灉鍦ㄨ瘝娉曞垎鏋愪腑鍙戠幇涓嶅悎娉曠殑鍏抽敭瀛楋紝鍒欎細鍑虹幇閿欒銆傚lucene AMD learned錛屽叾涓敱浜嶢ND鎷奸敊錛屽鑷碅MD浣滀負涓涓櫘閫氱殑鍗曡瘝鍙備笌鏌ヨ銆?/p>
璇硶鍒嗘瀽涓昏鏄牴鎹煡璇㈣鍙ョ殑璇硶瑙勫垯鏉ュ艦鎴愪竴媯佃娉曟爲銆?/strong>
濡傛灉鍙戠幇鏌ヨ璇彞涓嶆弧瓚寵娉曡鍒欙紝鍒欎細鎶ラ敊銆傚lucene NOT AND learned錛屽垯浼氬嚭閿欍?/p>
璇█澶勭悊鍚岀儲寮曡繃紼嬩腑鐨勮璦澶勭悊鍑犱箮鐩稿悓銆?/strong>
濡俵earned鍙樻垚learn絳夈?/p>
3銆佹悳绱㈢儲寮曪紝寰楀埌絎﹀悎璇硶鏍戠殑鏂囨。
姝ゆ楠ゆ湁鍒嗗嚑灝忔錛?/p>
棣栧厛錛屽湪鍙嶅悜绱㈠紩琛ㄤ腑錛屽垎鍒壘鍑哄寘鍚玪ucene錛宭earn錛宧adoop鐨勬枃妗ｉ摼琛ㄣ?strong>
鍏舵錛屽鍖呭惈lucene錛宭earn鐨勯摼琛ㄨ繘琛屽悎騫舵搷浣滐紝寰楀埌鏃㈠寘鍚玪ucene鍙堝寘鍚玪earn鐨勬枃妗ｉ摼琛ㄣ?strong>
鐒跺悗錛屽皢姝ら摼琛ㄤ笌hadoop鐨勬枃妗ｉ摼琛ㄨ繘琛屽樊鎿嶄綔錛屽幓闄ゅ寘鍚玥adoop鐨勬枃妗ｏ紝浠庤屽緱鍒版棦鍖呭惈lucene鍙堝寘鍚玪earn鑰屼笖涓嶅寘鍚玥adoop鐨勬枃妗ｉ摼琛ㄣ?strong>
鏈鍚庯紝姝ゆ枃妗ｉ摼琛ㄥ氨鏄垜浠鎵劇殑鏂囨。銆?
4銆佹牴鎹緱鍒扮殑鏂囨。鍜屾煡璇㈣鍙ョ殑鐩稿叧鎬э紝瀵圭粨鏋滆繘琛屾帓搴忋?/span>
铏界劧鍦ㄤ笂涓姝ワ紝鎴戜滑寰楀埌浜嗘兂瑕佺殑鏂囨。錛岀劧鑰屽浜庢煡璇㈢粨鏋滃簲璇ユ寜鐓т笌鏌ヨ璇彞鐨勭浉鍏蟲ц繘琛屾帓搴忥紝瓚婄浉鍏寵呰秺闈犲墠銆?/p>
濡備綍璁＄畻鏂囨。鍜屾煡璇㈣鍙ョ殑鐩稿叧鎬у憿錛?/span>
涓嶅鎴戜滑鎶婃煡璇㈣鍙ョ湅浣滀竴鐗囩煭灝忕殑鏂囨。錛屽鏂囨。涓庢枃妗ｄ箣闂寸殑鐩稿叧鎬?relevance)榪涜鎵撳垎(scoring)錛屽垎鏁伴珮鐨勭浉鍏蟲уソ錛屽氨搴旇鎺掑湪鍓嶉潰銆?/p>
閭ｄ箞鍙堟庝箞瀵規(guī)枃妗ｄ箣闂寸殑鍏崇郴榪涜鎵撳垎鍛紵
棣栧厛錛屼竴涓枃妗ｆ湁寰堝璇?Term)緇勬垚 錛?/span> 濡俿earch, lucene, full-text, this, a, what絳夈?/p>
鍏舵瀵逛簬鏂囨。涔嬮棿鐨勫叧緋伙紝涓嶅悓鐨凾erm閲嶈鎬т笉鍚?/strong> 錛?/span> 姣斿瀵逛簬鏈瘒鏂囨。錛宻earch, Lucene, full-text灝辯浉瀵歸噸瑕佷竴浜涳紝this, a , what鍙兘鐩稿涓嶉噸瑕佷竴浜涖傛墍浠ュ鏋滀袱綃囨枃妗ｉ兘鍖呭惈search, Lucene錛宖ulltext錛岃繖涓ょ瘒鏂囨。鐨勭浉鍏蟲уソ涓浜涳紝鐒惰屽氨綆椾竴綃囨枃妗ｅ寘鍚玹his, a, what錛屽彟涓綃囨枃妗ｄ笉鍖呭惈this, a, what錛屼篃涓嶈兘褰卞搷涓ょ瘒鏂囨。鐨勭浉鍏蟲с?/p>
鍥犺屽垽鏂枃妗ｄ箣闂寸殑鍏崇郴錛岄鍏堟壘鍑哄摢浜涜瘝(Term)瀵規(guī)枃妗ｄ箣闂寸殑鍏崇郴鏈閲嶈錛屽search, Lucene, fulltext銆傜劧鍚庡垽鏂繖浜涜瘝(Term)涔嬮棿鐨勫叧緋匯?/p>
鎵懼嚭璇?Term) 瀵規(guī)枃妗ｇ殑閲嶈鎬х殑榪囩▼縐頒負璁＄畻璇嶇殑鏉冮噸(Term weight) 鐨勮繃紼嬨?/strong> 璁＄畻璇嶇殑鏉冮噸(term weight)鏈変袱涓弬鏁幫紝絎竴涓槸璇?Term)錛岀浜屼釜鏄枃妗?Document)銆?/p>
璇嶇殑鏉冮噸(Term weight)琛ㄧず姝よ瘝(Term)鍦ㄦ鏂囨。涓殑閲嶈紼嬪害錛岃秺閲嶈鐨勮瘝(Term)鏈夎秺澶х殑鏉冮噸(Term weight)錛屽洜鑰屽湪璁＄畻鏂囨。涔嬮棿鐨勭浉鍏蟲т腑灝嗗彂鎸ユ洿澶х殑浣滅敤銆?/p>
鍒ゆ柇璇?Term) 涔嬮棿鐨勫叧緋諱粠鑰屽緱鍒版枃妗ｇ浉鍏蟲х殑榪囩▼搴旂敤涓縐嶅彨鍋氬悜閲忕┖闂存ā鍨嬬殑綆楁硶(Vector Space Model) 銆?/strong>
涓嬮潰浠旂粏鍒嗘瀽涓涓嬭繖涓や釜榪囩▼錛?/p>
褰卞搷涓涓瘝(Term)鍦ㄤ竴綃囨枃妗ｄ腑鐨勯噸瑕佹т富瑕佹湁涓や釜鍥犵礌錛?/p>
瀹規(guī)槗鐞嗚В鍚楋紵璇?Term)鍦ㄦ枃妗ｄ腑鍑虹幇鐨勬鏁拌秺澶氾紝璇存槑姝よ瘝(Term)瀵硅鏂囨。瓚婇噸瑕侊紝濡?#8220;鎼滅儲”榪欎釜璇嶏紝鍦ㄦ湰鏂囨。涓嚭鐜扮殑嬈℃暟寰堝錛岃鏄庢湰鏂囨。涓昏灝辨槸璁茶繖鏂歸潰鐨勪簨鐨勩傜劧鑰屽湪涓綃囪嫳璇枃妗ｄ腑錛宼his鍑虹幇鐨勬鏁版洿澶氾紝灝辮鏄庤秺閲嶈鍚楋紵涓嶆槸鐨勶紝榪欐槸鐢辯浜屼釜鍥犵礌榪涜璋冩暣錛岀浜屼釜鍥犵礌璇存槑錛屾湁瓚婂鐨勬枃妗ｅ寘鍚璇?Term), 璇存槑姝よ瘝(Term)澶櫘閫氾紝涓嶈凍浠ュ尯鍒嗚繖浜涙枃妗ｏ紝鍥犺岄噸瑕佹ц秺浣庛?/p>
榪欎篃濡傛垜浠▼搴忓憳鎵瀛︾殑鎶鏈紝瀵逛簬紼嬪簭鍛樻湰韜潵璇達紝榪欓」鎶鏈帉鎻¤秺娣辮秺濂斤紙鎺屾彙瓚婃繁璇存槑鑺辨椂闂寸湅鐨勮秺澶氾紝tf瓚婂ぇ錛夛紝鎵懼伐浣滄椂瓚婃湁绔炰簤鍔涖傜劧鑰屽浜庢墍鏈夌▼搴忓憳鏉ヨ錛岃繖欏規(guī)妧鏈噦寰楃殑浜鴻秺灝戣秺濂斤紙鎳傚緱鐨勪漢灝慸f灝忥級錛屾壘宸ヤ綔瓚婃湁绔炰簤鍔涖備漢鐨勪環(huán)鍊煎湪浜庝笉鍙浛浠ｆу氨鏄繖涓亾鐞嗐?/p>
閬撶悊鏄庣櫧浜嗭紝鎴戜滑鏉ョ湅鐪嬪叕寮忥細 榪欎粎浠呭彧term weight璁＄畻鍏紡鐨勭畝鍗曞吀鍨嬪疄鐜般傚疄鐜板叏鏂囨绱㈢郴緇熺殑浜轟細鏈夎嚜宸辯殑瀹炵幇錛孡ucene灝變笌姝ょ◢鏈変笉鍚屻?/p>
鎴戜滑鎶婃枃妗ｇ湅浣滀竴緋誨垪璇?Term)錛屾瘡涓涓瘝(Term)閮芥湁涓涓潈閲?Term weight)錛屼笉鍚岀殑璇?Term)鏍規(guī)嵁鑷繁鍦ㄦ枃妗ｄ腑鐨勬潈閲嶆潵褰卞搷鏂囨。鐩稿叧鎬х殑鎵撳垎璁＄畻銆?/p>
浜庢槸鎴戜滑鎶婃墍鏈夋鏂囨。涓瘝(term)鐨勬潈閲?term weight) 鐪嬩綔涓涓悜閲忋?/p>
Document = {term1, term2, …… ,term N} Document Vector = {weight1, weight2, …… ,weight N} 鍚屾牱鎴戜滑鎶婃煡璇㈣鍙ョ湅浣滀竴涓畝鍗曠殑鏂囨。錛屼篃鐢ㄥ悜閲忔潵琛ㄧず銆?/p>
Query = {term1, term 2, …… , term N} Query Vector = {weight1, weight2, …… , weight N} 鎴戜滑鎶婃墍鏈夋悳绱㈠嚭鐨勬枃妗ｅ悜閲忓強鏌ヨ鍚戦噺鏀懼埌涓涓狽緇寸┖闂翠腑錛屾瘡涓瘝(term)鏄竴緇淬?/p>
鎴戜滑璁や負涓や釜鍚戦噺涔嬮棿鐨勫す瑙掕秺灝忥紝鐩稿叧鎬ц秺澶с?/p>
鎵浠ユ垜浠綆楀す瑙掔殑浣欏雞鍊間綔涓虹浉鍏蟲х殑鎵撳垎錛屽す瑙掕秺灝忥紝浣欏雞鍊艱秺澶э紝鎵撳垎瓚婇珮錛岀浉鍏蟲ц秺澶с?/p>
鏈変漢鍙兘浼氶棶錛屾煡璇㈣鍙ヤ竴鑸槸寰堢煭鐨勶紝鍖呭惈鐨勮瘝(Term)鏄緢灝戠殑錛屽洜鑰屾煡璇㈠悜閲忕殑緇存暟寰堝皬錛岃屾枃妗ｅ緢闀匡紝鍖呭惈璇?Term)寰堝錛屾枃妗ｅ悜閲忕淮鏁板緢澶с備綘鐨勫浘涓袱鑰呯淮鏁版庝箞閮芥槸N鍛紵 鍦ㄨ繖閲岋紝鏃㈢劧瑕佹斁鍒扮浉鍚岀殑鍚戦噺絀洪棿錛岃嚜鐒剁淮鏁版槸鐩稿悓鐨勶紝涓嶅悓鏃訛紝鍙栦簩鑰呯殑騫墮泦錛屽鏋滀笉鍚煇涓瘝(Term)鏃訛紝鍒欐潈閲?Term Weight)涓?銆?/p>
鐩稿叧鎬ф墦鍒嗗叕寮忓涓嬶細 鏍規(guī)嵁榪欎釜鍏紡灝辮兘綆楀嚭鏂囨。涓庢煡璇箣闂寸殑鐩鎬技紼嬪害浜嗐?/p>
鍥涖佹葷粨 瀵逛笂榪扮儲寮曞垱寤哄拰鎼滅儲榪囩▼鎵涓涓葷粨錛屽鍥撅細 姝ゅ浘鍙傜収http://www.lucene.com.cn/about.htm 涓枃绔犮婂紑鏀炬簮浠ｇ爜鐨勫叏鏂囨绱㈠紩鎿嶭ucene銆?/p>
1. 绱㈠紩榪囩▼錛?/strong> a 鏈変竴緋誨垪琚儲寮曟枃浠?/strong> b) 琚儲寮曟枃浠剁粡榪囪娉曞垎鏋愬拰璇█澶勭悊褰㈡垚涓緋誨垪璇?Term) 銆?/strong> c) 緇忚繃绱㈠紩鍒涘緩褰㈡垚璇嶅吀鍜屽弽鍚戠儲寮曡〃銆?/strong> d) 閫氳繃绱㈠紩瀛樺偍灝嗙儲寮曞啓鍏ョ‖鐩樸?/strong> 2. 鎼滅儲榪囩▼錛?/strong> a) 鐢ㄦ埛杈撳叆鏌ヨ璇彞銆?/strong> b) 瀵規(guī)煡璇㈣鍙ョ粡榪囪娉曞垎鏋愬拰璇█鍒嗘瀽寰楀埌涓緋誨垪璇?Term) 銆?/strong> c) 閫氳繃璇硶鍒嗘瀽寰楀埌涓涓煡璇㈡爲銆?/strong> d) 閫氳繃绱㈠紩瀛樺偍灝嗙儲寮曡鍏ュ埌鍐呭瓨銆?/strong> e) 鍒╃敤鏌ヨ鏍戞悳绱㈢儲寮曪紝浠庤屽緱鍒版瘡涓瘝(Term) 鐨勬枃妗ｉ摼琛紝瀵規(guī)枃妗ｉ摼琛ㄨ繘琛屼氦錛屽樊錛屽茍寰楀埌緇撴灉鏂囨。銆?/strong> f) 灝嗘悳绱㈠埌鐨勭粨鏋滄枃妗ｅ鏌ヨ鐨勭浉鍏蟲ц繘琛屾帓搴忋?/strong> g) 榪斿洖鏌ヨ緇撴灉緇欑敤鎴楓?/strong> 1. 璁＄畻鏉冮噸(Term weight)鐨勮繃紼嬨?/strong>
2. 鍒ゆ柇Term涔嬮棿鐨勫叧緋諱粠鑰屽緱鍒版枃妗ｇ浉鍏蟲х殑榪囩▼錛屼篃鍗沖悜閲忕┖闂存ā鍨嬬殑綆楁硶(VSM)銆?/strong>
]]>