锘??xml version="1.0" encoding="utf-8" standalone="yes"?>国产亚洲真人做受在线观看,国产亚洲欧洲Aⅴ综合一区,亚洲第一视频在线观看免费http://www.tkk7.com/paulwong/category/54858.htmlzh-cnThu, 18 Jun 2015 05:28:33 GMTThu, 18 Jun 2015 05:28:33 GMT60Spark涓嶴hark鐨勫師鐞?/title><link>http://www.tkk7.com/paulwong/archive/2015/06/18/425773.html</link><dc:creator>paulwong</dc:creator><author>paulwong</author><pubDate>Thu, 18 Jun 2015 05:20:00 GMT</pubDate><guid>http://www.tkk7.com/paulwong/archive/2015/06/18/425773.html</guid><wfw:comment>http://www.tkk7.com/paulwong/comments/425773.html</wfw:comment><comments>http://www.tkk7.com/paulwong/archive/2015/06/18/425773.html#Feedback</comments><slash:comments>0</slash:comments><wfw:commentRss>http://www.tkk7.com/paulwong/comments/commentRss/425773.html</wfw:commentRss><trackback:ping>http://www.tkk7.com/paulwong/services/trackbacks/425773.html</trackback:ping><description><![CDATA[<p style="margin: 0px 0px 10px; padding: 0px; color: #616161; font-family: 'microsoft yahei'; line-height: 20px; background-color: #efefef;"><strong>1.Spark鐢熸佸湀</strong></p><p style="margin: 0px 0px 10px; padding: 0px; color: #616161; font-family: 'microsoft yahei'; line-height: 20px; background-color: #efefef;">濡備笅鍥炬墍紺轟負Spark鐨勬暣涓敓鎬佸湀錛屾渶搴曞眰涓鴻祫婧愮鐞嗗櫒錛岄噰鐢∕esos銆乊arn絳夎祫婧愮鐞嗛泦緹ゆ垨鑰匰park 鑷甫鐨凷tandalone妯″紡錛屽簳灞傚瓨鍌ㄤ負鏂囦歡緋葷粺鎴栬呭叾浠栨牸寮忕殑瀛樺偍緋葷粺濡侶Base銆係park浣滀負璁$畻妗嗘灦錛屼負涓婂眰澶氱搴旂敤鎻愪緵鏈嶅姟銆?Graphx鍜孧LBase鎻愪緵鏁版嵁鎸栨帢鏈嶅姟錛屽鍥捐綆楀拰鎸栨帢榪唬璁$畻絳夈係hark鎻愪緵SQL鏌ヨ鏈嶅姟錛屽吋瀹笻ive璇硶錛屾ц兘姣擧ive蹇?-50 鍊嶏紝BlinkDB鏄竴涓氳繃鏉冭 鏁版嵁綺劇‘搴︽潵鎻愬崌鏌ヨ鏅屽簲鏃墮棿鐨勪氦浜扴QL鏌ヨ寮曟搸錛屼簩鑰呴兘鍙綔涓轟氦浜掑紡鏌ヨ浣跨敤銆係park Streaming灝嗘祦寮忚綆楀垎瑙f垚涓緋誨垪鐭皬鐨勬壒澶勭悊璁$畻錛屽茍涓旀彁渚涢珮鍙潬鍜屽悶鍚愰噺鏈嶅姟銆?/p><p style="margin: 0px 0px 10px; padding: 0px; color: #616161; font-family: 'microsoft yahei'; line-height: 20px; background-color: #efefef;"><br /><img src="http://dl2.iteye.com/upload/attachment/0109/5585/a0237912-0cd8-3032-a9fe-cff365afb9e1.png" alt="" style="border: 0px;" /><br /> <strong>2.Spark鍩烘湰鍘熺悊</strong></p><p style="margin: 0px 0px 10px; padding: 0px; color: #616161; font-family: 'microsoft yahei'; line-height: 20px; background-color: #efefef;"><strong>Spark榪愯妗嗘灦濡備笅鍥炬墍紺猴紝棣栧厛鏈夐泦緹よ祫婧愮鐞嗘湇鍔★紙Cluster Manager錛夊拰榪愯浣滀笟浠誨姟鐨勭粨鐐癸紙Worker Node錛夛紝鐒跺悗灝辨槸姣忎釜搴旂敤鐨勪換鍔℃帶鍒剁粨鐐笵river鍜屾瘡涓満鍣ㄨ妭鐐逛笂鏈夊叿浣撲換鍔$殑鎵ц榪涚▼錛圗xecutor錛夈?/strong></p><p style="margin: 0px 0px 10px; padding: 0px; color: #616161; font-family: 'microsoft yahei'; line-height: 20px; background-color: #efefef;"><strong><br /><img src="http://dl2.iteye.com/upload/attachment/0109/5587/99853dc0-1bb5-323a-9f99-dea35197b965.png" alt="" style="border: 0px;" /><br /> </strong>涓嶮R璁$畻妗嗘灦鐩告瘮錛孍xecutor鏈変簩涓紭鐐癸細涓涓槸澶氱嚎紼嬫潵鎵ц鍏蜂綋鐨勪換鍔★紝鑰屼笉鏄儚MR閭f牱閲囩敤榪涚▼妯″瀷錛?鍑忓皯浜嗕換鍔$殑鍚姩寮紼嶃備簩涓槸Executor涓婁細鏈変竴涓狟lockManager瀛樺偍妯″潡錛岀被浼間簬KV緋葷粺錛堝唴瀛樺拰紓佺洏鍏卞悓浣滀負瀛樺偍璁懼錛夛紝褰撻渶瑕佽凱浠?澶氳疆鏃訛紝鍙互灝嗕腑闂磋繃紼嬬殑鏁版嵁鍏堟斁鍒拌繖涓瓨鍌ㄧ郴緇熶笂錛屼笅嬈¢渶瑕佹椂鐩存帴璇昏瀛樺偍涓婃暟鎹紝鑰屼笉闇瑕佽鍐欏埌hdfs絳夌浉鍏崇殑鏂囦歡緋葷粺閲岋紝鎴栬呭湪浜や簰寮忔煡璇㈠満鏅?涓嬶紝浜嬪厛灝嗚〃Cache鍒拌瀛樺偍緋葷粺涓婏紝鎻愰珮璇誨啓IO鎬ц兘銆傚彟澶朣park鍦ㄥ仛Shuffle鏃訛紝鍦℅roupby錛孞oin絳夊満鏅笅鍘繪帀浜嗕笉蹇呰鐨?Sort鎿嶄綔錛岀浉姣斾簬MapReduce鍙湁Map鍜孯educe浜岀妯″紡錛孲park榪樻彁渚涗簡鏇村姞涓板瘜鍏ㄩ潰鐨勮繍綆楁搷浣滃 filter,groupby,join絳夈?/p><p style="margin: 0px 0px 10px; padding: 0px; color: #616161; font-family: 'microsoft yahei'; line-height: 20px; background-color: #efefef;"> </p><p style="margin: 0px 0px 10px; padding: 0px; color: #616161; font-family: 'microsoft yahei'; line-height: 20px; background-color: #efefef;">Notes: 鍦ㄩ泦緹?cluster)鏂瑰紡涓? Cluster Manager榪愯鍦ㄤ竴涓猨vm榪涚▼涔嬩腑錛岃寃orker榪愯鍦ㄥ彟涓涓猨vm榪涚▼涓傚湪local cluster涓紝榪欎簺jvm榪涚▼閮藉湪鍚屼竴鍙版満鍣ㄤ腑錛屽鏋滄槸鐪熸鐨剆tandalone鎴朚esos鍙奩arn闆嗙兢錛寃orker涓巑aster鎴栧垎甯冧簬涓嶅悓鐨勪富鏈轟箣涓娿?/p><p style="margin: 0px 0px 10px; padding: 0px; color: #616161; font-family: 'microsoft yahei'; line-height: 20px; background-color: #efefef;"> </p><p style="margin: 0px 0px 10px; padding: 0px; color: #616161; font-family: 'microsoft yahei'; line-height: 20px; background-color: #efefef;">JOB鐨勭敓鎴愬拰榪愯</p><p style="margin: 0px 0px 10px; padding: 0px; color: #616161; font-family: 'microsoft yahei'; line-height: 20px; background-color: #efefef;">job鐢熸垚鐨勭畝鍗曟祦紼嬪涓?/p><p style="margin: 0px 0px 10px; padding: 0px; color: #616161; font-family: 'microsoft yahei'; line-height: 20px; background-color: #efefef;">1.棣栧厛搴旂敤紼嬪簭鍒涘緩SparkContext鐨勫疄渚嬶紝濡傚疄渚嬩負sc</p><p style="margin: 0px 0px 10px; padding: 0px; color: #616161; font-family: 'microsoft yahei'; line-height: 20px; background-color: #efefef;">2.鍒╃敤SparkContext鐨勫疄渚嬫潵鍒涘緩鐢熸垚RDD</p><p style="margin: 0px 0px 10px; padding: 0px; color: #616161; font-family: 'microsoft yahei'; line-height: 20px; background-color: #efefef;">3.緇忚繃涓榪炰覆鐨則ransformation鎿嶄綔錛屽師濮嬬殑RDD杞崲鎴愪負鍏跺畠綾誨瀷鐨凴DD</p><p style="margin: 0px 0px 10px; padding: 0px; color: #616161; font-family: 'microsoft yahei'; line-height: 20px; background-color: #efefef;">4.褰揳ction浣滅敤浜庤漿鎹箣鍚嶳DD鏃訛紝浼氳皟鐢⊿parkContext鐨剅unJob鏂規硶</p><p style="margin: 0px 0px 10px; padding: 0px; color: #616161; font-family: 'microsoft yahei'; line-height: 20px; background-color: #efefef;">5.sc.runJob鐨勮皟鐢ㄦ槸鍚庨潰涓榪炰覆鍙嶅簲鐨勮搗鐐癸紝鍏抽敭鎬х殑璺冨彉灝卞彂鐢熷湪姝ゅ</p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;">璋冪敤璺緞澶ц嚧濡備笅</p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;">1.sc.runJob->dagScheduler.runJob->submitJob</p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;">2.DAGScheduler::submitJob浼氬垱寤篔obSummitted鐨別vent鍙戦佺粰鍐呭祵綾籩ventProcessActor</p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;">3.eventProcessActor鍦ㄦ帴鏀跺埌JobSubmmitted涔嬪悗璋冪敤processEvent澶勭悊鍑芥暟</p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;">4.job鍒皊tage鐨勮漿鎹紝鐢熸垚finalStage騫舵彁浜よ繍琛岋紝鍏抽敭鏄皟鐢╯ubmitStage</p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;">5.鍦╯ubmitStage涓細璁$畻stage涔嬮棿鐨勪緷璧栧叧緋伙紝渚濊禆鍏崇郴鍒嗕負瀹戒緷璧栧拰紿勪緷璧栦袱縐?/p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;">6.濡傛灉璁$畻涓彂鐜板綋鍓嶇殑stage娌℃湁浠諱綍渚濊禆鎴栬呮墍鏈夌殑渚濊禆閮藉凡緇忓噯澶囧畬姣曪紝鍒欐彁浜ask</p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;">7.鎻愪氦task鏄皟鐢ㄥ嚱鏁皊ubmitMissingTasks鏉ュ畬鎴?/p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;">8.task鐪熸榪愯鍦ㄥ摢涓獁orker涓婇潰鏄敱TaskScheduler鏉ョ鐞嗭紝涔熷氨鏄笂闈㈢殑submitMissingTasks浼氳皟鐢═askScheduler::submitTasks</p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;">9.TaskSchedulerImpl涓細鏍規嵁Spark鐨勫綋鍓嶈繍琛屾ā寮忔潵鍒涘緩鐩稿簲鐨刡ackend,濡傛灉鏄湪鍗曟満榪愯鍒欏垱寤篖ocalBackend</p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;">10.LocalBackend鏀跺埌TaskSchedulerImpl浼犻掕繘鏉ョ殑ReceiveOffers浜嬩歡</p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;">11.receiveOffers->executor.launchTask->TaskRunner.run</p><p style="margin: 0px 0px 10px; padding: 0px; color: #616161; font-family: 'microsoft yahei'; line-height: 20px; background-color: #efefef;"> </p><p style="margin: 0px 0px 10px; padding: 0px; color: #616161; font-family: 'microsoft yahei'; line-height: 20px; background-color: #efefef;">Spark閲囩敤浜哠cala鏉ョ紪鍐欙紝鍦ㄥ嚱鏁拌〃杈句笂Scala鏈夊ぉ鐒剁殑浼樺娍錛屽洜姝ゅ湪琛ㄨ揪澶嶆潅鐨勬満鍣ㄥ涔犵畻娉曡兘鍔涙瘮鍏朵粬 璇█鏇村己涓旂畝鍗曟槗鎳傘傛彁渚涘悇縐嶆搷浣滃嚱鏁版潵寤虹珛璧稲DD鐨凞AG璁$畻妯″瀷銆傛妸姣忎竴涓搷浣滈兘鐪嬫垚鏋勫緩涓涓猂DD鏉ュ寰咃紝鑰孯DD鍒欒〃紺虹殑鏄垎甯冨湪澶氬彴鏈哄櫒涓婄殑 鏁版嵁闆嗗悎錛屽茍涓斿彲浠ュ甫涓婂悇縐嶆搷浣滃嚱鏁般傚涓嬪浘鎵紺猴細</p><p style="margin: 0px 0px 10px; padding: 0px; color: #616161; font-family: 'microsoft yahei'; line-height: 20px; background-color: #efefef;"><br /><img src="http://dl2.iteye.com/upload/attachment/0109/5589/76cf1a9e-5b47-33e8-b265-a5ca97b743d7.png" alt="" style="border: 0px;" /><br /> 棣栧厛浠巋dfs鏂囦歡閲岃鍙栨枃鏈唴瀹規瀯寤烘垚涓涓猂DD錛岀劧鍚庝嬌鐢╢ilter錛堬級鎿嶄綔鏉ュ涓婃鐨凴DD榪涜榪囨護錛屽啀浣?鐢╩ap錛堬級鎿嶄綔鍙栧緱璁板綍鐨勭涓涓瓧孌碉紝鏈鍚庡皢鍏禼ache鍦ㄥ唴瀛樹笂錛屽悗闈㈠氨鍙互瀵逛箣鍓峜ache榪囩殑鏁版嵁鍋氬叾浠栫殑鎿嶄綔銆傛暣涓繃紼嬮兘灝嗗艦鎴愪竴涓狣AG璁$畻 鍥撅紝姣忎釜鎿嶄綔姝ラ閮芥湁瀹歸敊鏈哄埗錛屽悓鏃惰繕鍙互灝嗛渶瑕佸嬈′嬌鐢ㄧ殑鏁版嵁cache璧鋒潵錛屼緵鍚庣畫榪唬浣跨敤.</p><p style="margin: 0px 0px 10px; padding: 0px; color: #616161; font-family: 'microsoft yahei'; line-height: 20px; background-color: #efefef;"> </p><p style="margin: 0px 0px 10px; padding: 0px; color: #616161; font-family: 'microsoft yahei'; line-height: 20px; background-color: #efefef;"><strong>3.Shark鐨勫伐浣滃師鐞?/strong></p><p style="margin: 0px 0px 10px; padding: 0px; color: #616161; font-family: 'microsoft yahei'; line-height: 20px; background-color: #efefef;">Shark鏄熀浜嶴park璁$畻妗嗘灦涔嬩笂涓斿吋瀹笻ive璇硶鐨凷QL鎵ц寮曟搸錛岀敱浜庡簳灞傜殑璁$畻閲囩敤浜哠park錛屾?鑳芥瘮MapReduce鐨凥ive鏅亶蹇?鍊嶄互涓婏紝濡傛灉鏄函鍐呭瓨璁$畻鐨凷QL錛岃蹇?鍊嶄互涓婏紝褰撴暟鎹叏閮╨oad鍦ㄥ唴瀛樼殑璇濓紝灝嗗揩10鍊嶄互涓婏紝鍥犳 Shark鍙互浣滀負浜や簰寮忔煡璇㈠簲鐢ㄦ湇鍔℃潵浣跨敤銆?br /><img src="http://dl2.iteye.com/upload/attachment/0109/5591/3c1dd765-73b0-3c56-8abb-38a0d3ddb915.png" alt="" style="border: 0px;" /><br /> 涓婂浘灝辨槸鏁翠釜Shark鐨勬鏋跺浘錛屼笌鍏朵粬鐨凷QL寮曟搸鐩告瘮錛岄櫎浜嗗熀浜嶴park鐨勭壒鎬у錛孲hark鏄畬鍏ㄥ吋瀹笻ive鐨勮娉曪紝琛ㄧ粨鏋勪互鍙奤DF鍑芥暟絳夛紝宸叉湁鐨凥iveSql鍙互鐩存帴榪涜榪佺Щ鑷砈hark涓娿?/p><p style="margin: 0px 0px 10px; padding: 0px; color: #616161; font-family: 'microsoft yahei'; line-height: 20px; background-color: #efefef;"><br /><strong>涓嶩ive鐩告瘮錛孲hark鐨勭壒鎬у涓嬶細</strong></p><p style="margin: 0px 0px 10px; padding: 0px; color: #616161; font-family: 'microsoft yahei'; line-height: 20px; background-color: #efefef;">1.浠ュ湪綰挎湇鍔$殑鏂瑰紡鎵ц浠誨姟錛岄伩鍏嶄換鍔¤繘紼嬬殑鍚姩鍜岄攢姣佸紑紼嶏紝閫氬父MapReduce閲岀殑姣忎釜浠誨姟閮芥槸鍚姩鍜屽叧闂繘紼嬬殑鏂瑰紡鏉ヨ繍琛岀殑錛岃屽湪Shark涓紝Server榪愯鍚庯紝鎵鏈夌殑宸ヤ綔鑺傜偣涔熼殢涔嬪惎鍔紝闅忓悗浠ュ父椹繪湇鍔$殑褰㈠紡涓嶆柇鐨勬帴鍙桽erver鍙戞潵鐨勪換鍔°?/p><p style="margin: 0px 0px 10px; padding: 0px; color: #616161; font-family: 'microsoft yahei'; line-height: 20px; background-color: #efefef;">2.Groupby鍜孞oin鎿嶄綔涓嶉渶瑕丼ort宸ヤ綔錛屽綋鏁版嵁閲忓唴瀛樿兘瑁呬笅鏃訛紝涓杈規帴鏀舵暟鎹竴杈規墽琛岃綆楁搷浣溿傚湪Hive涓紝涓嶇浠諱綍鎿嶄綔鍦∕ap鍒癛educe鐨勮繃紼嬮兘闇瑕佸Key榪涜Sort鎿嶄綔銆?/p><p style="margin: 0px 0px 10px; padding: 0px; color: #616161; font-family: 'microsoft yahei'; line-height: 20px; background-color: #efefef;">3.瀵逛簬鎬ц兘瑕佹眰鏇撮珮鐨勮〃錛屾彁渚涘垎甯冨紡Cache緋葷粺灝嗚〃鏁版嵁浜嬪厛Cache鑷沖唴瀛樹腑錛屽悗緇殑鏌ヨ灝嗙洿鎺ヨ闂唴瀛樻暟鎹紝涓嶅啀闇瑕佺鐩樺紑紼嶃?/p><p style="margin: 0px 0px 10px; padding: 0px; color: #616161; font-family: 'microsoft yahei'; line-height: 20px; background-color: #efefef;">4.榪樻湁寰堝Spark鐨勭壒鎬э紝濡傚彲浠ラ噰鐢═orrent鏉ュ箍鎾彉閲忓拰灝忔暟鎹紝灝嗘墽琛岃鍒掔洿鎺ヤ紶閫佺粰Task錛孌AG榪囩▼涓殑涓棿鏁版嵁涓嶉渶瑕佽惤鍦板埌Hdfs鏂囦歡緋葷粺銆?/p><img src ="http://www.tkk7.com/paulwong/aggbug/425773.html" width = "1" height = "1" /><br><br><div align=right><a style="text-decoration:none;" href="http://www.tkk7.com/paulwong/" target="_blank">paulwong</a> 2015-06-18 13:20 <a href="http://www.tkk7.com/paulwong/archive/2015/06/18/425773.html#Feedback" target="_blank" style="text-decoration:none;">鍙戣〃璇勮</a></div>]]></description></item><item><title>SPARK鏋舵瀯涓庢祦紼?/title><link>http://www.tkk7.com/paulwong/archive/2015/06/18/425772.html</link><dc:creator>paulwong</dc:creator><author>paulwong</author><pubDate>Thu, 18 Jun 2015 05:17:00 GMT</pubDate><guid>http://www.tkk7.com/paulwong/archive/2015/06/18/425772.html</guid><wfw:comment>http://www.tkk7.com/paulwong/comments/425772.html</wfw:comment><comments>http://www.tkk7.com/paulwong/archive/2015/06/18/425772.html#Feedback</comments><slash:comments>0</slash:comments><wfw:commentRss>http://www.tkk7.com/paulwong/comments/commentRss/425772.html</wfw:commentRss><trackback:ping>http://www.tkk7.com/paulwong/services/trackbacks/425772.html</trackback:ping><description><![CDATA[<p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;">Spark鐨勬暣浣撴祦紼嬩負錛欳lient 鎻愪氦搴旂敤錛孧aster鎵懼埌涓涓猈orker鍚姩Driver錛孌river鍚慚aster鎴栬呰祫婧愮鐞嗗櫒鐢寵璧勬簮錛屼箣鍚庡皢搴旂敤杞寲涓篟DD Graph錛屽啀鐢盌AGScheduler灝哛DD Graph杞寲涓篠tage鐨勬湁鍚戞棤鐜浘鎻愪氦緇橳askScheduler錛岀敱TaskScheduler鎻愪氦浠誨姟緇橢xecutor鎵ц銆傚湪浠誨姟鎵ц鐨勮繃紼嬩腑錛屽叾浠栫粍浠跺崗鍚屽伐浣滐紝紜繚鏁翠釜搴旂敤欏哄埄鎵ц銆?/p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;">Spark鏋舵瀯閲囩敤浜嗗垎甯冨紡璁$畻涓殑Master-Slave妯″瀷銆侻aster鏄搴旈泦緹や腑鐨勫惈鏈塎aster榪涚▼鐨勮妭鐐癸紙ClusterManager錛夛紝Slave鏄泦緹や腑鍚湁Worker榪涚▼鐨勮妭鐐廣侻aster浣滀負鏁翠釜闆嗙兢鐨勬帶鍒跺櫒錛岃礋璐f暣涓泦緹ょ殑姝e父榪愯錛沇orker鐩稿綋浜庢槸璁$畻鑺傜偣錛屾帴鏀朵富鑺傜偣鍛戒護涓庤繘琛岀姸鎬佹眹鎶ワ紱Executor璐熻矗浠誨姟鐨勬墽琛岋紱Client浣滀負鐢ㄦ埛鐨勫鎴風璐熻矗鎻愪氦搴旂敤錛孌river璐熻矗鎺у埗涓涓簲鐢ㄧ殑鎵ц錛屽鍥句笅鍥撅細</p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;"><br /><img alt="" src="http://dl2.iteye.com/upload/attachment/0109/5772/08f9d15b-10ea-3486-b6c9-a65809e27b0b.png" title="鐐瑰嚮鏌ョ湅鍘熷澶у皬鍥劇墖" width="760" height="401" style="border: 0px; cursor: url(http://www.iteye.com/images/magplus.gif), pointer;" /><br />                                                                                 Spark 妗嗘灦鍥?/p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;"> </p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;">Spark闆嗙兢閮ㄧ講鍚庯紝闇瑕佸湪涓昏妭鐐瑰拰浠庤妭鐐瑰垎鍒惎鍔∕aster榪涚▼鍜學orker榪涚▼錛屽鏁翠釜闆嗙兢榪涜鎺у埗銆傚湪涓涓猄park搴旂敤鐨勬墽琛岃繃紼嬩腑錛孌river鍜學orker鏄袱涓噸瑕佽鑹層侱river 紼嬪簭鏄簲鐢ㄩ昏緫鎵ц鐨勮搗鐐癸紝璐熻矗浣滀笟鐨勮皟搴︼紝鍗砊ask浠誨姟鐨勫垎鍙戯紝鑰屽涓猈orker鐢ㄦ潵綆$悊璁$畻鑺傜偣鍜屽垱寤篍xecutor騫惰澶勭悊浠誨姟銆傚湪鎵ц闃舵錛孌river浼氬皢Task鍜孴ask鎵渚濊禆鐨刦ile鍜宩ar搴忓垪鍖栧悗浼犻掔粰瀵瑰簲鐨刉orker鏈哄櫒錛屽悓鏃禘xecutor瀵圭浉搴旀暟鎹垎鍖虹殑浠誨姟榪涜澶勭悊銆?/p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;"> </p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;"><strong>Spark鐨勬灦鏋勪腑鐨勫熀鏈粍浠朵粙緇嶏細</strong></p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;"><strong>ClusterManager</strong>錛氬湪Standalone妯″紡涓嵆涓篗aster錛堜富鑺傜偣錛夛紝鎺у埗鏁翠釜闆嗙兢錛岀洃鎺orker銆傚湪YARN妯″紡涓負璧勬簮綆$悊鍣ㄣ?/p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;"><strong>Worker</strong>錛氫粠鑺傜偣錛岃礋璐f帶鍒惰綆楄妭鐐癸紝鍚姩Executor鎴朌river銆傚湪YARN妯″紡涓負NodeManager錛岃礋璐h綆楄妭鐐圭殑鎺у埗銆?/p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;"><strong>Driver</strong>錛氳繍琛孉pplication鐨刴ain()鍑芥暟騫跺垱寤篠parkContext銆?/p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;"><strong>Executor</strong>錛氭墽琛屽櫒錛屽湪worker node涓婃墽琛屼換鍔$殑緇勪歡銆佺敤浜庡惎鍔ㄧ嚎紼嬫睜榪愯浠誨姟銆傛瘡涓狝pplication鎷ユ湁鐙珛鐨勪竴緇凟xecutors銆?/p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;"><strong>SparkContext</strong>錛氭暣涓簲鐢ㄧ殑涓婁笅鏂囷紝鎺у埗搴旂敤鐨勭敓鍛藉懆鏈熴?/p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;"><strong>RDD</strong>錛歋park鐨勫熀鏈綆楀崟鍏冿紝涓緇凴DD鍙艦鎴愭墽琛岀殑鏈夊悜鏃犵幆鍥綬DD Graph銆?/p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;"><strong>DAG Scheduler</strong>錛氬疄鐜板皢Spark浣滀笟鍒嗚В鎴愪竴鍒板涓猄tage錛屾瘡涓猄tage鏍規嵁RDD鐨凱artition涓暟鍐沖畾Task鐨勪釜鏁幫紝鐒跺悗鐢熸垚鐩稿簲鐨凾ask set鏀懼埌TaskScheduler涓?/p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;"><strong>TaskScheduler</strong>錛氬皢浠誨姟錛圱ask錛夊垎鍙戠粰Executor鎵ц銆?/p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;"><strong>Stage</strong>錛氫竴涓猄park浣滀笟涓鑸寘鍚竴鍒板涓猄tage銆?/p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;"><strong>Task</strong>錛氫竴涓猄tage鍖呭惈涓鍒板涓猅ask錛岄氳繃澶氫釜Task瀹炵幇騫惰榪愯鐨勫姛鑳姐?/p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;"><strong>Transformations</strong>錛氳漿鎹?Transformations) (濡傦細map, filter, groupBy, join絳?錛孴ransformations鎿嶄綔鏄疞azy鐨勶紝涔熷氨鏄浠庝竴涓猂DD杞崲鐢熸垚鍙︿竴涓猂DD鐨勬搷浣滀笉鏄┈涓婃墽琛岋紝Spark鍦ㄩ亣鍒癟ransformations鎿嶄綔鏃跺彧浼氳褰曢渶瑕佽繖鏍風殑鎿嶄綔錛屽茍涓嶄細鍘繪墽琛岋紝闇瑕佺瓑鍒版湁Actions鎿嶄綔鐨勬椂鍊欐墠浼氱湡姝e惎鍔ㄨ綆楄繃紼嬭繘琛岃綆椼?/p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;"><strong>Actions</strong>錛氭搷浣?Actions) (濡傦細count, collect, save絳?錛孉ctions鎿嶄綔浼氳繑鍥炵粨鏋滄垨鎶奟DD鏁版嵁鍐欏埌瀛樺偍緋葷粺涓侫ctions鏄Е鍙慡park鍚姩璁$畻鐨勫姩鍥犮?/p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;"><strong>SparkEnv</strong>錛氱嚎紼嬬駭鍒殑涓婁笅鏂囷紝瀛樺偍榪愯鏃剁殑閲嶈緇勪歡鐨勫紩鐢ㄣ?/p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;"><em>SparkEnv鍐呭垱寤哄茍鍖呭惈濡備笅涓浜涢噸瑕佺粍浠剁殑寮曠敤銆?/em></p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;"><em>MapOutPutTracker錛氳礋璐huffle鍏冧俊鎭殑瀛樺偍銆?/em></p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;"><em>BroadcastManager錛氳礋璐e箍鎾彉閲忕殑鎺у埗涓庡厓淇℃伅鐨勫瓨鍌ㄣ?/em></p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;"><em>BlockManager錛氳礋璐e瓨鍌ㄧ鐞嗐佸垱寤哄拰鏌ユ壘鍧椼?/em></p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;"><em>MetricsSystem錛氱洃鎺ц繍琛屾椂鎬ц兘鎸囨爣淇℃伅銆?/em></p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;"><em>SparkConf錛氳礋璐e瓨鍌ㄩ厤緗俊鎭?/em></p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;"> </p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;"><br /><img alt="" src="http://dl2.iteye.com/upload/attachment/0109/5774/5fce0961-1030-3805-b649-acafae85170b.png" style="border: 0px;" /><br />                                                                       Spark榪愯閫昏緫鍥?/p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;"> </p><p style="margin: 0px; padding: 0px; font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2000007629395px; background-color: #efefef;">鍦⊿park搴旂敤涓紝鏁翠釜鎵ц嫻佺▼鍦ㄩ昏緫涓婁細褰㈡垚鏈夊悜鏃犵幆鍥撅紙DAG錛夈侫ction綆楀瓙瑙﹀彂涔嬪悗錛屽皢鎵鏈夌瘡縐殑綆楀瓙褰㈡垚涓涓湁鍚戞棤鐜浘錛岀劧鍚庣敱璋冨害鍣ㄨ皟搴﹁鍥句笂鐨勪換鍔¤繘琛岃繍綆椼係park鐨勮皟搴︽柟寮忎笌MapReduce鏈夋墍涓嶅悓銆係park鏍規嵁RDD涔嬮棿涓嶅悓鐨勪緷璧栧叧緋誨垏鍒嗗艦鎴愪笉鍚岀殑闃舵錛圫tage錛夛紝涓涓樁孌靛寘鍚竴緋誨垪鍑芥暟鎵ц嫻佹按綰褲傚浘涓殑A銆丅銆丆銆丏銆丒銆丗鍒嗗埆浠h〃涓嶅悓鐨凴DD錛孯DD鍐呯殑鏂規浠h〃鍒嗗尯銆傛暟鎹粠HDFS杈撳叆Spark錛屽艦鎴怰DD A鍜孯DD C錛孯DD C涓婃墽琛宮ap鎿嶄綔錛岃漿鎹負RDD D錛?RDD B鍜?RDD E鎵цjoin鎿嶄綔錛岃漿鎹負F錛岃屽湪B鍜孍榪炴帴杞寲涓篎鐨勮繃紼嬩腑鍙堜細鎵цShuffle錛屾渶鍚嶳DD F 閫氳繃鍑芥暟saveAsSequenceFile杈撳嚭騫朵繚瀛樺埌HDFS鎴?Hbase涓?/p><img src ="http://www.tkk7.com/paulwong/aggbug/425772.html" width = "1" height = "1" /><br><br><div align=right><a style="text-decoration:none;" href="http://www.tkk7.com/paulwong/" target="_blank">paulwong</a> 2015-06-18 13:17 <a href="http://www.tkk7.com/paulwong/archive/2015/06/18/425772.html#Feedback" target="_blank" style="text-decoration:none;">鍙戣〃璇勮</a></div>]]></description></item></channel></rss> <footer> <div class="friendship-link"> <p>感谢您访问我们的网站,您可能还对以下资源感兴趣:</p> <a href="http://www.tkk7.com/" title="亚洲av成人片在线观看">亚洲av成人片在线观看</a> <div class="friend-links"> </div> </div> </footer> 主站蜘蛛池模板: <a href="http://yinyinai155.com" target="_blank">亚洲欧美国产精品专区久久</a>| <a href="http://888-28.com" target="_blank">亚洲美女视频网站</a>| <a href="http://ttooyuyu.com" target="_blank">免费国产va在线观看</a>| <a href="http://langse3.com" target="_blank">免费无码一区二区三区蜜桃大 </a>| <a href="http://selangcun.com" target="_blank">国产色无码精品视频免费</a>| <a href="http://www50884.com" target="_blank">亚洲国产精品尤物YW在线观看</a>| <a href="http://96j6.com" target="_blank">老司机午夜免费视频</a>| <a href="http://xseporn.com" target="_blank">亚洲AV无码一区二三区 </a>| <a href="http://cztshw.com" target="_blank">国产午夜免费福利红片</a>| <a href="http://770144.com" target="_blank">亚洲6080yy久久无码产自国产</a>| <a href="http://8888kkk.com" target="_blank">日韩毛片免费在线观看</a>| <a href="http://580jp.com" target="_blank">一级毛片正片免费视频手机看</a>| <a href="http://929119.com" target="_blank">亚洲毛片网址在线观看中文字幕 </a>| <a href="http://twtairun.com" target="_blank">免费精品久久久久久中文字幕 </a>| <a href="http://8884493.com" target="_blank">亚洲日韩在线视频</a>| <a href="http://930mk.com" target="_blank">成人片黄网站色大片免费</a>| <a href="http://jst-hosp.com" target="_blank">亚洲成a∨人片在无码2023</a>| <a href="http://bxd888.com" target="_blank">亚洲AV中文无码乱人伦</a>| <a href="http://mysb123.com" target="_blank">伊人久久大香线蕉免费视频</a>| <a href="http://828556.com" target="_blank">77777_亚洲午夜久久多人</a>| <a href="http://cc88899.com" target="_blank">日韩视频在线精品视频免费观看</a>| <a href="http://caobi97.com" target="_blank">亚洲永久网址在线观看</a>| <a href="http://01shanzhai.com" target="_blank">日产国产精品亚洲系列</a>| <a href="http://ninggelang.com" target="_blank">中国一级特黄高清免费的大片中国一级黄色片 </a>| <a href="http://sswg2.com" target="_blank">日韩在线一区二区三区免费视频</a>| <a href="http://hbqueena.com" target="_blank">国产av无码专区亚洲国产精品</a>| <a href="http://khushkhush.com" target="_blank">APP在线免费观看视频</a>| <a href="http://367316.com" target="_blank">亚洲AV一二三区成人影片</a>| <a href="http://ww11axax.com" target="_blank">四虎永久成人免费</a>| <a href="http://8568008.com" target="_blank">免费看男人j放进女人j免费看</a>| <a href="http://470577.com" target="_blank">亚洲性猛交xx乱</a>| <a href="http://tuopumao.com" target="_blank">无码欧精品亚洲日韩一区夜夜嗨</a>| <a href="http://2886868.com" target="_blank">AAA日本高清在线播放免费观看</a>| <a href="http://liangdy.com" target="_blank">亚洲一级在线观看</a>| <a href="http://www6yg6yg.com" target="_blank">亚洲一区二区精品视频</a>| <a href="http://aaaaa123.com" target="_blank">99久久99热精品免费观看国产</a>| <a href="http://hqshimo.com" target="_blank">亚洲另类自拍丝袜第五页</a>| <a href="http://zaideqin.com" target="_blank">国产亚洲美女精品久久久久狼</a>| <a href="http://42329c.com" target="_blank">四虎国产精品免费久久</a>| <a href="http://295301.com" target="_blank">插鸡网站在线播放免费观看</a>| <a href="http://bohaolg.com" target="_blank">亚洲一级毛片中文字幕</a>| <script> (function(){ var bp = document.createElement('script'); var curProtocol = window.location.protocol.split(':')[0]; if (curProtocol === 'https') { bp.src = 'https://zz.bdstatic.com/linksubmit/push.js'; } else { bp.src = 'http://push.zhanzhang.baidu.com/push.js'; } var s = document.getElementsByTagName("script")[0]; s.parentNode.insertBefore(bp, s); })(); </script> </body>