锘??xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:trackback="http://madskills.com/public/xml/rss/module/trackback/" xmlns:wfw="http://wellformedweb.org/CommentAPI/" xmlns:slash="http://purl.org/rss/1.0/modules/slash/"><channel><title>&#22269;&#20135;&#20081;&#36744;&#36890;&#20262;&#24433;&#29255;&#22312;&#32447;&#25773;&#25918;&#20122;&#27954;&#13;&#44;&#20122;&#27954;&#65;&#86;&#26080;&#30721;&#26085;&#38889;&#65;&#86;&#26080;&#30721;&#23548;&#33322;&#44;&#20122;&#27954;&#35270;&#39057;&#19968;&#21306;&#35843;&#25945;</title><link>http://www.tkk7.com/wangxinsh55/category/55043.html</link><description /><language>zh-cn</language><lastBuildDate>Thu, 26 May 2016 06:17:05 GMT</lastBuildDate><pubDate>Thu, 26 May 2016 06:17:05 GMT</pubDate><ttl>60</ttl><item><title>Spark History Server閰嶇疆浣跨敤</title><link>http://www.tkk7.com/wangxinsh55/archive/2016/05/26/430665.html</link><dc:creator>SIMONE</dc:creator><author>SIMONE</author><pubDate>Thu, 26 May 2016 06:12:00 GMT</pubDate><guid>http://www.tkk7.com/wangxinsh55/archive/2016/05/26/430665.html</guid><wfw:comment>http://www.tkk7.com/wangxinsh55/comments/430665.html</wfw:comment><comments>http://www.tkk7.com/wangxinsh55/archive/2016/05/26/430665.html#Feedback</comments><slash:comments>0</slash:comments><wfw:commentRss>http://www.tkk7.com/wangxinsh55/comments/commentRss/430665.html</wfw:commentRss><trackback:ping>http://www.tkk7.com/wangxinsh55/services/trackbacks/430665.html</trackback:ping><description><![CDATA[<div>http://www.cnblogs.com/luogankun/p/3981645.html</div><br /><div><div   class="z97djdx"   id="cnblogs_post_body"><p><strong><span style="font-size: 14pt;">Spark history Server浜х敓鑳屾櫙</span></strong></p> <p>浠tandalone榪愯妯″紡涓轟緥錛屽湪榪愯Spark  Application鐨勬椂鍊欙紝Spark浼氭彁渚涗竴涓猈EBUI鍒楀嚭搴旂敤紼嬪簭鐨勮繍琛屾椂淇℃伅錛涗絾璇EBUI闅忕潃Application鐨勫畬鎴?鎴愬姛/澶?璐?鑰屽叧闂紝涔熷氨鏄錛孲park Application榪愯瀹?鎴愬姛/澶辮觸)鍚庯紝灝嗘棤娉曟煡鐪婣pplication鐨勫巻鍙茶褰曪紱</p> <p>Spark history  Server灝辨槸涓轟簡搴斿榪欑鎯呭喌鑰屼駭鐢熺殑錛岄�氳繃閰嶇疆鍙互鍦ˋpplication鎵ц鐨勮繃紼嬩腑璁板綍涓嬩簡鏃ュ織浜嬩歡淇℃伅錛岄偅涔堝湪Application鎵ц 緇撴潫鍚庯紝WEBUI灝辮兘閲嶆柊娓叉煋鐢熸垚UI鐣岄潰灞曠幇鍑鴻Application鍦ㄦ墽琛岃繃紼嬩腑鐨勮繍琛屾椂淇℃伅錛?/p> <p>Spark榪愯鍦▂arn鎴栬�卪esos涔嬩笂錛岄�氳繃spark鐨刪istory server浠嶇劧鍙互閲嶆瀯鍑轟竴涓凡緇忓畬鎴愮殑Application鐨勮繍琛屾椂鍙傛暟淇℃伅錛堝亣濡侫pplication榪愯鐨勪簨浠舵棩蹇椾俊鎭凡緇忚褰曚笅鏉ワ級錛?/p> <p>&nbsp;</p> <p><strong><span style="font-size: 14pt;">閰嶇疆&amp;浣跨敤Spark History Server</span></strong></p> <p>浠ラ粯璁ら厤緗殑鏂瑰紡鍚姩spark history server錛?/p> <div> <pre>cd $SPARK_HOME/<span style="color: #000000;">sbin <span style="color: #ff0000;"><strong>start</strong></span></span><span style="color: #ff0000;"><strong>-history-server.sh</strong></span></pre> </div> <p>鎶ラ敊錛?/p> <div> <pre>starting org.apache.spark.deploy.history.HistoryServer, logging to /home/spark/software/source/compile/deploy_spark/sbin/../logs/spark-spark-org.apache.spark.deploy.history.HistoryServer-<span style="color: #800080;">1</span>-<span style="color: #000000;">hadoop000.out failed to launch org.apache.spark.deploy.history.HistoryServer:         at org.apache.spark.deploy.history.FsHistoryProvider.</span>&lt;init&gt;(FsHistoryProvider.scala:<span style="color: #800080;">44</span><span style="color: #000000;">)         ... </span><span style="color: #800080;">6</span> <span style="color: #0000ff;">more</span></pre> </div> <p>闇�瑕佸湪鍚姩鏃舵寚瀹氱洰褰曪細</p> <div> <pre><span style="color: #ff0000;"><strong>start-history-server.sh</strong></span> <span style="color: #0000ff;"><strong>hdfs://hadoop000:8020/directory</strong></span></pre> </div> <p>hdfs://hadoop000:8020/directory鍙互閰嶇疆鍦ㄩ厤緗枃浠朵腑錛岄偅涔堝湪鍚姩history-server鏃跺氨涓嶉渶瑕佹寚瀹氾紝鍚庣畫浠嬬粛鎬庝箞閰嶇疆錛?/p> <p><span style="color: #ff0000;"><strong>娉細璇ョ洰褰曢渶瑕佷簨鍏堝湪hdfs涓婂垱寤哄ソ錛屽惁鍒檋istory-server鍚姩鎶ラ敊銆?/strong></span></p> <p>鍚姩瀹屾垚涔嬪悗鍙互閫氳繃WEBUI璁塊棶錛岄粯璁ょ鍙ｆ槸18080錛歨ttp://hadoop000:18080</p> <p>榛樿鐣岄潰鍒楄〃淇℃伅鏄┖鐨勶紝涓嬮潰鎴浘鏄垜璺戜簡鍑犳spark-sql嫻嬭瘯鍚庡嚭鐜扮殑銆?/p> <p><img src="http://images.cnitblog.com/blog/635295/201409/191450158003642.png" alt="" height="337" width="962" /></p> <p>&nbsp;</p> <p><strong>history server鐩稿叧鐨勯厤緗弬鏁版弿榪?/strong></p> <p>1錛?spark.history.updateInterval<br />	銆�銆�榛樿鍊鹼細10<br />	銆�銆�浠ョ涓哄崟浣嶏紝鏇存柊鏃ュ織鐩稿叧淇℃伅鐨勬椂闂撮棿闅?/p> <p>2錛塻park.history.retainedApplications<br />	銆�銆�榛樿鍊鹼細50<br />	銆�銆�鍦ㄥ唴瀛樹腑淇濆瓨Application鍘嗗彶璁板綍鐨勪釜鏁幫紝濡傛灉瓚呰繃榪欎釜鍊鹼紝鏃х殑搴旂敤紼嬪簭淇℃伅灝嗚鍒犻櫎錛屽綋鍐嶆璁塊棶宸茶鍒犻櫎鐨勫簲鐢ㄤ俊鎭椂闇�瑕侀噸鏂版瀯寤洪〉闈€�?/p> <p>3錛?span style="color: #ff0000;"><strong>spark.history.ui.port</strong></span><br />	銆�銆�榛樿鍊鹼細18080<br />	銆�銆�HistoryServer鐨剋eb绔彛</p> <p>4錛塻park.history.kerberos.enabled<br />銆�銆�榛樿鍊鹼細false<br />	銆�銆�鏄惁浣跨敤kerberos鏂瑰紡鐧誨綍璁塊棶HistoryServer錛屽浜庢寔涔呭眰浣嶄簬瀹夊叏闆嗙兢鐨凥DFS涓婃槸鏈夌敤鐨勶紝濡傛灉璁劇疆涓簍rue錛屽氨瑕侀厤緗笅闈㈢殑涓や釜灞炴�?/p> <p>5錛塻park.history.kerberos.principal<br />銆�銆�榛樿鍊鹼細鐢ㄤ簬HistoryServer鐨刱erberos涓諱綋鍚嶇О</p> <p>6錛塻park.history.kerberos.keytab<br />	銆�銆�鐢ㄤ簬HistoryServer鐨刱erberos keytab鏂囦歡浣嶇疆</p> <p>7錛塻park.history.ui.acls.enable<br />	銆�銆�榛樿鍊鹼細false<br />	銆�銆�鎺堟潈鐢ㄦ埛鏌ョ湅搴旂敤紼嬪簭淇℃伅鐨勬椂鍊欐槸鍚︽鏌cl銆傚鏋滃惎鐢紝鍙湁搴旂敤紼嬪簭鎵�鏈夎�呭拰spark.ui.view.acls鎸囧畾鐨勭敤鎴峰彲浠ユ煡鐪嬪簲鐢ㄧ▼搴忎俊鎭?鍚﹀垯錛屼笉鍋氫換浣曟鏌?/p> <p>8錛?span style="color: #ff0000;"><strong>spark.eventLog.enabled	</strong></span><br />	銆�銆�榛樿鍊鹼細false	<br />	銆�銆�鏄惁璁板綍Spark浜嬩歡錛岀敤浜庡簲鐢ㄧ▼搴忓湪瀹屾垚鍚庨噸鏋剋ebUI</p> <p>9錛?span style="color: #ff0000;"><strong>spark.eventLog.dir</strong></span><br />	銆�銆�榛樿鍊鹼細file:///tmp/spark-events<br />	銆�銆�淇濆瓨鏃ュ織鐩稿叧淇℃伅鐨勮礬寰勶紝鍙互鏄痟dfs://寮�澶寸殑HDFS璺緞錛屼篃鍙互鏄痜ile://寮�澶寸殑鏈湴璺緞錛岄兘闇�瑕佹彁鍓嶅垱寤?/p> <p>10錛?span style="color: #ff0000;"><strong>spark.eventLog.compress	</strong></span><br />	銆�銆�榛樿鍊鹼細false	<br />	銆�銆�鏄惁鍘嬬緝璁板綍Spark浜嬩歡錛屽墠鎻恠park.eventLog.enabled涓簍rue錛岄粯璁や嬌鐢ㄧ殑鏄痵nappy</p> <p><strong>浠park.history寮�澶寸殑闇�瑕侀厤緗湪spark-env.sh涓殑SPARK_HISTORY_OPTS錛屼互spark.eventLog寮�澶寸殑閰嶇疆鍦╯park-defaults.conf</strong></p> <p>&nbsp;</p> <p>鎴戝湪嫻嬭瘯榪囩▼涓殑閰嶇疆濡備笅錛?/p> <p>spark-defaults.conf</p> <div> <pre>spark.eventLog.enabled  <span style="color: #0000ff;">true</span><span style="color: #000000;"> spark.eventLog.dir      hdfs:</span><span style="color: #008000;">//</span><span style="color: #008000;">hadoop000:8020/directory</span> spark.eventLog.compress <span style="color: #0000ff;">true</span></pre> </div> <p>spark-env.sh</p> <div> <pre>export SPARK_HISTORY_OPTS="-D<strong><span style="color: #ff0000;">spark.history.ui.port</span></strong>=7777 -D<span style="color: #ff0000;"><strong>spark.history.retainedApplications</strong></span>=3 -D<span style="color: #ff0000;"><strong>spark.history.fs.logDirectory</strong></span>=hdfs://had oop000:8020/directory"</pre> </div> <p>鍙傛暟鎻忚堪錛?/p> <p>spark.history.ui.port=7777 &nbsp;璋冩暣WEBUI璁塊棶鐨勭鍙ｅ彿涓?777</p> <p>spark.history.fs.logDirectory=hdfs://hadoop000:8020/directory &nbsp;閰嶇疆浜嗚灞炴�у悗錛屽湪start-history-server.sh鏃跺氨鏃犻渶鍐嶆樉紺虹殑鎸囧畾璺緞</p> <p>spark.history.retainedApplications=3 &nbsp; 鎸囧畾淇濆瓨Application鍘嗗彶璁板綍鐨勪釜鏁幫紝濡傛灉瓚呰繃榪欎釜鍊鹼紝鏃х殑搴旂敤紼嬪簭淇℃伅灝嗚鍒犻櫎</p> <p>&nbsp;</p> <p>璋冩暣鍙傛暟鍚庡惎鍔╯tart-history-server.sh</p> <div> <pre>start-history-server.<span style="color: #0000ff;">sh</span> </pre> </div> <p>璁塊棶WEBUI錛?http://hadoop000:7777</p> <p><img src="http://images.cnitblog.com/blog/635295/201409/191457178315444.png" alt="" height="215" width="843" /></p> <p>&nbsp;</p> <p>鍦ㄤ嬌鐢╯park history server鐨勮繃紼嬩腑浜х敓鐨勫嚑涓枒闂細</p> <p><strong>鐤戦棶1錛歴park.history.fs.logDirectory鍜宻park.eventLog.dir鎸囧畾鐩綍鏈夊暐鍖哄埆錛?/strong></p> <p>緇忔祴璇曞悗鍙戠幇錛?/p> <p>spark.eventLog.dir錛欰pplication鍦ㄨ繍琛岃繃紼嬩腑鎵�鏈夌殑淇℃伅鍧囪褰曞湪璇ュ睘鎬ф寚瀹氱殑璺緞涓嬶紱</p> <p>spark.history.fs.logDirectory錛歋park History Server欏甸潰鍙睍紺鴻鎸囧畾璺緞涓嬬殑淇℃伅錛?/p> <p>姣斿錛歴park.eventLog.dir鍒氬紑濮嬫椂鎸囧畾鐨勬槸hdfs://hadoop000:8020/directory錛岃�屽悗淇敼鎴恏dfs://hadoop000:8020/directory2</p> <p>閭ｄ箞spark.history.fs.logDirectory濡傛灉鎸囧畾鐨勬槸hdfs://hadoop000:8020/directory錛屽氨鍙兘鏄劇ず鍑鴻鐩綍涓嬬殑鎵�鏈堿pplication榪愯鐨勬棩蹇椾俊鎭紱鍙嶄箣浜︾劧銆?/p> <p>&nbsp;</p> <p><strong>鐤戦棶2錛歴park.history.retainedApplications=3 璨屼技娌＄敓鏁堬紵錛燂紵錛燂紵錛?/strong></p> <p>The History Server will list all applications. It will just retain a  max number of them in memory. That option does not control how many  applications are show, it controls how much memory the HS will need.</p> <p>娉ㄦ剰錛氳鍙傛暟騫朵笉鏄篃欏甸潰涓樉紺虹殑application鐨勮褰曟暟錛岃�屾槸瀛樻斁鍦ㄥ唴瀛樹腑鐨勪釜鏁幫紝鍐呭瓨涓殑淇℃伅鍦ㄨ闂〉闈㈡椂鐩存帴璇誨彇娓叉煋鏃㈠彲錛?/p> <p>姣斿璇磋鍙傛暟閰嶇疆浜?0涓紝閭ｄ箞鍐呭瓨涓氨鏈�澶氬彧鑳藉瓨鏀?0涓猘pplicaiton鐨勬棩蹇椾俊鎭紝褰撶11涓姞鍏ユ椂錛岀涓�涓氨浼氳韙㈤櫎錛屽綋鍐嶆璁塊棶絎?涓猘pplication鐨勯〉闈俊鎭椂灝?span style="font-size: 14px; line-height: 1.5;">闇�瑕侀噸鏂拌鍙栨寚瀹氳礬寰勪笂鐨勬棩蹇椾俊鎭潵娓叉煋灞曠ず欏甸潰銆?nbsp;</span></p> <p>璇﹁瀹樻柟鏂囨。錛歨ttp://spark.apache.org/docs/latest/monitoring.html</p></div></div><img src ="http://www.tkk7.com/wangxinsh55/aggbug/430665.html" width = "1" height = "1" /><br><br><div align=right><a style="text-decoration:none;" href="http://www.tkk7.com/wangxinsh55/" target="_blank">SIMONE</a> 2016-05-26 14:12 <a href="http://www.tkk7.com/wangxinsh55/archive/2016/05/26/430665.html#Feedback" target="_blank" style="text-decoration:none;">鍙戣〃璇勮</a></div>]]></description></item><item><title>Spark On Yarn涓璼park.yarn.jar灞炴�х殑浣跨敤</title><link>http://www.tkk7.com/wangxinsh55/archive/2016/05/26/430664.html</link><dc:creator>SIMONE</dc:creator><author>SIMONE</author><pubDate>Thu, 26 May 2016 06:11:00 GMT</pubDate><guid>http://www.tkk7.com/wangxinsh55/archive/2016/05/26/430664.html</guid><wfw:comment>http://www.tkk7.com/wangxinsh55/comments/430664.html</wfw:comment><comments>http://www.tkk7.com/wangxinsh55/archive/2016/05/26/430664.html#Feedback</comments><slash:comments>0</slash:comments><wfw:commentRss>http://www.tkk7.com/wangxinsh55/comments/commentRss/430664.html</wfw:commentRss><trackback:ping>http://www.tkk7.com/wangxinsh55/services/trackbacks/430664.html</trackback:ping><description><![CDATA[<div>http://www.cnblogs.com/luogankun/p/4191796.html</div><br /><div><p>浠婂ぉ鍦ㄦ祴璇晄park-sql榪愯鍦▂arn涓婄殑榪囩▼涓紝鏃犳剰闂翠粠鏃ュ織涓彂鐜頒簡涓�涓棶棰橈細</p> <div> <pre>spark-sql --master yarn</pre> </div> <div><div><a title="澶嶅埗浠ｇ爜"><img src="http://common.cnblogs.com/images/copycode.gif" alt="澶嶅埗浠ｇ爜" /></a></div> <pre><span style="color: #800080;">14</span>/<span style="color: #800080;">12</span>/<span style="color: #800080;">29</span> <span style="color: #800080;">15</span>:<span style="color: #800080;">23</span>:<span style="color: #800080;">17</span> INFO Client: Requesting a new application from cluster with <span style="color: #800080;">1</span><span style="color: #000000;"> NodeManagers </span><span style="color: #800080;">14</span>/<span style="color: #800080;">12</span>/<span style="color: #800080;">29</span> <span style="color: #800080;">15</span>:<span style="color: #800080;">23</span>:<span style="color: #800080;">17</span> INFO Client: Verifying our application has not requested <span style="color: #0000ff;">more</span> than the maximum memory capability of the cluster (<span style="color: #800080;">8192</span><span style="color: #000000;"> MB per container) </span><span style="color: #800080;">14</span>/<span style="color: #800080;">12</span>/<span style="color: #800080;">29</span> <span style="color: #800080;">15</span>:<span style="color: #800080;">23</span>:<span style="color: #800080;">17</span> INFO Client: Will allocate AM container, with <span style="color: #800080;">896</span> MB memory including <span style="color: #800080;">384</span><span style="color: #000000;"> MB overhead </span><span style="color: #800080;">14</span>/<span style="color: #800080;">12</span>/<span style="color: #800080;">29</span> <span style="color: #800080;">15</span>:<span style="color: #800080;">23</span>:<span style="color: #800080;">17</span> INFO Client: Setting up container launch context <span style="color: #0000ff;">for</span><span style="color: #000000;"> our AM </span><span style="color: #800080;">14</span>/<span style="color: #800080;">12</span>/<span style="color: #800080;">29</span> <span style="color: #800080;">15</span>:<span style="color: #800080;">23</span>:<span style="color: #800080;">17</span> INFO Client: Preparing resources <span style="color: #0000ff;">for</span><span style="color: #000000;"> our AM container </span><span style="color: #800080;">14</span>/<span style="color: #800080;">12</span>/<span style="color: #800080;">29</span> <span style="color: #800080;">15</span>:<span style="color: #800080;">23</span>:<span style="color: #800080;">17</span> INFO Client: <span style="color: #ff0000;"><strong>Uploading resource file:/home/spark/software/source/compile/deploy_spark/assembly/target/scala-2.10/spark-assembly-1.3.0-SNAPSHOT-hadoop2.3.0-cdh5.0.0.jar -&gt; hdfs://hadoop000:8020/user/spark/.sparkStaging/<span style="color: #00ff00;">application_1416381870014_0093</span>/spark-assembly-1.3.0-SNAPSHOT-hadoop2.3.0-cdh5.0.0.jar</strong></span> <span style="color: #800080;">14</span>/<span style="color: #800080;">12</span>/<span style="color: #800080;">29</span> <span style="color: #800080;">15</span>:<span style="color: #800080;">23</span>:<span style="color: #800080;">18</span> INFO Client: Setting up the launch environment <span style="color: #0000ff;">for</span> our AM container</pre> <div><a title="澶嶅埗浠ｇ爜"><img src="http://common.cnblogs.com/images/copycode.gif" alt="澶嶅埗浠ｇ爜" /></a></div></div> <p>鍐嶅紑鍚竴涓猻park-sql鍛戒護琛岋紝浠庢棩蹇椾腑鍐嶆鍙戠幇錛?/p> <div><div><a title="澶嶅埗浠ｇ爜"><img src="http://common.cnblogs.com/images/copycode.gif" alt="澶嶅埗浠ｇ爜" /></a></div> <pre><span style="color: #800080;">14</span>/<span style="color: #800080;">12</span>/<span style="color: #800080;">29</span> <span style="color: #800080;">15</span>:<span style="color: #800080;">24</span>:<span style="color: #800080;">03</span> INFO Client: Requesting a new application from cluster with <span style="color: #800080;">1</span><span style="color: #000000;"> NodeManagers </span><span style="color: #800080;">14</span>/<span style="color: #800080;">12</span>/<span style="color: #800080;">29</span> <span style="color: #800080;">15</span>:<span style="color: #800080;">24</span>:<span style="color: #800080;">03</span> INFO Client: Verifying our application has not requested <span style="color: #0000ff;">more</span> than the maximum memory capability of the cluster (<span style="color: #800080;">8192</span><span style="color: #000000;"> MB per container) </span><span style="color: #800080;">14</span>/<span style="color: #800080;">12</span>/<span style="color: #800080;">29</span> <span style="color: #800080;">15</span>:<span style="color: #800080;">24</span>:<span style="color: #800080;">03</span> INFO Client: Will allocate AM container, with <span style="color: #800080;">896</span> MB memory including <span style="color: #800080;">384</span><span style="color: #000000;"> MB overhead </span><span style="color: #800080;">14</span>/<span style="color: #800080;">12</span>/<span style="color: #800080;">29</span> <span style="color: #800080;">15</span>:<span style="color: #800080;">24</span>:<span style="color: #800080;">03</span> INFO Client: Setting up container launch context <span style="color: #0000ff;">for</span><span style="color: #000000;"> our AM </span><span style="color: #800080;">14</span>/<span style="color: #800080;">12</span>/<span style="color: #800080;">29</span> <span style="color: #800080;">15</span>:<span style="color: #800080;">24</span>:<span style="color: #800080;">03</span> INFO Client: Preparing resources <span style="color: #0000ff;">for</span><span style="color: #000000;"> our AM container </span><span style="color: #800080;">14</span>/<span style="color: #800080;">12</span>/<span style="color: #800080;">29</span> <span style="color: #800080;">15</span>:<span style="color: #800080;">24</span>:<span style="color: #800080;">03</span> INFO Client: <span style="color: #ff0000;"><strong>Uploading resource file:/home/spark/software/source/compile/deploy_spark/assembly/target/scala-2.10/spark-assembly-1.3.0-SNAPSHOT-hadoop2.3.0-cdh5.0.0.jar -&gt; hdfs://hadoop000:8020/user/spark/.sparkStaging/<span style="color: #00ff00;">application_1416381870014_0094</span>/spark-assembly-1.3.0-SNAPSHOT-hadoop2.3.0-cdh5.0.0.jar</strong></span> <span style="color: #800080;">14</span>/<span style="color: #800080;">12</span>/<span style="color: #800080;">29</span> <span style="color: #800080;">15</span>:<span style="color: #800080;">24</span>:<span style="color: #800080;">05</span> INFO Client: Setting up the launch environment <span style="color: #0000ff;">for</span> our AM container</pre> <div><a title="澶嶅埗浠ｇ爜"><img src="http://common.cnblogs.com/images/copycode.gif" alt="澶嶅埗浠ｇ爜" /></a></div></div> <p>鐒跺悗鏌ョ湅HDFS涓婄殑鏂囦歡錛?/p> <div> <pre>hadoop fs -<span style="color: #0000ff;">ls</span> hdfs:<span style="color: #008000;">//</span><span style="color: #008000;">hadoop000:8020/user/spark/.sparkStaging/</span></pre> </div> <div> <pre>drwx------   - spark supergroup          <span style="color: #800080;">0</span> <span style="color: #800080;">2014</span>-<span style="color: #800080;">12</span>-<span style="color: #800080;">29</span> <span style="color: #800080;">15</span>:<span style="color: #800080;">23</span> hdfs:<span style="color: #008000;">//</span><span style="color: #008000;">hadoop000:8020/user/spark/.sparkStaging/<span style="color: #00ff00;">application_1416381870014_0093</span></span> drwx------   - spark supergroup          <span style="color: #800080;">0</span> <span style="color: #800080;">2014</span>-<span style="color: #800080;">12</span>-<span style="color: #800080;">29</span> <span style="color: #800080;">15</span>:<span style="color: #800080;">24</span> hdfs:<span style="color: #008000;">//</span><span style="color: #008000;">hadoop000:8020/user/spark/.sparkStaging/<span style="color: #00ff00;">application_1416381870014_0094</span></span></pre> </div> <p>姣忎釜Application閮戒細涓婁紶涓�涓猻park-assembly-x.x.x-SNAPSHOT-hadoopx.x.x-cdhx.x.x.jar鐨刯ar鍖咃紝褰卞搷HDFS鐨勬�ц兘浠ュ強鍗犵敤HDFS鐨勭┖闂淬�?/p> <p>&nbsp;</p> <p>鍦⊿park鏂囨。(http://spark.apache.org/docs/latest/running-on-yarn.html)涓彂鐜?span style="color: #ff0000;"><strong>spark.yarn.jar</strong></span>灞炴�э紝灝唖park-assembly-xxxxx.jar瀛樻斁鍦╤dfs://hadoop000:8020/spark_lib/涓?/p> <p>鍦╯park-defaults.conf娣誨姞灞炴�ч厤緗細</p> <div> <pre><strong><span style="color: #ff0000;">spark.yarn.jar</span></strong> hdfs://hadoop000:8020/spark_lib/spark-assembly-1.3.0-SNAPSHOT-hadoop2.3.0-cdh5.0.0.jar</pre> </div> <p>鍐嶆鍚姩spark-sql --master yarn瑙傚療鏃ュ織錛?/p> <div><div><a title="澶嶅埗浠ｇ爜"><img src="http://common.cnblogs.com/images/copycode.gif" alt="澶嶅埗浠ｇ爜" /></a></div> <pre><span style="color: #000000;">14/12/29 15:39:02 INFO Client: Requesting a new application from cluster with 1 NodeManagers 14/12/29 15:39:02 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container) 14/12/29 15:39:02 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead 14/12/29 15:39:02 INFO Client: Setting up container launch context for our AM 14/12/29 15:39:02 INFO Client: Preparing resources for our AM container 14/12/29 15:39:02 INFO Client: <strong><span style="color: #ff0000;">Source and destination file systems are the same. Not copying hdfs://hadoop000:8020/spark_lib/spark-assembly-1.3.0-SNAPSHOT-hadoop2.3.0-cdh5.0.0.jar</span></strong> 14/12/29 15:39:02 INFO Client: Setting up the launch environment for our AM container</span></pre> <div><a title="澶嶅埗浠ｇ爜"><img src="http://common.cnblogs.com/images/copycode.gif" alt="澶嶅埗浠ｇ爜" /></a></div></div> <p>瑙傚療HDFS涓婃枃浠?/p> <div> <pre>hadoop fs -ls hdfs://hadoop000:8020/user/spark/.sparkStaging/application_1416381870014_0097</pre> </div> <p>璇pplication瀵瑰簲鐨勭洰褰曚笅娌℃湁spark-assembly-xxxxx.jar浜嗭紝浠庤�岃妭鐪乤ssembly鍖呬笂浼犵殑榪囩▼浠ュ強HDFS絀洪棿鍗犵敤銆?/p> <p>&nbsp;</p> <p>鎴戝湪嫻嬭瘯榪囩▼涓亣鍒頒簡綾諱技濡備笅鐨勯敊璇細</p> <div> <pre>Application application_xxxxxxxxx_yyyy failed 2 times due to AM Container for application_xxxxxxxxx_yyyy <br /><br />exited with exitCode: -1000 due to: java.io.FileNotFoundException: File /tmp/hadoop-spark/nm-local-dir/filecache does not exist</pre> </div> <p>鍦?tmp/hadoop-spark/nm-local-dir璺緞涓嬪垱寤篺ilecache鏂囦歡澶瑰嵆鍙В鍐蟲姤閿欓棶棰樸�?/p></div><img src ="http://www.tkk7.com/wangxinsh55/aggbug/430664.html" width = "1" height = "1" /><br><br><div align=right><a style="text-decoration:none;" href="http://www.tkk7.com/wangxinsh55/" target="_blank">SIMONE</a> 2016-05-26 14:11 <a href="http://www.tkk7.com/wangxinsh55/archive/2016/05/26/430664.html#Feedback" target="_blank" style="text-decoration:none;">鍙戣〃璇勮</a></div>]]></description></item></channel></rss>                                                 <footer>
<div class="friendship-link">
<p>感谢您访问我们的网站，您可能还对以下资源感兴趣：</p>
<a href="http://www.tkk7.com/" title="&#20122;&#27954;&#97;&#118;&#25104;&#20154;&#29255;&#22312;&#32447;&#35266;&#30475;">&#20122;&#27954;&#97;&#118;&#25104;&#20154;&#29255;&#22312;&#32447;&#35266;&#30475;</a>

<div class="friend-links">


</div>
</div>

</footer>

主站蜘蛛池模板：
<a href="http://hbjinmaitang.com" target="_blank">免费看一级做a爰片久久</a>|
<a href="http://xsdin.com" target="_blank">暖暖在线日本免费中文</a>|
<a href="http://www-554949.com" target="_blank">亚洲毛片av日韩av无码</a>|
<a href="http://tltcn.com" target="_blank">日韩亚洲综合精品国产</a>|
<a href="http://lawelites.com" target="_blank">免费人成在线观看网站品爱网日本</a>|
<a href="http://szyxfhm.com" target="_blank">亚洲久热无码av中文字幕</a>|
<a href="http://cao9999.com" target="_blank">日本免费中文字幕在线看</a>|
<a href="http://hetaoqpj.com" target="_blank">国产精品无码亚洲精品2021</a>|
<a href="http://hzsprfm.com" target="_blank">日本高清免费中文字幕不卡</a>|
<a href="http://yhanalati.com" target="_blank">爱情岛亚洲论坛在线观看</a>|
<a href="http://5s6b.com" target="_blank">亚洲成A人片在线观看无码3D</a>|
<a href="http://733807.com" target="_blank">人体大胆做受免费视频</a>|
<a href="http://kan63.com" target="_blank">亚洲精品国产美女久久久</a>|
<a href="http://scienfood.com" target="_blank">久久免费视频99</a>|
<a href="http://www96008.com" target="_blank">亚洲人成电影院在线观看</a>|
<a href="http://aa7852.com" target="_blank">日本特黄特黄刺激大片免费</a>|
<a href="http://ningxialanh.com" target="_blank">老司机亚洲精品影院在线观看</a>|
<a href="http://socgl.com" target="_blank">亚洲国产一成久久精品国产成人综合</a>|
<a href="http://0755szyxcm.com" target="_blank">a一级毛片免费高清在线</a>|
<a href="http://zhidianzh.com" target="_blank">亚洲国产精品无码久久一区二区</a>|
<a href="http://qq367.com" target="_blank">久久青草91免费观看</a>|
<a href="http://g8zb.com" target="_blank">亚洲一区中文字幕</a>|
<a href="http://199291.com" target="_blank">国产91久久久久久久免费</a>|
<a href="http://99999pp.com" target="_blank">黄 色一级 成 人网站免费</a>|
<a href="http://hljc988.com" target="_blank">无码乱人伦一区二区亚洲</a>|
<a href="http://saozib.com" target="_blank">一二三四免费观看在线视频中文版</a>|
<a href="http://zf91.com" target="_blank">亚洲国产精华液2020</a>|
<a href="http://2030188.com" target="_blank">亚洲综合色自拍一区</a>|
<a href="http://dstbxg.com" target="_blank">免费能直接在线观看黄的视频</a>|
<a href="http://qiyupu.com" target="_blank">亚洲最大av资源站无码av网址</a>|
<a href="http://hidiaoyan.com" target="_blank">免费国产精品视频</a>|
<a href="http://cnepu.com" target="_blank">无码av免费一区二区三区</a>|
<a href="http://0755szyxcm.com" target="_blank">亚洲女女女同性video</a>|
<a href="http://www79909c.com" target="_blank">一本色道久久综合亚洲精品</a>|
<a href="http://shyangpuks.com" target="_blank">免费不卡在线观看AV</a>|
<a href="http://nn227.com" target="_blank">国产精品自拍亚洲</a>|
<a href="http://see13.com" target="_blank">亚洲蜜芽在线精品一区</a>|
<a href="http://992fanli.com" target="_blank">波多野结衣一区二区免费视频
</a>|
<a href="http://01shanzhai.com" target="_blank">在线观看国产一区亚洲bd</a>|
<a href="http://dou18.com" target="_blank">亚洲韩国精品无码一区二区三区</a>|
<a href="http://513net.com" target="_blank">99久久国产热无码精品免费</a>|
<script>
(function(){
    var bp = document.createElement('script');
    var curProtocol = window.location.protocol.split(':')[0];
    if (curProtocol === 'https') {
        bp.src = 'https://zz.bdstatic.com/linksubmit/push.js';
    }
    else {
        bp.src = 'http://push.zhanzhang.baidu.com/push.js';
    }
    var s = document.getElementsByTagName("script")[0];
    s.parentNode.insertBefore(bp, s);
})();
</script>
</body>