??xml version="1.0" encoding="utf-8" standalone="yes"?>亚洲精品无码专区,精品亚洲一区二区三区在线播放 ,亚洲av成本人无码网站http://www.tkk7.com/shenh062326/zh-cnSun, 11 May 2025 06:21:22 GMTSun, 11 May 2025 06:21:22 GMT60招聘 hadoop开发工E师http://www.tkk7.com/shenh062326/archive/2013/09/15/404093.html俞灵俞灵Sun, 15 Sep 2013 10:21:00 GMThttp://www.tkk7.com/shenh062326/archive/2013/09/15/404093.htmlhttp://www.tkk7.com/shenh062326/comments/404093.htmlhttp://www.tkk7.com/shenh062326/archive/2013/09/15/404093.html#Feedback1http://www.tkk7.com/shenh062326/comments/commentRss/404093.htmlhttp://www.tkk7.com/shenh062326/services/trackbacks/404093.html

淘宝招聘hadoop工程师若q? 面向在校?2014q毕?,工作地点:杭州或北?/strong>

Hadoop研发工程?br />职位描述
您将负责Q?br />1Q预研、开发、测试hdfs/mapreduce/hive/hbase的功能、性能和扩展;
2Q对有助于提升集处理能?高可用?高扩展性的各种解决Ҏq行跟踪和落圎ͼ
3Q解xv量数据不断增镉K临的挑战Q解决业务需求?/span>

您需要具备:
1、熟l运用java语言Q?br />2、熟悉jvmq行机制、熟悉linuxQ?br />3、至熟悉hadoop、hbase、hive{Y件之一Q?br />

 

有意者请发送邮件到 yuling.sh@taobao.com 



俞灵 2013-09-15 18:21 发表评论
]]>
hadoop各种输入Ҏ(InputFormat)汇?/title><link>http://www.tkk7.com/shenh062326/archive/2012/07/03/hadoop.html</link><dc:creator>俞灵</dc:creator><author>俞灵</author><pubDate>Tue, 03 Jul 2012 14:17:00 GMT</pubDate><guid>http://www.tkk7.com/shenh062326/archive/2012/07/03/hadoop.html</guid><wfw:comment>http://www.tkk7.com/shenh062326/comments/382115.html</wfw:comment><comments>http://www.tkk7.com/shenh062326/archive/2012/07/03/hadoop.html#Feedback</comments><slash:comments>2</slash:comments><wfw:commentRss>http://www.tkk7.com/shenh062326/comments/commentRss/382115.html</wfw:commentRss><trackback:ping>http://www.tkk7.com/shenh062326/services/trackbacks/382115.html</trackback:ping><description><![CDATA[<div> <p><span style="font-size:12.0pt;font-family:Consolas;color:black;"><br /></span><span style="font-size:12.0pt;font-family:宋体;color:black;"></span></p><p style="text-indent:21.0pt"><span style="font-size:10.0pt;font-family:Consolas; color:black;">mapreduce</span><span style="font-size:10.0pt; font-family:宋体;color:black;">?/span><span style="font-size:10.0pt;font-family:Consolas; color:black;">,</span><span style="font-size:10.0pt; font-family:宋体;color:black;">一?/span><span style="font-size:10.0pt;font-family:Consolas; color:black;">job</span><span style="font-size:10.0pt; font-family:宋体;color:black;">?/span><span style="font-size:10.0pt;font-family:Consolas; color:black;">map</span><span style="font-size:10.0pt; font-family:宋体;color:black;">个数</span><span style="font-size:10.0pt;font-family:Consolas; color:black;">, </span><span style="font-size:10.0pt; font-family:宋体;color:black;">每个</span><span style="font-size:10.0pt;font-family:Consolas; color:black;">map</span><span style="font-size:10.0pt; font-family:宋体;color:black;">处理的数据量是如何决定的?/span><span style="font-size:10.0pt;font-family:Consolas;color:black;">? </span><span style="font-size: 10.0pt;font-family:宋体;color:black;">另外每个</span><span style="font-size:10.0pt;font-family:Consolas;color:black;">map</span><span style="font-size: 10.0pt;font-family:宋体;color:black;">又是如何d输入文g的内容呢</span><span style="font-size:10.0pt;font-family:Consolas;color:black;">? </span><span style="font-size: 10.0pt;font-family:宋体;color:black;">用户是否可以自己军_输入方式</span><span style="font-size:10.0pt;font-family:Consolas;color:black;">, </span><span style="font-size: 10.0pt;font-family:宋体;color:black;">军_</span><span style="font-size:10.0pt;font-family:Consolas;color:black;">map</span><span style="font-size: 10.0pt;font-family:宋体;color:black;">个数?/span><span style="font-size:10.0pt;font-family:Consolas;color:black;">? </span><span style="font-size: 10.0pt;font-family:宋体;color:black;">q篇文章详l讲q?/span><span style="font-size:10.0pt;font-family:Consolas;color:black;">hadoop</span><span style="font-size: 10.0pt;font-family:宋体;color:black;">中各U?/span><span style="font-size:9.0pt;font-family:"微Y雅黑","sans-serif"">InputFormat</span><span style="font-size:9.0pt;font-family:"微Y雅黑","sans-serif"">的功能和如何~写自定义的InputFormat. </span></p> <p> </p> <p><span style="font-size:10.0pt;font-family:宋体;color:black;">?/span><span style="font-size:10.0pt; font-family:Consolas;color:black;">: mapreduce</span><span style="font-size:10.0pt;font-family:宋体;color:black;">作业会根据输入目录生多?/span><span style="font-size: 10.0pt;font-family:Consolas;color:black;">map</span><span style="font-size:10.0pt;font-family:宋体; color:black;">d</span><span style="font-size: 10.0pt;font-family:Consolas;color:black;">, </span><span style="font-size:10.0pt;font-family:宋体; color:black;">通过多个</span><span style="font-size: 10.0pt;font-family:Consolas;color:black;">map</span><span style="font-size:10.0pt;font-family:宋体; color:black;">dq行执行来提高作业运行速度</span><span style="font-size:10.0pt;font-family:Consolas; color:black;">, </span><span style="font-size:10.0pt; font-family:宋体;color:black;">但如?/span><span style="font-size:10.0pt;font-family:Consolas; color:black;">map</span><span style="font-size:10.0pt; font-family:宋体;color:black;">数量q少</span><span style="font-size:10.0pt;font-family:Consolas; color:black;">, </span><span style="font-size:10.0pt; font-family:宋体;color:black;">q行量低</span><span style="font-size:10.0pt;font-family:Consolas; color:black;">, </span><span style="font-size:10.0pt; font-family:宋体;color:black;">作业执行?/span><span style="font-size:10.0pt;font-family:Consolas; color:black;">, </span><span style="font-size:10.0pt; font-family:宋体;color:black;">如果</span><span style="font-size:10.0pt;font-family:Consolas; color:black;">map</span><span style="font-size:10.0pt; font-family:宋体;color:black;">数过?/span><span style="font-size:10.0pt;font-family:Consolas; color:black;">, </span><span style="font-size:10.0pt; font-family:宋体;color:black;">资源有限</span><span style="font-size:10.0pt;font-family:Consolas; color:black;">, </span><span style="font-size:10.0pt; font-family:宋体;color:black;">也会增加调度开销</span><span style="font-size:10.0pt;font-family:Consolas; color:black;">. </span><span style="font-size:10.0pt; font-family:宋体;color:black;">因此</span><span style="font-size:10.0pt;font-family:Consolas; color:black;">, </span><span style="font-size:10.0pt; font-family:宋体;color:black;">Ҏ输入产生合理?/span><span style="font-size:10.0pt;font-family:Consolas; color:black;">map</span><span style="font-size:10.0pt; font-family:宋体;color:black;">?/span><span style="font-size:10.0pt;font-family:Consolas; color:black;">, </span><span style="font-size:10.0pt; font-family:宋体;color:black;">为每?/span><span style="font-size:10.0pt;font-family:Consolas; color:black;">map</span><span style="font-size:10.0pt; font-family:宋体;color:black;">分配合适的数据?/span><span style="font-size:10.0pt;font-family:Consolas; color:black;">, </span><span style="font-size:10.0pt; font-family:宋体;color:black;">能有效的提升资源利用?/span><span style="font-size:10.0pt;font-family:Consolas; color:black;">, </span><span style="font-size:10.0pt; font-family:宋体;color:black;">q作业q行速度加快</span><span style="font-size:10.0pt;font-family:Consolas; color:black;">.</span></p> <p><span style="font-size:10.0pt;font-family:Consolas;color:black;">    </span><span style="font-size:10.0pt; font-family:宋体;color:black;">?/span><span style="font-size:10.0pt;font-family:Consolas; color:black;">mapreduce</span><span style="font-size:10.0pt; font-family:宋体;color:black;">?/span><span style="font-size:10.0pt;font-family:Consolas; color:black;">, </span><span style="font-size:10.0pt; font-family:宋体;color:black;">每个作业都会通过</span> <span style="font-size: 9.0pt;font-family:"微Y雅黑","sans-serif"">InputFormat</span><span style="font-size:10.0pt;font-family:宋体;color:black;">来决?/span><span style="font-size:10.0pt;font-family:Consolas;color:black;">map</span><span style="font-size: 10.0pt;font-family:宋体;color:black;">数量</span><span style="font-size:10.0pt;font-family:Consolas;color:black;">. </span><span style="font-size:9.0pt;font-family:"微Y雅黑","sans-serif"">InputFormat</span><span style="font-size:9.0pt;font-family:"微Y雅黑","sans-serif"">是一个接?span>, </span>提供两个Ҏ:</span></p> <p style="text-align:left;text-autospace:none" align="left"><span style="font-size:9.0pt;font-family: Consolas;color:black;">InputSplit[] getSplits(JobConf job, </span><strong><span style="font-size:9.0pt; font-family:Consolas;color:#7F0055;">int</span></strong><span style="font-size:9.0pt;font-family:Consolas;color:black;"> numSplits) </span><strong><span style="font-size:9.0pt;font-family:Consolas;color:#7F0055;">throws</span></strong><span style="font-size:9.0pt;font-family:Consolas; color:black;"> IOException;</span></p> <p style="text-align:left;text-autospace:none" align="left"><span style="font-size:9.0pt;font-family: Consolas;color:black;">RecordReader<K, V> getRecordReader(InputSplit split,</span></p> <p style="text-align:left;text-autospace:none" align="left"><span style="font-size:9.0pt;font-family: Consolas;color:black;">                                     JobConf job, </span></p> <p><span style="font-size:9.0pt;font-family:Consolas;color:black;">                                     Reporter reporter) </span><strong><span style="font-size:9.0pt;font-family:Consolas;color:#7F0055;">throws</span></strong><span style="font-size:9.0pt;font-family:Consolas;color:black;"> IOException;</span></p> <p><span style="font-size:10.0pt;font-family:Consolas;color:black;">    </span><span style="font-size:10.0pt; font-family:宋体;color:black;">其中</span><span style="font-size:10.0pt;font-family:Consolas; color:black;">getSplits</span><span style="font-size:10.0pt; font-family:宋体;color:black;">Ҏ会根据输入目录?/span><span style="font-size:10.0pt;font-family:Consolas; color:black;">InputSplit</span><span style="font-size:10.0pt; font-family:宋体;color:black;">数组</span><span style="font-size:10.0pt;font-family:Consolas; color:black;">, </span><span style="font-size:10.0pt; font-family:宋体;color:black;">每个</span><span style="font-size:10.0pt;font-family:Consolas; color:black;">InputSplit</span><span style="font-size:10.0pt; font-family:宋体;color:black;">会相应生一?/span><span style="font-size:10.0pt;font-family:Consolas; color:black;">map</span><span style="font-size:10.0pt; font-family:宋体;color:black;">d</span><span style="font-size:10.0pt;font-family:Consolas; color:black;">, map</span><span style="font-size:10.0pt; font-family:宋体;color:black;">的输入定义在</span><span style="font-size:10.0pt;font-family:Consolas; color:black;">InputSplit</span><span style="font-size:10.0pt; font-family:宋体;color:black;">?/span><span style="font-size:10.0pt;font-family:Consolas; color:black;">. getRecordReader</span><span style="font-size:10.0pt;font-family:宋体;color:black;">Ҏq回一?/span><span style="font-size:10.0pt;font-family:Consolas;color:black;">RecordReader</span><span style="font-size:10.0pt;font-family:宋体;color:black;">对象</span><span style="font-size:10.0pt;font-family:Consolas;color:black;">, RecordReader</span><span style="font-size:10.0pt;font-family:宋体;color:black;">军_?/span><span style="font-size:10.0pt;font-family:Consolas;color:black;">map</span><span style="font-size: 10.0pt;font-family:宋体;color:black;">d如何d输入数据</span><span style="font-size:10.0pt;font-family:Consolas;color:black;">, </span><span style="font-size: 10.0pt;font-family:宋体;color:black;">例如一行一行的dq是一个字节一个字节的d</span><span style="font-size:10.0pt;font-family:Consolas;color:black;">, </span><span style="font-size: 10.0pt;font-family:宋体;color:black;">{等</span><span style="font-size:10.0pt;font-family:Consolas;color:black;">.</span></p> <p><span style="font-size:10.0pt;font-family:Consolas;color:black;">    </span><span style="font-size:10.0pt; font-family:宋体;color:black;">下图?/span><span style="font-size:9.0pt;font-family:"微Y雅黑","sans-serif"">InputFormat</span><span style="font-size:9.0pt;font-family:"微Y雅黑","sans-serif"">的实现类:</span></p> <p><span style="font-size:9.0pt;font-family:"微Y雅黑","sans-serif"">       (暂时无法上传)</span><span style="font-size: 12.0pt;font-family:宋体;"></span></p> <p><span style="font-size:10.0pt;font-family:Consolas;color:black;">    </span><span style="font-size:10.0pt; font-family:宋体;">q理详细介绍</span><span style="font-size: 10.0pt;font-family:Consolas;">FileInputFormat</span><span style="font-size:10.0pt;font-family:宋体;">?/span><span style="font-size:10.0pt; font-family:Consolas;">CombineFileInputFormat, </span><span style="font-size:10.0pt;font-family:宋体;">其它不常?/span><span style="font-size:10.0pt;font-family:Consolas;">,</span><span style="font-size:10.0pt; font-family:宋体;">有兴的可以自己查看</span><span style="font-size:10.0pt;font-family:Consolas;">hadoop</span><span style="font-size:10.0pt;font-family: 宋体;">源码</span><span style="font-size: 10pt; font-family: Consolas;">.</span></p><p><span style="font-size:10.0pt; font-family:Consolas;"><br /></span></p> <p> </p><div> <p><span style="font-size:14.0pt;font-family:Consolas;">FileInputFormat(</span><span style="font-size:14.0pt;font-family:宋体;">旧接?/span><span style="font-size:10.0pt;font-family:Consolas;color:black;">org.apache.hadoop.mapred</span><span style="font-size:14.0pt;font-family:Consolas;">)</span></p> </div><span style="font-size:14.0pt;font-family:宋体;"></span><span style="font-size: 10.0pt;font-family:Consolas;color:black;"></span><span style="font-size:14.0pt;font-family:Consolas;"></span><span style="font-size:14.0pt;font-family:Consolas;"></span><p> </p> <p style="text-indent:21.0pt"><span style="font-size:10.0pt;font-family:Consolas; color:black;">mapreduce</span><span style="font-size:10.0pt; font-family:宋体;color:black;">默认使用</span><span style="font-size:10.0pt;font-family:Consolas; color:black;">TextInputFormat</span><span style="font-size: 10.0pt;font-family:宋体;color:black;">Q?/span><span style="font-size:10.0pt;font-family:Consolas;color:black;">TextInputFormat</span><span style="font-size:10.0pt;font-family:宋体;color:black;">没有实现自己?/span><span style="font-size:9.0pt;font-family:Consolas;color:black;">getSplits</span><span style="font-size:9.0pt;font-family:宋体;color:black;">Ҏ</span><span style="font-size:9.0pt;font-family:Consolas;color:black;">,</span><span style="font-size:9.0pt; font-family:宋体;color:black;">?/span><span style="font-size:10.0pt; font-family:宋体;color:black;">l承?/span><span style="font-size:10.0pt;font-family:Consolas;">FileInputFormat, </span><span style="font-size:10.0pt; font-family:宋体;">因此使用?/span><span style="font-size: 10.0pt;font-family:Consolas;">FileInputFormat?</span></p> <p style="text-indent:21.0pt"><span style="font-size:10.0pt;font-family:Consolas; color:black;">org.apache.hadoop.mapred.</span><span style="font-size:10.0pt;font-family:Consolas;">FileInputFormat</span><span style="font-size: 10.0pt;font-family:宋体;">?/span><span style="font-size:9.0pt;font-family:Consolas; color:black;">getSplits</span><span style="font-size:9.0pt; font-family:宋体;color:black;">程</span><span style="font-size:9.0pt;font-family:Consolas; color:black;">:</span></p> <p style="text-indent:21.0pt"><span style="font-size:9.0pt; font-family:宋体;">两个配置</span></p> <p style="margin-left:21.0pt;text-indent:21.0pt"><span style="font-size: 10pt; font-family: Consolas;">mapred.min.split.size</span>        <span style="font-size:10.0pt;font-family:Consolas;">(</span><span style="font-size:10.0pt;font-family:宋体;">一?/span><span style="font-size:10.0pt;font-family:Consolas;">map</span><span style="font-size:10.0pt; font-family:宋体;">最输入长?/span><span style="font-size: 10.0pt;font-family:Consolas;">), </span></p> <p style="margin-left:16.0pt;text-indent:26.0pt;"><span style="font-size: 10pt; font-family: Consolas;">mapred.map.tasks</span>                <span style="font-size:10.0pt;font-family:Consolas;">(</span><span style="font-size: 10.0pt;font-family:宋体;">推荐</span><span style="font-size:10.0pt;font-family:Consolas;">map</span><span style="font-size:10.0pt;font-family:宋体;">数量</span><span style="font-size:10.0pt; font-family:Consolas;">)</span></p> <p style="text-indent:21.0pt"><span style="font-size:10.0pt; font-family:宋体;">如何军_每个</span><span style="font-size: 10.0pt;font-family:Consolas;">map</span><span style="font-size:10.0pt;font-family:宋体;">输入长度?/span><span style="font-size:10.0pt;font-family:Consolas;">? </span><span style="font-size:10.0pt;font-family:宋体;">首先获取输入目录下所有文件的长度?/span><span style="font-size:10.0pt;font-family:Consolas;">, </span><span style="font-size:10.0pt; font-family:宋体;">除以</span><span style="font-size:10.0pt; font-family:Consolas;">mapred.map.tasks</span><span style="font-size:10.0pt;font-family:宋体;">得到一个推荐长?/span><span style="font-size:10.0pt;font-family:Consolas;">goalSize, </span><span style="font-size:10.0pt; font-family:宋体;">然后通过式子</span><span style="font-size: 10.0pt;font-family:Consolas;">: Math.<em>max</em>(minSize, Math.<em>min</em>(goalSize, blockSize))</span><span style="font-size:10.0pt;font-family:宋体;">军_</span><span style="font-size:10.0pt;font-family:Consolas;">map</span><span style="font-size:10.0pt; font-family:宋体;">输入长度</span><span style="font-size: 10.0pt;font-family:Consolas;">. </span><span style="font-size:10.0pt;font-family:宋体;">q里?/span><span style="font-size:10.0pt;font-family:Consolas;">minSize</span><span style="font-size:10.0pt;font-family:宋体;">?/span><span style="font-size:10.0pt;font-family:Consolas;">mapred.min.split.size, blockSize</span><span style="font-size:10.0pt;font-family:宋体;">为相应文件的</span><span style="font-size:10.0pt;font-family:Consolas;">block</span><span style="font-size:10.0pt; font-family:宋体;">长度</span><span style="font-size:10.0pt; font-family:Consolas;">. </span><span style="font-size:10.0pt;font-family:宋体;">q式子能保证一?/span><span style="font-size:10.0pt;font-family:Consolas;">map</span><span style="font-size:10.0pt; font-family:宋体;">的输入至大?/span><span style="font-size: 10.0pt;font-family:Consolas;">mapred.min.split.size, </span><span style="font-size:10.0pt;font-family: 宋体;">对于推荐?/span><span style="font-size:10.0pt; font-family:Consolas;">map</span><span style="font-size:10.0pt;font-family:宋体;">长度</span><span style="font-size:10.0pt;font-family:Consolas;">,</span><span style="font-size:10.0pt; font-family:宋体;">只有它的长度于</span><span style="font-size:10.0pt;font-family:Consolas;">blockSize</span><span style="font-size:10.0pt;font-family: 宋体;">且大?/span><span style="font-size:10.0pt; font-family:Consolas;">mapred.min.split.size</span><span style="font-size:10.0pt;font-family:宋体;">才会有效?/span><span style="font-size:10.0pt;font-family:Consolas;">. </span><span style="font-size:10.0pt; font-family:宋体;">׃</span><span style="font-size:10.0pt; font-family:Consolas;">mapred.min.split.size</span><span style="font-size:10.0pt;font-family:宋体;">默认长度?/span><span style="font-size:10.0pt;font-family:Consolas;">1, </span><span style="font-size:10.0pt; font-family:宋体;">因此通常情况下只要小?/span><span style="font-size:10.0pt;font-family:Consolas;">blockSize</span><span style="font-size:10.0pt;font-family: 宋体;">有效果</span><span style="font-size:10.0pt; font-family:Consolas;">,</span><span style="font-size:10.0pt;font-family:宋体;">否则使用</span><span style="font-size:10.0pt;font-family:Consolas;">blockSize</span><span style="font-size:10.0pt; font-family:宋体;">做ؓ</span><span style="font-size:10.0pt; font-family:Consolas;">map</span><span style="font-size:10.0pt;font-family:宋体;">输入长度</span><span style="font-size:10.0pt;font-family:Consolas;">.</span></p> <p style="text-indent:21.0pt"><span style="font-size:10.0pt; font-family:宋体;">因此</span><span style="font-size:10.0pt; font-family:Consolas;">, </span><span style="font-size:10.0pt;font-family:宋体;">如果惛_?/span><span style="font-size:10.0pt;font-family:Consolas;">map</span><span style="font-size:10.0pt; font-family:宋体;">?/span><span style="font-size:10.0pt; font-family:Consolas;">, </span><span style="font-size:10.0pt;font-family:宋体;">可以?/span><span style="font-size:10.0pt;font-family:Consolas;">mapred.min.split.size</span><span style="font-size:10.0pt;font-family:宋体;">调小</span><span style="font-size:10.0pt;font-family:Consolas;">(</span><span style="font-size:10.0pt; font-family:宋体;">其实默认值即?/span><span style="font-size: 10.0pt;font-family:Consolas;">), </span><span style="font-size:10.0pt;font-family:宋体;">另外q需要把</span><span style="font-size:10.0pt;font-family:Consolas;">mapred.map.tasks</span><span style="font-size:10.0pt;font-family:宋体;">讄?</span><span style="font-size:10.0pt;font-family:Consolas;"> </span><span style="font-size:10.0pt;font-family:Consolas;"></span></p> <p style="text-indent:21.0pt"><span style="font-size:10.0pt; font-family:宋体;">如果需要减?/span><span style="font-size: 10.0pt;font-family:Consolas;">map</span><span style="font-size:10.0pt;font-family:宋体;">?/span><span style="font-size:10.0pt;font-family:Consolas;">,</span><span style="font-size:10.0pt;font-family:宋体;">可以?/span><span style="font-size:10.0pt;font-family:Consolas;">mapred.min.split.size</span><span style="font-size:10.0pt;font-family:宋体;">调大</span><span style="font-size:10.0pt;font-family:Consolas;">, </span><span style="font-size:10.0pt; font-family:宋体;">另外?/span><span style="font-size: 10.0pt;font-family:Consolas;">mapred.map.tasks</span><span style="font-size:10.0pt;font-family:宋体;">调小</span><span style="font-size:10.0pt; font-family:Consolas;">.</span></p> <p style="text-indent:21.0pt"><span style="font-size:9.0pt; font-family:宋体;color:black;">q里要特别指出的?/span><span style="font-size:10.0pt;font-family:Consolas;">FileInputFormat</span><span style="font-size:10.0pt; font-family:宋体;">会让每个输入文g臛_产生一?/span><span style="font-size:10.0pt;font-family:Consolas;">map</span><span style="font-size:10.0pt;font-family:宋体;">d</span><span style="font-size:10.0pt; font-family:Consolas;">, </span><span style="font-size:10.0pt;font-family:宋体;">因此如果你的输入目录下有许多文g</span><span style="font-size:10.0pt;font-family:Consolas;">, </span><span style="font-size:10.0pt; font-family:宋体;">而每个文仉很小</span><span style="font-size:10.0pt;font-family:Consolas;">, </span><span style="font-size:10.0pt;font-family:宋体;">例如几十</span><span style="font-size:10.0pt; font-family:Consolas;">kb, </span><span style="font-size:10.0pt;font-family:宋体;">那么每个文g都生一?/span><span style="font-size:10.0pt;font-family:Consolas;">map</span><span style="font-size:10.0pt; font-family:宋体;">会增加调度开销</span><span style="font-size: 10.0pt;font-family:Consolas;">. </span><span style="font-size:10.0pt;font-family:宋体;">作业变慢</span><span style="font-size:10.0pt;font-family:Consolas;">.</span></p> <p style="text-indent: 21pt;"><span style="font-size:10.0pt; font-family:宋体;">那么如何防止q种问题?/span><span style="font-size:10.0pt;font-family:Consolas;">? <span style="color:red">CombineFileInputFormat</span></span><span style="font-size:10.0pt;font-family:宋体;">能有效的减少</span><span style="font-size:10.0pt;font-family:Consolas;">map</span><span style="font-size:10.0pt; font-family:宋体;">数量</span><span style="font-size: 10pt; font-family: Consolas;">.</span></p><p style="text-indent:21.0pt"><span style="font-size:10.0pt; font-family:Consolas;"><br /></span></p> <div> <p><span style="font-size:14.0pt;font-family:Consolas;">FileInputFormat(</span><span style="font-size:14.0pt;font-family:宋体;">新接?/span><span style="font-size:10.0pt;font-family:Consolas;color:black;">org.apache.hadoop.mapreduce.lib.input</span><span style="font-size:14.0pt;font-family:Consolas;">)</span></p> <p style="text-indent:21.0pt"><span style="font-size:9.0pt;font-family:Consolas;">Hadoop 0.20</span><span style="font-size:9.0pt; font-family:宋体;">开始定义了一套新?/span><span style="font-size:9.0pt;font-family:Consolas;">mapreduce</span><span style="font-size:9.0pt;font-family: 宋体;">~程接口</span><span style="font-size:9.0pt; font-family:Consolas;">, </span><span style="font-size:9.0pt;font-family:宋体;">使用新的</span><span style="font-size:9.0pt;font-family:Consolas;">FileInputFormat, </span><span style="font-size: 9.0pt;font-family:宋体;">它与旧接口下?/span><span style="font-size:9.0pt;font-family:Consolas;">FileInputFormat</span><span style="font-size: 9.0pt;font-family:宋体;">主要区别在于</span><span style="font-size:9.0pt;font-family:Consolas;">, </span><span style="font-size:9.0pt; font-family:宋体;">它不再?/span><span style="font-size: 10.0pt;font-family:Consolas;">mapred.map.tasks, </span><span style="font-size:10.0pt;font-family:宋体;">而?/span><span style="font-size:10.0pt; font-family:Consolas;color:#2A00FF">mapred.max.split.size</span><span style="font-size:10.0pt;font-family:宋体;color:black;">参数代替</span><span style="font-size:10.0pt;font-family:Consolas;">goalSize, </span><span style="font-size:10.0pt; font-family:宋体;">通过</span><span style="font-size:10.0pt; font-family:Consolas;">Math.<em>max</em>(minSize, Math.<em>min</em>(maxSize, blockSize))</span><span style="font-size:10.0pt; font-family:宋体;">军_</span><span style="font-size:10.0pt; font-family:Consolas;">map</span><span style="font-size:10.0pt;font-family:宋体;">输入长度</span><span style="font-size:10.0pt;font-family:Consolas;">, </span><span style="font-size:10.0pt; font-family:宋体;">一?/span><span style="font-size:10.0pt; font-family:Consolas;">map</span><span style="font-size:10.0pt;font-family:宋体;">的输入要大于</span><span style="font-size:10.0pt;font-family:Consolas;">minSize,</span><span style="font-size:10.0pt; font-family:宋体;">于</span></p> <p><span style="font-size:10.0pt;font-family:Consolas;">Math.<em>min</em>(maxSize, blockSize).</span></p> <p><span style="font-size:10.0pt;font-family:Consolas;">    </span><span style="font-size:10.0pt;font-family:宋体;">若需增加</span><span style="font-size:10.0pt;font-family:Consolas;">map</span><span style="font-size:10.0pt;font-family:宋体;">?/span><span style="font-size:10.0pt;font-family:Consolas;">,</span><span style="font-size:10.0pt; font-family:宋体;">可以?/span><span style="font-size: 10.0pt;font-family:Consolas;">mapred.min.split.size</span><span style="font-size:10.0pt;font-family: 宋体;">调小</span><span style="font-size:10.0pt; font-family:Consolas;">,</span><span style="font-size:10.0pt;font-family:宋体;">?/span><span style="font-size:10.0pt;font-family:Consolas;color:#2A00FF">mapred.max.split.size</span><span style="font-size:10.0pt;font-family:宋体;color:black;">调大</span><span style="font-size:10.0pt;font-family:Consolas;color:black;">. </span><span style="font-size:10.0pt;font-family:宋体;color:black;">若需减少</span><span style="font-size:10.0pt;font-family:Consolas;color:black;">map</span><span style="font-size:10.0pt;font-family:宋体;color:black;">?/span><span style="font-size:10.0pt;font-family:Consolas;color:black;">,</span> <span style="font-size:10.0pt;font-family:宋体;">可以?/span><span style="font-size:10.0pt;font-family:Consolas;">mapred.min.split.size</span><span style="font-size:10.0pt;font-family:宋体;">调大</span><span style="font-size:10.0pt;font-family:Consolas;">, </span><span style="font-size:10.0pt; font-family:宋体;">q把</span><span style="font-size:10.0pt; font-family:Consolas;color:#2A00FF">mapred.max.split.size</span><span style="font-size:10.0pt;font-family:宋体;">调小</span><span style="font-size:10.0pt;font-family:Consolas;">.</span></p> </div><br /> <p><span style="font-size:14.0pt;font-family:Consolas;color:red;">CombineFileInputFormat</span></p> <p style="text-indent:21.0pt"><span style="font-size:10.0pt; font-family:宋体;">֐思义</span><span style="font-size: 10.0pt;font-family:Consolas;">, CombineFileInputFormat</span><span style="font-size:10.0pt;font-family: 宋体;">的作用是把许多文件合q作Z?/span><span style="font-size: 10.0pt;font-family:Consolas;">map</span><span style="font-size:10.0pt;font-family:宋体;">的输?/span><span style="font-size:10.0pt;font-family:Consolas;">.</span></p> <p style="text-indent:21.0pt"><span style="font-size:10.0pt; font-family:宋体;">在它之前</span><span style="font-size: 10.0pt;font-family:Consolas;">,</span><span style="font-size:10.0pt;font-family:宋体;">可以使用</span><s><span style="font-size:10.0pt;font-family:Consolas;color:black;">MultiFileInputFormat</span></s><span style="font-size:10.0pt;font-family:Consolas;color:black;">,</span><span style="font-size:10.0pt; font-family:宋体;color:black;">不过其功能太?/span><span style="font-size:10.0pt;font-family:Consolas; color:black;">, </span><span style="font-size:10.0pt; font-family:宋体;color:black;">?/span><span style="font-family: 宋体;">以文件ؓ单位Q一个文件至多分l一?/span>map<span style="font-family:宋体;">处理</span>, <span style="font-family:宋体;">如果某个目录下有许多文?/span><span>, </span><span style="font-family:宋体;">另外q有一个超大文?/span>, <span style="font-family:宋体;">处理大文件的</span>map<span style="font-family:宋体;">会严重偏?/span>.</p> <p style="text-indent:21.0pt"><span style="font-size:10.0pt;font-family:Consolas;">CombineFileInputFormat</span><span style="font-size:10.0pt; font-family:宋体;">是一个被推荐使用?/span><span style="font-size:10.0pt;font-family:Consolas;">InputFormat. </span><span style="font-size:10.0pt; font-family:宋体;">它有三个配置</span><span style="font-size: 10.0pt;font-family:Consolas;">:</span></p> <p style="margin-left:21.0pt;text-indent:21.0pt"><span style="font-size:10.0pt;font-family:Consolas;color:#2A00FF;">mapred.min.split.size.per.node</span><span style="font-size:10.0pt;font-family:宋体;color:#2A00FF;">Q?/span> <span style="font-size:10.0pt; font-family:宋体;color:#2A00FF;">一个节点上</span><span style="font-size:10.0pt;font-family:Consolas; color:#2A00FF;">split</span><span style="font-size:10.0pt; font-family:宋体;color:#2A00FF;">的至的大小</span></p> <p style="margin-left:21.0pt;text-indent:21.0pt"><span style="font-size:10.0pt;font-family:Consolas;color:#2A00FF;">mapred.min.split.size.per.rack   </span><span style="font-size:10.0pt; font-family:宋体;color:#2A00FF;">一个交换机?/span><span style="font-size:10.0pt;font-family:Consolas; color:#2A00FF;">split</span><span style="font-size:10.0pt; font-family:宋体;color:#2A00FF;">臛_的大?/span></p> <p style="margin-left:21.0pt;text-indent:21.0pt"><span style="font-size:10.0pt;font-family:Consolas;color:#2A00FF;">mapred.max.split.size             </span><span style="font-size: 10.0pt;font-family:宋体;color:#2A00FF;">一?/span><span style="font-size:10.0pt;font-family:Consolas;color:#2A00FF;">split</span><span style="font-size:10.0pt;font-family:宋体;color:#2A00FF;">最大的大小</span></p> <p style="text-indent:21.0pt"><span style="font-size:10.0pt; font-family:宋体;">它的主要思\是把输入目录下的大文件分成多?/span><span style="font-size:10.0pt;font-family:Consolas;">map</span><span style="font-size:10.0pt;font-family:宋体;">的输?/span><span style="font-size:10.0pt; font-family:Consolas;">, </span><span style="font-size:10.0pt;font-family:宋体;">q合q小文g</span><span style="font-size:10.0pt;font-family:Consolas;">, </span><span style="font-size:10.0pt; font-family:宋体;">做ؓ一?/span><span style="font-size: 10.0pt;font-family:Consolas;">map</span><span style="font-size:10.0pt;font-family:宋体;">的输?/span><span style="font-size:10.0pt;font-family:Consolas;">. </span><span style="font-size:10.0pt;font-family:宋体;">具体的原理是下述三步</span><span style="font-size:10.0pt;font-family:Consolas;">: </span></p> <p style="text-indent:21.0pt"><span style="font-size:10.0pt;font-family:Consolas;">1.</span><span style="font-size:10.0pt;font-family:宋体;">Ҏ输入目录下的每个文g</span><span style="font-size: 10.0pt;font-family:Consolas;">,</span><span style="font-size:10.0pt;font-family:宋体;">如果光度超q?/span><span style="font-size:10.0pt;font-family:Consolas;">mapred.max.split.size,</span><span style="font-size:10.0pt;font-family:宋体;">?/span><span style="font-size:10.0pt;font-family:Consolas;">block</span><span style="font-size:10.0pt; font-family:宋体;">为单位分成多?/span><span style="font-size: 10.0pt;font-family:Consolas;">split(</span><span style="font-size:10.0pt;font-family:宋体;">一?/span><span style="font-size:10.0pt;font-family:Consolas;">split</span><span style="font-size:10.0pt;font-family:宋体;">是一?/span><span style="font-size:10.0pt;font-family:Consolas;">map</span><span style="font-size:10.0pt; font-family:宋体;">的输?/span><span style="font-size: 10.0pt;font-family:Consolas;">),</span><span style="font-size:10.0pt;font-family:宋体;">每个</span><span style="font-size:10.0pt;font-family:Consolas;">split</span><span style="font-size:10.0pt;font-family:宋体;">的长度都大于</span><span style="font-size:10.0pt;font-family:Consolas;">mapred.max.split.size, </span><span style="font-size:10.0pt;font-family:宋体;">因ؓ?/span><span style="font-size:10.0pt;font-family:Consolas;">block</span><span style="font-size:10.0pt; font-family:宋体;">为单?/span><span style="font-size: 10.0pt;font-family:Consolas;">, </span><span style="font-size:10.0pt;font-family:宋体;">因此也会大于</span><span style="font-size:10.0pt;font-family:Consolas;">blockSize, </span><span style="font-size:10.0pt;font-family:宋体;">此文件剩下的长度如果大于</span><span style="font-size:10.0pt;font-family:Consolas;">mapred.min.split.size.per.node, </span><span style="font-size:10.0pt;font-family:宋体;">则生成一?/span><span style="font-size:10.0pt;font-family:Consolas;">split, </span><span style="font-size:10.0pt; font-family:宋体;">否则先暂时保?/span><span style="font-size: 10.0pt;font-family:Consolas;">.</span></p> <p style="text-indent:21.0pt"><span style="font-size:10.0pt;font-family:Consolas;">2. </span><span style="font-size:10.0pt;font-family:宋体;">现在剩下的都是一些长度效短的片</span><span style="font-size: 10.0pt;font-family:Consolas;">,</span><span style="font-size:10.0pt;font-family:宋体;">把每?/span><span style="font-size:10.0pt;font-family:Consolas;">rack</span><span style="font-size:10.0pt;font-family:宋体;">下碎片合q?/span><span style="font-size:10.0pt;font-family:Consolas;">, </span><span style="font-size:10.0pt; font-family:宋体;">只要长度过</span><span style="font-size: 10.0pt;font-family:Consolas;">mapred.max.split.size</span><span style="font-size:10.0pt;font-family: 宋体;">合q成一?/span><span style="font-size:10.0pt; font-family:Consolas;">split, </span><span style="font-size:10.0pt;font-family:宋体;">最后如果剩下的片?/span><span style="font-size:10.0pt;font-family:Consolas;">mapred.min.split.size.per.rack</span><span style="font-size:10.0pt;font-family:宋体;">?/span><span style="font-size:10.0pt;font-family:Consolas;">, </span><span style="font-size:10.0pt; font-family:宋体;">合q成一?/span><span style="font-size: 10.0pt;font-family:Consolas;">split, </span><span style="font-size:10.0pt;font-family:宋体;">否则暂时保留</span><span style="font-size:10.0pt;font-family:Consolas;">.</span></p> <p style="text-indent:21.0pt"><span style="font-size:10.0pt;font-family:Consolas;">3. </span><span style="font-size:10.0pt;font-family:宋体;">把不?/span><span style="font-size:10.0pt; font-family:Consolas;">rack</span><span style="font-size:10.0pt;font-family:宋体;">下的片合ƈ</span><span style="font-size:10.0pt;font-family:Consolas;">, </span><span style="font-size:10.0pt; font-family:宋体;">只要长度过</span><span style="font-size: 10.0pt;font-family:Consolas;">mapred.max.split.size</span><span style="font-size:10.0pt;font-family: 宋体;">合q成一?/span><span style="font-size:10.0pt; font-family:Consolas;">split, </span><span style="font-size:10.0pt;font-family:宋体;">剩下的碎片无论长?/span><span style="font-size:10.0pt;font-family:Consolas;">, </span><span style="font-size:10.0pt; font-family:宋体;">合ƈ成一?/span><span style="font-size: 10.0pt;font-family:Consolas;">split.</span></p> <p style="text-indent:21.0pt"><span style="font-size:10.0pt; font-family:宋体;">举例</span><span style="font-size:10.0pt; font-family:Consolas;">: mapred.max.split.size=1000</span></p> <p style="text-indent:21.0pt"><span style="font-size:10.0pt;font-family:Consolas;">     mapred.min.split.size.per.node=300</span></p> <p style="text-indent:21.0pt;"><span style="font-size:10.0pt;font-family:Consolas;">      mapred.min.split.size.per.rack=100</span></p> <p style="text-indent:21.0pt;"><span style="font-size:10.0pt;font-family:宋体;">输入目录下五个文?/span><span style="font-size:10.0pt;font-family:Consolas;">,rack1</span><span style="font-size:10.0pt; font-family:宋体;">下三个文?/span><span style="font-size: 10.0pt;font-family:Consolas;">,</span><span style="font-size:10.0pt;font-family:宋体;">长度?/span><span style="font-size:10.0pt;font-family:Consolas;">2050,1499,10, rack2</span><span style="font-size:10.0pt;font-family:宋体;">下两个文?/span><span style="font-size:10.0pt;font-family:Consolas;">,</span><span style="font-size:10.0pt; font-family:宋体;">长度?/span><span style="font-size: 10.0pt;font-family:Consolas;">1010,80. </span><span style="font-size:10.0pt;font-family:宋体;">另外</span><span style="font-size:10.0pt;font-family:Consolas;">blockSize</span><span style="font-size:10.0pt;font-family:宋体;">?/span><span style="font-size:10.0pt;font-family:Consolas;">500.</span></p> <p style="text-indent:21.0pt;"><span style="font-size:10.0pt;font-family:宋体;">l过W一?/span><span style="font-size:10.0pt;font-family:Consolas;">, </span><span style="font-size:10.0pt; font-family:宋体;">生成五个</span><span style="font-size: 10.0pt;font-family:Consolas;">split: 1000,1000,1000,499,1000. </span><span style="font-size:10.0pt; font-family:宋体;">剩下的碎片ؓ</span><span style="font-size: 10.0pt;font-family:Consolas;">rack1</span><span style="font-size:10.0pt;font-family:宋体;">?/span><span style="font-size:10.0pt;font-family:Consolas;">:50,10; rack2</span><span style="font-size:10.0pt;font-family:宋体;">?/span><span style="font-size:10.0pt;font-family:Consolas;">10:80</span></p> <p style="text-indent:21.0pt;"><span style="font-size:10.0pt;font-family:宋体;">׃两个</span><span style="font-size:10.0pt;font-family:Consolas;">rack</span><span style="font-size:10.0pt; font-family:宋体;">下的片和都不超q?/span><span style="font-size:10.0pt;font-family:Consolas;">100, </span><span style="font-size:10.0pt;font-family: 宋体;">所以经q第二步</span><span style="font-size:10.0pt; font-family:Consolas;">, split</span><span style="font-size:10.0pt;font-family:宋体;">和碎片都没有变化</span><span style="font-size:10.0pt;font-family:Consolas;">.</span></p> <p style="text-indent:21.0pt;"><span style="font-size:10.0pt;font-family:宋体;">W三?/span><span style="font-size:10.0pt;font-family:Consolas;">,</span><span style="font-size:10.0pt; font-family:宋体;">合ƈ四个片成一?/span><span style="font-size:10.0pt;font-family:Consolas;">split, </span><span style="font-size:10.0pt;font-family: 宋体;">长度?/span><span style="font-size:10.0pt; font-family:Consolas;">150.</span></p> <p style="text-indent:21.0pt"> </p> <p style="text-indent:21.0pt"><span style="font-size:10.0pt; font-family:宋体;">如果要减?/span><span style="font-size: 10.0pt;font-family:Consolas;">map</span><span style="font-size:10.0pt;font-family:宋体;">数量</span><span style="font-size:10.0pt;font-family:Consolas;">, </span><span style="font-size:10.0pt;font-family:宋体;">可以调大</span><span style="font-size:10.0pt;font-family:Consolas;">mapred.max.split.size, </span><span style="font-size:10.0pt;font-family:宋体;">否则调小卛_</span><span style="font-size:10.0pt;font-family:Consolas;">.</span></p> <p style="text-indent: 21pt;"><span style="font-size:10.0pt; font-family:宋体;color:black;">其特Ҏ</span><span style="font-size:10.0pt;font-family:Consolas; color:black;">: </span><span style="font-size:10.0pt; font-family:宋体;color:black;">一个块臛_作ؓ一?/span><span style="font-size:10.0pt;font-family:Consolas; color:black;">map</span><span style="font-size:10.0pt; font-family:宋体;color:black;">的输入,一个文件可能有多个块,一个文件可能因为块多分l做Z?/span><span style="font-size:10.0pt;font-family:Consolas;color:black;">map</span><span style="font-size: 10.0pt;font-family:宋体;color:black;">的输入,</span> <span style="font-size:10.0pt; font-family:宋体;color:black;">一?/span><span style="font-size:10.0pt;font-family:Consolas; color:black;">map</span><span style="font-size: 10pt; font-family: 宋体; color: black;">可能处理多个块,可能处理多个文g?/span></p><p style="text-indent: 21pt;"><span style="font-size: 10pt; font-family: 宋体; color: black;"><br /></span></p><p style="text-indent: 21pt;"><span style="font-size: 10pt; font-family: 宋体; color: black;"></span></p><div> <p><span style="font-size:10.0pt;font-family:宋体;color:black;"></span></p><div> <p><span style="font-size:14.0pt;font-family:宋体;color:black;">~写自己?/span><span style="font-size:14.0pt; font-family:Consolas;color:black;">InputFormat</span></p> </div><span style="font-size:10.0pt; font-family:Consolas;color:black;"></span><p> </p> <p><span style="font-size:10.0pt;font-family:Consolas;color:black;">    </span><span style="font-size:10.0pt; font-family:宋体;color:black;">待箋</span></p> </div><br /><p> </p> </div><img src ="http://www.tkk7.com/shenh062326/aggbug/382115.html" width = "1" height = "1" /><br><br><div align=right><a style="text-decoration:none;" href="http://www.tkk7.com/shenh062326/" target="_blank">俞灵</a> 2012-07-03 22:17 <a href="http://www.tkk7.com/shenh062326/archive/2012/07/03/hadoop.html#Feedback" target="_blank" style="text-decoration:none;">发表评论</a></div>]]></description></item><item><title>Yarn(MR2)上的应用汇?/title><link>http://www.tkk7.com/shenh062326/archive/2012/06/03/379845.html</link><dc:creator>俞灵</dc:creator><author>俞灵</author><pubDate>Sun, 03 Jun 2012 03:43:00 GMT</pubDate><guid>http://www.tkk7.com/shenh062326/archive/2012/06/03/379845.html</guid><wfw:comment>http://www.tkk7.com/shenh062326/comments/379845.html</wfw:comment><comments>http://www.tkk7.com/shenh062326/archive/2012/06/03/379845.html#Feedback</comments><slash:comments>0</slash:comments><wfw:commentRss>http://www.tkk7.com/shenh062326/comments/commentRss/379845.html</wfw:commentRss><trackback:ping>http://www.tkk7.com/shenh062326/services/trackbacks/379845.html</trackback:ping><description><![CDATA[<div> <p>Yarn<span style="font-family:宋体;">做ؓ</span>hadoop<span style="font-family:宋体;">下一代集资源管理和调度q_</span>, <span style="font-family:宋体;">其上能支持多U计框?/span>, <span style="font-family:宋体;">本文q要介l一下这些计框?/span>.</p> <p><br /></p> <p style="margin-left:18.0pt;text-indent:-18.0pt;"><span><span>1.<span style="font:7.0pt "Times New Roman"">       </span></span></span>MapReduce</p> <p style="margin-left:18.0pt;text-indent:0cm;"><span style="font-family:宋体;">首先是大家熟悉的</span>mapreduce, <span style="font-family:宋体;">?/span>MR2<span style="font-family:宋体;">之前</span>, hadoop<span style="font-family:宋体;">包括</span>HDFS<span style="font-family:宋体;">?/span>mapreduce, <span style="font-family:宋体;">做ؓ</span>hadoop<span style="font-family:宋体;">上唯一的分布式计算框架</span>, <span style="font-family:宋体;">其优Ҏ用户可以很方便的~写分布式计程?/span>, <span style="font-family:宋体;">q支持许多的应用</span>, <span style="font-family:宋体;">?/span>hive, mahout, pig<span style="font-family:宋体;">{?/span>. <span style="font-family:宋体;">但是其缺Ҏ无法充分利用集群资源</span>, <span style="font-family:宋体;">不支?/span>DAG, <span style="font-family:宋体;">q代式计等</span>. <span style="font-family:宋体;">Z解决q些问题</span>, yahoo<span style="font-family:宋体;">提出?/span>Yarn (next generation mapreduce), <span style="font-family: 宋体;">一个分布式集群集群资源理和调度^?/span>. <span style="font-family:宋体;">q样除了</span>mapreduce<span style="font-family:宋体;">?/span>, <span style="font-family:宋体;">q可以支持各U计框?/span>.</p> <p style="margin-left:18.0pt;text-indent:-18.0pt;"><span><span>2.<span style="font:7.0pt "Times New Roman"">       </span></span></span>Spark</p> <p style="margin-left:18.0pt;text-indent:0cm;">Spark<span style="font-family:宋体;">是一U与</span>mapreduce<span style="font-family:宋体;">怼的开源计框?/span>, <span style="font-family:宋体;">不同之处在于</span>Spark<span style="font-family:宋体;">在某些工作负载方面表现更?/span>, <span style="font-family:宋体;">因ؓ它用了内存分布式数据集</span>, <span style="font-family:宋体;">另外除了提供交互式查询外</span>, <span style="font-family:宋体;">它还可以优化q代工作负蝲</span>.</p> <p style="margin-left:18.0pt;text-indent:-18.0pt;"><span><span>3.<span style="font:7.0pt "Times New Roman"">       </span></span></span>Apache HAMA</p> <p style="margin-left:18.0pt"><strong>Apache Hama 是一个运行在HDFS上的</strong>BSP(Bulk Synchronous Parallel大容量同步ƈ? 计算框架, 主要针对大规模科学计?如矩? 囑փ, |络法{?当前它有一下功?</p> <ul type="disc"><li style=" text-align:left;"><span style="font-family:宋体;">作业提交和管理接?/span></li><li style=" text-align:left;"><span style="font-family:宋体;">单节点上q行多个d</span></li><li style=" text-align:left;"><span style="font-family:宋体;">输入</span>/<span style="font-family:宋体;">输出格式?/span></li><li style=" text-align:left;"><span style="font-family:宋体;">备䆾恢复</span></li><li style=" text-align:left;"><span style="font-family:宋体;">支持通过</span><a >Apache Whirr</a><span style="font-family:宋体;">q行在云?/span></li><li style=" text-align:left;"><span style="font-family:宋体;">支持?/span>Yarn<span style="font-family:宋体;">一赯?/span></li></ul> <p style="margin-left:18.0pt;text-indent:-18.0pt;"><span><span>4.<span style="font:7.0pt "Times New Roman"">       </span></span></span>Apache Giraph</p> <p style="margin-left:18.0pt;text-indent:0cm;"><span style="font-family:宋体;">囑փ处理q_上运行这大型法</span>(<span style="font-family:宋体;">?/span>page rank, shared connections, personalization-based popularity <span style="font-family:宋体;">{?/span>)<span style="font-family:宋体;">已经很流?/span>, Giraph<span style="font-family:宋体;">采用</span>BSP<span style="font-family:宋体;">模型Q?/span><span>bulk-synchronous parallel model</span><span style="font-family:宋体;">Q,可用于等q代cȝ法?/span></p> <p style="margin-left:18.0pt;text-indent:-18.0pt;"><span><span>5.<span style="font:7.0pt "Times New Roman"">       </span></span></span>Open MPI</p> <p style="margin-left:18.0pt;text-indent:0cm;"><span style="font-family:宋体;">q是一个高性能计算函数库,通常?/span>HPC<span style="font-family:宋体;">Q?/span>High Performance Computing<span style="font-family:宋体;">Q中采用Q与</span>MapReduce<span style="font-family:宋体;">相比Q其性能更高Q用户可控性更强,但编E复杂,定w性差Q可以说Q各有所长,在实际应用中Q针对不?/span> <span style="font-family:宋体;">该应用会采用</span>MPI<span style="font-family:宋体;">或?/span>MapReduce<span style="font-family:宋体;">?/span></p> <p style="margin-left:18.0pt;text-indent:-18.0pt;"><span><span>6.<span style="font:7.0pt "Times New Roman"">       </span></span></span>Apache HBase</p> <p style="margin-left: 18pt; text-indent: 0cm;">HBase<span style="font-family:宋体;">是一?/span>hadoop<span style="font-family:宋体;">数据?/span>, <span style="font-family:宋体;">其特Ҏ分布式,可扩展的Q存储大数据。当有需要随?/span>,<span style="font-family:宋体;">实时d的大数据? </span><span style="font-family:宋体;">使用HBase</span><span><span style="font-family: 宋体;">很适合</span>. <br /></span></p><p style="margin-left: 18pt; text-indent: 0cm;"><span>本文参?</span></p><p style="margin-left: 18pt; text-indent: 0cm;"><span><div>http://wiki.apache.org/hadoop/PoweredByYarn<br /><div>http://www.oschina.net/p/open+mpi</div></div></span></p><p style="margin-left:18.0pt;text-indent:0cm;"><span><div>http://incubator.apache.org/hama/<br /><div>http://incubator.apache.org/giraph/</div></div></span></p> <p style="margin-left:18.0pt;text-indent:0cm;"><a >http://hbase.apache.org/</a></p> </div><img src ="http://www.tkk7.com/shenh062326/aggbug/379845.html" width = "1" height = "1" /><br><br><div align=right><a style="text-decoration:none;" href="http://www.tkk7.com/shenh062326/" target="_blank">俞灵</a> 2012-06-03 11:43 <a href="http://www.tkk7.com/shenh062326/archive/2012/06/03/379845.html#Feedback" target="_blank" style="text-decoration:none;">发表评论</a></div>]]></description></item><item><title>VirtualBox虚拟安装Ubuntu 不能讄宽屏的解军_?/title><link>http://www.tkk7.com/shenh062326/archive/2012/05/24/379026.html</link><dc:creator>俞灵</dc:creator><author>俞灵</author><pubDate>Thu, 24 May 2012 06:46:00 GMT</pubDate><guid>http://www.tkk7.com/shenh062326/archive/2012/05/24/379026.html</guid><wfw:comment>http://www.tkk7.com/shenh062326/comments/379026.html</wfw:comment><comments>http://www.tkk7.com/shenh062326/archive/2012/05/24/379026.html#Feedback</comments><slash:comments>0</slash:comments><wfw:commentRss>http://www.tkk7.com/shenh062326/comments/commentRss/379026.html</wfw:commentRss><trackback:ping>http://www.tkk7.com/shenh062326/services/trackbacks/379026.html</trackback:ping><description><![CDATA[<div><div> <h1><span style="font-size: 12pt;">转蝲 </span><div><span style="font-size: 12pt;">http://fujun.sinaapp.com/2011/11/02/68.html</span></div></h1><p> W一步,打开l端Q看看你的显卡Ubuntu能认出多显C分辨率讄Q输入命?br /> <code><br /> wufujun@wufujun-VirtualBox:~$ xrandr<br /> </code></p> <p>pȝl出的结?/p> <p>Screen 0: minimum 64 x 64, current 1024 x 768, maximum 32000 x 32000<br /> VBOX0 connected 1024×768+0+0 0mm x 0mm<br /> 1024×768 60.0 + 60.0<br /> 1600×1200 60.0<br /> 1440×1050 60.0<br /> 1280×960 60.0<br /> 800×600 60.0<br /> 640×480 60.0</p> <p>q里可以看到Q没?6Q?的的分L率设|?/p> <p>W二步,用cvt命o试1368×768是否可用<br /> <code><br /> wufujun@wufujun-VirtualBox:~$ cvt 1368 768<br /> </code><br /> 昄l果如下<br /> # 1368×768 59.88 Hz (CVT) hsync: 47.79 kHz; pclk: 85.86 MHz<br /> Modeline “1368x768_60.00″ 85.25 1368 1440 1576 1784 768 771 781 798 -hsync +vsync</p> <p>从这个结果里可以刎ͼ16Q?的分辨率是可以用?/p> <p>W三?输入<br /> <code><br /> wufujun@wufujun-VirtualBox:~$ sudo xrandr --newmode "1368x768" 85.86 1368 1440 1576 1784 768 771 781 798 -hsync +vsync<br /> </code><br /> 建立新的分L率模?368×768Q把刚才cvt得到的数据写q参?/p> <p>W四?l箋输入<br /> <code><br /> sudo xrandr --addmode VBOX0 "1368x768"<br /> </code><br /> l当前显C器VBOX0增加1368×768分L率设|?/p> <p>做完以上操作后,可以?#8221;昄“讄里面看到昄的分辨率列表中多了一?1368×768(16:9)的选项。选中q个选项Q点d用,完美的宽屏显C回来了Q?/p> <p>l过试Q上面的Ҏ做完以后Q每ơ注销后就又变回了4Q?的比例,而且会有的报错,没办法,按上面的修改完毕后,q要再修改一?etc/X11/xorg.confq个文gQ这个配|文件在现在的版里已l取消了Q所以需要我们新Z?/p> <p><code><br /> $ sudo gedit /etc/X11/xorg.conf<br /> </code></p> <p>~辑内容为:<br /> <code><br /> Section "Device"<br /> Identifier "Configured Video Device"<br /> EndSection</code></p><code> <p>Section "Monitor"<br /> Identifier "Configured Monitor"<br /> Modeline "1368x768_60.00" 85.86 1368 1440 1584 1800 768 769 772 795 -HSync +Vsync<br /> EndSection</p> </code><p><code>Section "Screen"<br /> Identifier "Default Screen"<br /> Monitor "Configured Monitor"<br /> Device "Configured Video Device"<br /> SubSection "Display"<br /> Modes "1368x768@60"<br /> EndSubSection<br /> EndSection<br /> </code><br /> 其中 Modeline “1368x768_60.00″ 85.86 1368 1440 1584 1800 768 769 772 795 -HSync +Vsync 是? cvt 1368 768得到的倹{也可以? gtf 1368 768 60命o来得到这个Modeline的|q个命o中,1368 768是分辨率 60为刷新率Q用q个命o得到的值可能会更ؓ准确一些?br /> <code><br /> SubSection "Display"<br /> Modes "1368x768@60"<br /> EndSubSection<br /> </code><br /> q段是设|默认显C最佛_辨率?/p> <p>注意q段文g中的一些规?/p> <p>Section “Device”区块中,Identifier指定了显卡的唯一名称Q这个名U可以随便取Q但一定要与Section “Screen”区块中的device选项中的名称相同。在Section “Monitor”区块中,Identifier指定了显C器的唯一名称Q这个名U可以随便取Q但一定要与Section “Screen”区块中的Monitor选项中所指定的名U相同。Section “Screen”区块中的Identifier选项Q指定了q个昑֍与显C器相结合的唯一名称。这个名UC可以随便取的。这个名U需要与Section “ServerLayout” 区块中的名称相同。这个Section “ServerLayout” 区块我们一般不必编?/p> </div></div><img src ="http://www.tkk7.com/shenh062326/aggbug/379026.html" width = "1" height = "1" /><br><br><div align=right><a style="text-decoration:none;" href="http://www.tkk7.com/shenh062326/" target="_blank">俞灵</a> 2012-05-24 14:46 <a href="http://www.tkk7.com/shenh062326/archive/2012/05/24/379026.html#Feedback" target="_blank" style="text-decoration:none;">发表评论</a></div>]]></description></item><item><title>实现自定义的classLoader加蝲classpath中的classhttp://www.tkk7.com/shenh062326/archive/2012/05/20/yuling20120520.html俞灵俞灵Sun, 20 May 2012 11:43:00 GMThttp://www.tkk7.com/shenh062326/archive/2012/05/20/yuling20120520.htmlhttp://www.tkk7.com/shenh062326/comments/378623.htmlhttp://www.tkk7.com/shenh062326/archive/2012/05/20/yuling20120520.html#Feedback1http://www.tkk7.com/shenh062326/comments/commentRss/378623.htmlhttp://www.tkk7.com/shenh062326/services/trackbacks/378623.html阅读全文

俞灵 2012-05-20 19:43 发表评论
]]>
High Availability for the HDFS NamenodeQ翻译)http://www.tkk7.com/shenh062326/archive/2012/03/24/yuling111.html俞灵俞灵Sat, 24 Mar 2012 13:38:00 GMThttp://www.tkk7.com/shenh062326/archive/2012/03/24/yuling111.htmlhttp://www.tkk7.com/shenh062326/comments/372624.htmlhttp://www.tkk7.com/shenh062326/archive/2012/03/24/yuling111.html#Feedback2http://www.tkk7.com/shenh062326/comments/commentRss/372624.htmlhttp://www.tkk7.com/shenh062326/services/trackbacks/372624.html阅读全文

俞灵 2012-03-24 21:38 发表评论
]]>
JAVA代码~写?0条徏?http://www.tkk7.com/shenh062326/archive/2011/11/28/364989.html俞灵俞灵Mon, 28 Nov 2011 06:34:00 GMThttp://www.tkk7.com/shenh062326/archive/2011/11/28/364989.htmlhttp://www.tkk7.com/shenh062326/comments/364989.htmlhttp://www.tkk7.com/shenh062326/archive/2011/11/28/364989.html#Feedback0http://www.tkk7.com/shenh062326/comments/commentRss/364989.htmlhttp://www.tkk7.com/shenh062326/services/trackbacks/364989.html本文转自:
http://blog.csdn.net/zhouysh/article/details/304767

JAVA代码~写?0条徏?
(1) cd首字母应该大写。字Dc方法以及对象(句柄Q的首字母应写。对于所有标识符Q其中包含的所有单词都应紧靠在一P而且大写中间单词的首字母。例如:
ThisIsAClassName
thisIsMethodOrFieldName
若在定义中出C常数初始化字W,则大写static final基本cd标识W中的所有字母。这样便可标志出它们属于~译期的常数?
Java包(PackageQ属于一U特D情况:它们全都是小写字母,即便中间的单词亦是如此。对于域名扩展名Uͼ如comQorgQnet或者edu{,全部都应写Q这也是Java 1.1和Java 1.2的区别之一Q?

(2) Z常规用途而创Z个类Ӟ请采?l典形式"Qƈ包含对下q元素的定义Q?

equals()
hashCode()
toString()
clone()Qimplement CloneableQ?
implement Serializable

(3) 对于自己创徏的每一个类Q都考虑|入一个main()Q其中包含了用于试那个cȝ代码。ؓ使用一个项目中的类Q我们没必要删除试代码。若q行了Q何Ş式的改动Q可方便地返回测试。这些代码也可作为如何用类的一个示例用?

(4) 应将Ҏ设计成简要的、功能性单元,用它描述和实C个不q箋的类接口部分。理x况下Q方法应明扼要。若长度很大Q可考虑通过某种方式其分割成较短的几个Ҏ。这样做也便于类内代码的重复使用Q有些时候,Ҏ必须非常大,但它们仍应只做同L一件事情)?

(5) 设计一个类Ӟ误w处Cؓ客户E序员考虑一下(cȝ使用Ҏ应该是非常明的Q。然后,再设w处Cؓ理代码的h考虑一下(预计有可能进行哪些Ş式的修改Q想想用什么方法可把它们变得更单)?
(6) 使类可能短精悍,而且只解决一个特定的问题。下面是对类设计的一些徏议:
■一个复杂的开兌句:考虑采用"多Ş"机制
■数量众多的方法涉及到cd差别极大的操作:考虑用几个类来分别实?
■许多成员变量在特征上有很大的差别Q考虑使用几个c?

(7) 让一切东襉K可能地"U有"--private。可使库的某一部分"公共?Q一个方法、类或者一个字D늭{)Q就永远不能把它拿出。若拿出Q就?能破坏其他h现有的代码,使他们不得不重新~写和设计。若只公布自己必d布的Q就可放心大胆地改变其他M东西。在多线E环境中Q隐U是特别重要的一?因素--只有private字段才能在非同步使用的情况下受到保护?

(8) 谨惕"巨大对象l合?。对一些习惯于序~程思维、且初涉OOP领域的新手,往往喜欢先写一个顺序执行的E序Q再把它嵌入一个或两个巨大的对象里。根据编E原理,对象表达的应该是应用E序的概念,而非应用E序本n?

(9) 若不得已q行一些不太雅观的~程Q至应该把那些代码|于一个类的内部?

(10) M时候只要发现类与类之间l合得非常紧密,需要考虑是否采用内部c,从而改善编码及l护工作Q参见第14?4.1.2节?用内部类改进代码"Q?

(11) 可能细致地加上注释Qƈ用javadoc注释文档语法生成自己的程序文档?

(12) 避免使用"术数字"Q这些数字很难与代码很好地配合。如以后需要修改它Q无疑会成ؓ一场噩梦,因ؓҎ不知?100"到底是指"数组大小"q是"其他 全然不同的东?。所以,我们应创Z个常敎ͼqؓ其用具有说服力的描q性名Uͼq在整个E序中都采用常数标识W。这样可使程序更易理解以及更易维护?

(13) 涉及构徏器和异常的时候,通常希望重新丢弃在构建器中捕LM异常--如果它造成了那个对象的创徏p|。这样一来,调用者就不会以ؓ那个对象已正地创徏Q从而盲目地l箋?

(14) 当客L序员用完对象以后Q若你的c要求进行Q何清除工作,可考虑清除代码置于一个良好定义的Ҏ里,采用cM于cleanup()q样的名字,明确?明自q用途。除此以外,可在cd攄一个booleanQ布)标记Q指出对象是否已被清除。在cȝfinalize()Ҏ里,L定对象已被清除, q已丢弃了从RuntimeExceptionl承的一个类Q如果还没有的话Q,从而指Z个编E错误。在采取象这LҎ之前Q请定 finalize()能够在自qpȝ中工作(可能需要调用System.runFinalizersOnExit(true)Q从而确保这一行ؓQ?

(15) 在一个特定的作用域内Q若一个对象必L除(非由垃圾攉机制处理Q,请采用下q方法:初始化对象;若成功,则立卌入一个含有finally从句的try块,开始清除工作?

(16) 若在初始化过E中需要覆盖(取消Qfinalize()Q请C调用super.finalize()Q若Object属于我们的直接超c,则无此必 要)。在对finalize()q行覆盖的过E中Q对super.finalize()的调用应属于最后一个行动,而不应是W一个行动,q样可确保在需?基础cȝ件的时候它们依然有效?

(17) 创徏大小固定的对象集合时Q请它们传输至一个数l(若准备从一个方法里q回q个集合Q更应如此操作)。这样一来,我们可享受到数l在~译期进行类型检查的好处。此外,Z用它们,数组的接收者也许ƈ不需要将对象"造型"到数l里?

(18) 量使用interfacesQ不要用abstractcR若已知某样东西准备成ؓ一个基c,那么W一个选择应是其变成一个interfaceQ接 口)。只有在不得不用方法定义或者成员变量的时候,才需要将其变成一个abstractQ抽象)cR接口主要描qC客户希望做什么事情,而一个类则致?于(或允许)具体的实施细节?

(19) 在构建器内部Q只q行那些对象设为正状态所需的工作。尽可能地避免调用其他方法,因ؓ那些Ҏ可能被其他h覆盖或取消,从而在构徏q程中生不可预知的l果Q参见第7章的详细说明Q?

(20) 对象不应只是单地容纳一些数据;它们的行Z应得到良好的定义?

(21) 在现成类的基上创建新cLQ请首先选择"新徏"?创作"。只有自q设计要求必须l承Ӟ才应考虑q方面的问题。若在本来允许新建的场合使用了承,则整个设计会变得没有必要地复杂?

(22) 用承及Ҏ覆盖来表C为间的差异,而用字段表示状态间的区别。一个非常极端的例子是通过对不同类的承来表示颜色Q这是绝对应该避免的Q应直接使用一?颜色"字段?

(23) 为避免编E时遇到ȝQ请保证在自q路径指到的Q何地方,每个名字都仅对应一个类。否则,~译器可能先扑ֈ同名的另一个类Qƈ报告出错消息。若怀疑自qCc\径问题,误试在c\径的每一个v点,搜烦一下同名的.class文g?

(24) 在Java 1.1 AWT中用事?适配?Ӟ特别Ҏ到一个陷阱。若覆盖了某个适配器方法,同时拼写Ҏ没有特别讲究Q最后的l果是新添加一个方法,而不是覆盖现 成方法。然而,׃q样做是完全合法的,所以不会从~译器或q行期系l获得Q何出错提C?-只不q代码的工作变得不正常了?

(25) 用合理的设计Ҏ消除"伪功?。也是_假若只需要创建类的一个对象,׃要提前限制自׃用应用程序,q加上一?只生成其中一?注释。请考虑?其封装成一?独生?的Ş式。若在主E序里有大量散ؕ的代码,用于创徏自己的对象,误虑采纳一U创造性的ҎQ将些代码封装v来?

(26) 警惕"分析瘫痪"。请CQ无论如何都要提前了解整个项目的状况Q再去考察其中的细节。由于把握了全局Q可快速认识自己未知的一些因素,防止在考察l节的时候陷?死逻辑"中?

(27) 警惕"q早优化"。首先让它运行v来,再考虑变得更快--但只有在自己必须q样做、而且l证实在某部分代码中的确存在一个性能瓉的时候,才应q行优化?除非用专门的工具分析瓉Q否则很有可能是在浪费自q旉。性能提升的隐含代h自己的代码变得难于理解,而且难于l护?

(28) 误住,阅读代码的时间比写代码的旉多得多。思\清晰的设计可获得易于理解的程序,但注释、细致的解释以及一些示例往往h不可估量的h倹{无论对你自 己,q是对后来的人,它们都是相当重要的。如Ҏ仍有怀疑,那么误惌p图从联机Java文档里找出有用信息时到的挫折,q样或许能将你说服?

(29) 如认己已q行了良好的分析、设计或者实施,那么L微更换一下思维角度。试试邀请一些外来h?-q不一定是专家Q但可以是来自本公司其他部门的h?请他们用完全新鲜的眼光考察你的工作Q看看是否能扑և你一度熟视无睹的问题。采取这U方式,往往能在最适合修改的阶D|Z些关键性的问题Q避免品发?后再解决问题而造成的金钱及_֊斚w的损失?

(30) 良好的设计能带来最大的回报。简a之,对于一个特定的问题Q通常会花较长的时间才能找CU最恰当的解x案。但一旦找C正确的方法,以后的工作就L 多了Q再也不用经历数时、数天或者数月的痛苦挣扎。我们的努力工作会带来最大的回报Q甚x可估量)。而且׃自己倾注了大量心血Q最l获得一个出色的 设计ҎQ成功的快感也是令h心动的。坚持抵制草草完工的诱惑--那样做往往得不偿失


俞灵 2011-11-28 14:34 发表评论
]]>
大象的崛PHadoop七年发展风雨?/title><link>http://www.tkk7.com/shenh062326/archive/2011/11/21/364390.html</link><dc:creator>俞灵</dc:creator><author>俞灵</author><pubDate>Mon, 21 Nov 2011 01:04:00 GMT</pubDate><guid>http://www.tkk7.com/shenh062326/archive/2011/11/21/364390.html</guid><wfw:comment>http://www.tkk7.com/shenh062326/comments/364390.html</wfw:comment><comments>http://www.tkk7.com/shenh062326/archive/2011/11/21/364390.html#Feedback</comments><slash:comments>0</slash:comments><wfw:commentRss>http://www.tkk7.com/shenh062326/comments/commentRss/364390.html</wfw:commentRss><trackback:ping>http://www.tkk7.com/shenh062326/services/trackbacks/364390.html</trackback:ping><description><![CDATA[<div> <h2><div>本文转自it186云计频道,原文地址Q?a >cloud.it168.com</a></div><br /></h2><p>在互联网q个领域一直有q样的说法:“如果老二无法战胜老大Q那么就把老大赖以生存的东西开源吧”。当qYahoo!与Googleq是处在 强烈竞争关系时候,招聘了Doug(Hadoop创始?Q把Google老大赖以生存的DFS与Map-Reduce开源了Q开始了Hadoop的童q?时期。差不多?008q的时候,Hadoop才算逐渐成熟?/p> <p>从初创到现在QHadoopl过了至?q的U篏Q现在的Hadoop不仅是当q的老二Yahoo的专用品了Q从Hadoop镉K的用户名单中Q?可以看到Facebook、Linkedin、AmazonQ可以看到EMC、eBay、Twitter、IBM、Microsoft,、Apple?HP…国内的公司有淘宝、百度等{?/p> <p><img title="hadoop" src="http://pic.yupoo.com/iammutex/BmrJ41KM/medium.jpg" alt="" width="500" height="483" /></p> <p>本文对Hadoop七年Q?004-2011Q的<a title="查看 发展历程 的全部文? target="_blank">发展历程</a>q?行梳理。读完本文后Q将不难看出QHadoop的发展基本上l历了这样一个过E:从一个开源的Apache基金会项目,随着来多的用L加入Q不断地 使用、A献和完善QŞ成一个强大的生态系l,?009q开始,随着云计和大数据的发展QHadoop作ؓ量数据分析的最佌x案,开始受到许?IT厂商的关注,从而出C许多Hadoop的商业版以及支持Hadoop的品,包括软g和硬件?/p> <ul><li>2004q_Google发表论文Q向全世界介l了MapReduce?/li><li>2005q初Qؓ了支持Nutch搜烦引擎目QNutch的开发者基于Google发布的MapReduce报告Q在Nutch上开发了一个可工作的MapReduce应用?/li><li>2005q年中,所有主要的Nutch法被移植到使用MapReduce和NDFS(Nutch Distributed File System )来运行?/li><li>2006q?月,Doug Cutting加入雅虎QYahoo!提供一个专门的团队和资源将Hadoop发展成一个可在网l上q行的系l?/li><li>2006q?月,Apache Hadoop目正式启动以支持MapReduce和HDFS的独立发展?/li><li>2007q_癑ֺ开始用Hadoop做离U处理,目前差不?0%的Hadoop集群用作日志处理?/li><li>2007q_中国Ud开始在“大云”研究中用Hadoop技术,规模过1000台?/li><li>2008q_淘宝开始投入研I基于Hadoop的系l?#8212;—云梯Qƈ其用于处理<a title="电子商务" target="_blank">电子商务</a>相关数据。云?的d量大概ؓ9.3PBQ包含了1100台机器,每天处理U?8000道作业,扫描500TB数据?/li><li>2008q?月,Hadoop成ؓApache目?/li><li>2008q?月,Yahoo!宣布其搜索引擎品部|在一个拥?万个内核的Hadoop集群上?/li><li>2008q?月,Hadoop打破1TB数据排序基准试记录。Yahoo!的一个Hadoop集群?09U完?TB数据的排?Q比上一q的U录保持者保持的297U快了将q?0U?/li><li>2009 q?3 月,Cloudera推出CDH(Cloudera’s Distribution including Apache Hadoop)q_Q完全由开放源码Y件组成,目前已经q入W?版?/li><li>2009q?月,Yahoo的团队用Hadoop? TB的数据进行排序只׃62U时间?/li><li>2009q??QHadoop Core目更名为Hadoop Common;</li><li>2009q??QMapReduce ?Hadoop Distributed File System (HDFS) 成ؓHadoop目的独立子目?/li><li>2009q??QAvro ?Chukwa 成ؓHadoop新的子项目?/li><li>2010q??QAvroqHadoop目Q成为Apache目?/li><li>2010q??QHBaseqHadoop目Q成为Apache目?/li><li>2010q?月,IBM提供了基于Hadoop 的大数据分析软g——InfoSphere BigInsightsQ包括基版和企业版?/li><li>2010q?月,Hive( Facebook) qHadoopQ成为Apache目?/li><li>2010q?月,PigqHadoopQ成为Apache目?/li><li>2011q?月,ZooKeeper qHadoopQ成为Apache目?/li><li>2011q?月,Apache Hadoop获得Media Guardian Innovation Awards ?/li><li>2011q?月, Platform Computing 宣布在它的Symphony软g中支持Hadoop MapReduce API?/li><li>2011q?月,Mapr Technologies公司推出分布式文件系l和MapReduce引擎——MapR Distribution for Apache Hadoop?/li><li>2011q?月,HCatalog 1.0发布。该目由Hortonworks ?010q?月䆾提出QHCatalog主要用于解决数据存储、元数据的问题,主要解决HDFS的瓶颈,它提供了一个地Ҏ存储数据的状态信息,q?数据清理和归档工具可以很Ҏ的进行处理?/li><li>2011q?月,SGI( Silicon Graphics International )ZSGI Rackable和CloudRack<a title="服务? target="_blank">服务?/a>产品U提供Hadoop优化的解x案?/li><li>2011q?月,EMC为客hZU新的基于开源Hadoop解决Ҏ的数据中心设?#8212;—GreenPlum HDQ以助其满客户日益增长的数据分析需求ƈ加快利用开源数据分析Y件。Greenplum是EMC?010q?月收购的一家开源数据仓库公司?/li><li>2011q?月,在收购了Engenio之后Q?NetApp推出与Hadoop应用l合的品E5400存储pȝ?/li><li>2011q?月,Calxeda公司(之前公司的名字是Smooth-Stone)发v?#8220;开拓者行?#8221;Q一个由10家Y件公司组成的团队ؓZCalxeda卛_推出的ARMpȝ上芯片设计的<a title="服务? target="_blank">服务?/a>提供支持。ƈ为Hadoop提供低功耗服务器技术?/li><li>2011q?月,数据集成供应商Informatica发布了其旗舰产品Q品设计初h处理当今事务和社会媒体所产生的v量数据,同时支持Hadoop?/li><li>2011q?月,Yahoo!和硅谷风险投资公?Benchmark Capital创徏了Hortonworks 公司Q旨在让Hadoop更加鲁棒(可靠)Qƈ让企业用hҎ安装、管理和使用Hadoop?/li><li>2011q?月,Cloudera公布了一Ҏ益于合作伙伴生态系l的计划——创徏一个生态系l,以便g供应商、Y件供应商以及pȝ集成商可以一h索如何用Hadoop更好的洞察数据?/li><li>2011q?月,Dell与Cloudera联合推出Hadoop解决Ҏ——Cloudera Enterprise。Cloudera EnterpriseZDell PowerEdge C2100机架服务器以及Dell PowerConnect 6248以太|?a title="交换? target="_blank">交换?/a> ?/li></ul> <p><a target="_blank"><img title="hadoop map" src="http://pic.yupoo.com/iammutex/BmrJ2XRi/medium.jpg" alt="" width="500" height="374" /></a></p> <p>在梳理的q程中,W者发C上图Q它很好地展CHadoop生态系l是如何在用中一步一步成长v来的?/p></div><img src ="http://www.tkk7.com/shenh062326/aggbug/364390.html" width = "1" height = "1" /><br><br><div align=right><a style="text-decoration:none;" href="http://www.tkk7.com/shenh062326/" target="_blank">俞灵</a> 2011-11-21 09:04 <a href="http://www.tkk7.com/shenh062326/archive/2011/11/21/364390.html#Feedback" target="_blank" style="text-decoration:none;">发表评论</a></div>]]></description></item><item><title>让你提升命o行效率的 Bash 快捷?[完整版]http://www.tkk7.com/shenh062326/archive/2011/11/15/363791.html俞灵俞灵Tue, 15 Nov 2011 02:04:00 GMThttp://www.tkk7.com/shenh062326/archive/2011/11/15/363791.htmlhttp://www.tkk7.com/shenh062326/comments/363791.htmlhttp://www.tkk7.com/shenh062326/archive/2011/11/15/363791.html#Feedback0http://www.tkk7.com/shenh062326/comments/commentRss/363791.htmlhttp://www.tkk7.com/shenh062326/services/trackbacks/363791.html

本文转自:
http://linuxtoy.org/archives/bash-shortcuts.html

生活?Bash shell 中,熟记以下快捷键,极大的提高你的命o行操作效率?

~辑命o

  • Ctrl + a Q移到命令行?/li>
  • Ctrl + e Q移到命令行?/li>
  • Ctrl + f Q按字符前移Q右向)
  • Ctrl + b Q按字符后移Q左向)
  • Alt + f Q按单词前移Q右向)
  • Alt + b Q按单词后移Q左向)
  • Ctrl + xxQ在命o行首和光标之间移?/li>
  • Ctrl + u Q从光标处删除至命o行首
  • Ctrl + k Q从光标处删除至命o行尾
  • Ctrl + w Q从光标处删除至字首
  • Alt + d Q从光标处删除至字尾
  • Ctrl + d Q删除光标处的字W?/li>
  • Ctrl + h Q删除光标前的字W?/li>
  • Ctrl + y Q粘贴至光标?/li>
  • Alt + c Q从光标处更改ؓ首字母大写的单词
  • Alt + u Q从光标处更改ؓ全部大写的单?/li>
  • Alt + l Q从光标处更改ؓ全部写的单?/li>
  • Ctrl + t Q交换光标处和之前的字符
  • Alt + t Q交换光标处和之前的单词
  • Alt + BackspaceQ与 Ctrl + w 相同cMQ分隔符有些差别 [感谢 rezilla 指正]

重新执行命o

  • Ctrl + rQ逆向搜烦命o历史
  • Ctrl + gQ从历史搜烦模式退?/li>
  • Ctrl + pQ历史中的上一条命?/li>
  • Ctrl + nQ历史中的下一条命?/li>
  • Alt + .Q用上一条命令的最后一个参?/li>

控制命o

  • Ctrl + lQ清?/li>
  • Ctrl + oQ执行当前命令,q择上一条命?/li>
  • Ctrl + sQ阻止屏q输?/li>
  • Ctrl + qQ允许屏q输?/li>
  • Ctrl + cQ终止命?/li>
  • Ctrl + zQ挂起命?/li>

Bang (!) 命o

  • !!Q执行上一条命?/li>
  • !blahQ执行最q的?blah 开头的命oQ如 !ls
  • !blah:pQ仅打印输出Q而不执行
  • !$Q上一条命令的最后一个参敎ͼ?Alt + . 相同
  • !$:pQ打印输?!$ 的内?/li>
  • !*Q上一条命令的所有参?/li>
  • !*:pQ打印输?!* 的内?/li>
  • ^blahQ删除上一条命令中?blah
  • ^blah^fooQ将上一条命令中?blah 替换?foo
  • ^blah^foo^Q将上一条命令中所有的 blah 都替换ؓ foo

友情提示Q?/p>

  1. 以上介绍的大多数 Bash 快捷键仅当在 emacs ~辑模式时有效,若你?Bash 配置?vi ~辑模式Q那遵?vi 的按键绑定。Bash 默认?emacs ~辑模式。如果你?Bash 不在 emacs ~辑模式Q可通过 set -o emacs 讄?/li>
  2. ^S、^Q、^C、^Z 是由l端讑֤处理的,可用 stty 命o讄?/li>


俞灵 2011-11-15 10:04 发表评论
]]>
国外java l典地址http://www.tkk7.com/shenh062326/archive/2011/11/15/363789.html俞灵俞灵Tue, 15 Nov 2011 01:45:00 GMThttp://www.tkk7.com/shenh062326/archive/2011/11/15/363789.htmlhttp://www.tkk7.com/shenh062326/comments/363789.htmlhttp://www.tkk7.com/shenh062326/archive/2011/11/15/363789.html#Feedback0http://www.tkk7.com/shenh062326/comments/commentRss/363789.htmlhttp://www.tkk7.com/shenh062326/services/trackbacks/363789.html
希望大家喜欢Q自q个备份,没事逛逛!Q?br />
http://www.javaalmanac.com - Java开发者年鉴一书的在线版本. 要想快速查到某UJava技巧的用法及示例代? q是一个不错的d.
http://www.onjava.com - O'Reilly的Java|站. 每周都有新文?
http://java.sun.com - 官方的Java开发者网?- 每周都有新文章发?
http://www.developer.com/java - 由Gamelan.com l护的Java技术文章网?
http://www.java.net - Sun公司l护的一个JavaC֌|站.
http://www.builder.com - Cnet的Builder.com|站 - 所有的技术文? 以JavaZ.
http://www.ibm.com/developerworks/java - IBM的Developerworks技术网? q是其中的Java技术主?
http://www.javaworld.com - 最早的一个Java站点. 每周更新Java技术文?
http://www.devx.com/java - DevXl护的一个Java技术文章网?
http://www.fawcette.com/javapro - JavaPro在线杂志|站.
http://www.sys-con.com/java - Java Developers Journal的在U杂志网?
http://www.javadesktop.org - 位于Java.net的一个Java桌面技术社区网?
http://www.theserverside.com - q是一个讨论所有Java服务器端技术的|站.
http://www.jars.com - 提供Java评论服务. 包括各种framework和应用程?
http://www.jguru.com - 一个非常棒的采用Q&A形式的Java技术资源社?
http://www.javaranch.com - 一个论坛,得到Java问题{案的地方,初学者的好去处?
http://www.ibiblio.org/javafaq/javafaq.html - comp.lang.java的FAQ站点 - 攉了来自comp.lang.java新闻l的问题和答案的分类目录.
http://java.sun.com/docs/books/tutorial/ - 来自SUN公司的官方Java指南 - 对于了解几乎所有的java技术特性非常有帮助.
http://www.javablogs.com - 互联|上最z跃的一个Java Blog|站.
http://java.about.com/ - 来自About.com的Java新闻和技术文章网?

俞灵 2011-11-15 09:45 发表评论
]]>
վ֩ģ壺 ѲСƵ| ޾ƷƷۺ| AVһ| ˬˬƵѿԿ| avѵӰ| 컶ëƬ| ձһѸ| ڵƵѲ| һƷ| AVƬ߲Ψ| ԻƵ| ޾Ʒ| һA| ɫַѹۿ| վþþ| ɫwwwۺ| ޻Ƭֻѹۿ| ĻӰԺww4164h| gayվ| ޾߹ۿ| ޹ۺϾþ| ѵƵݮ| ˳Ƶ| Թ| ޾ƷĻþò| Ƶվ| 99߹ۿ| ˿wwwƵ| av| ޹ŮƷþ| ޳av| ޾ƷƬ߹ۿ | 99þùۺϾƷ| ޸ľƷ26U| һëƬڲ| ҳվ߿| ޾þһ | ѹۿ| Ʒպһ91| ˳ɵӰվ| ޳˼|