??xml version="1.0" encoding="utf-8" standalone="yes"?>亚洲AV永久无码精品一百度影院,亚洲Av无码专区国产乱码DVD ,国产成人精品日本亚洲http://www.tkk7.com/matuobasyouca/archive/2012/08/08/385087.html一酌散千忧一酌散千忧Wed, 08 Aug 2012 12:21:00 GMThttp://www.tkk7.com/matuobasyouca/archive/2012/08/08/385087.htmlhttp://www.tkk7.com/matuobasyouca/comments/385087.htmlhttp://www.tkk7.com/matuobasyouca/archive/2012/08/08/385087.html#Feedback0http://www.tkk7.com/matuobasyouca/comments/commentRss/385087.htmlhttp://www.tkk7.com/matuobasyouca/services/trackbacks/385087.html

Hadoop实施已经有快一个月?jin),对Hadoop的概느解、用,Linux与shell脚本Q甚至mysql都有?jin)更多的理解?br />

目背景Q用于互联网信息攉后的关键词匹配与内容提取?/span>

主要pȝ架构分ؓ(f)互联|爬虫、分析、业务应用三块:(x)

单架构描q?/span>

׃我在当中的角色主要负责分析架构的搭徏Q所以其他两块都d单,下面也不?x)过多的描述?br />

Hadoop理解Q提到Hadoop都想到的是云、分布式计算Q在一D|间的实施之后有了(jin)一些具体的理解?/span>

Hadoop的优势:(x)

针对性能指标Q当业务数据量总量或增速上升到一定别,依靠关系型数据库一定无法支持。对于非关系型数据库Q包括Nosql和Solr一cd储方式,E显复杂Q对于机器集性能要求偏高Q相对于文gpȝQ。从数据使用模式上来Ԍ目前量数据的常常是不包含复杂逻辑的简单统计整理(比如上述pȝ中的关键词匹配)(j)。这时候文件系l的优势反而比较明显(l构单,逻辑单)(j)?/span>

如上q系l的应用场景是怎么L(fng)呢,在一个强大的爬虫pȝ之下Q每个小时的数据增量在G?0G的别,需要搜索所有的文gQ获取关键字的匹配,q且对匹配内容进行摘要。很cM我们windows里面的搜索功能,需要解决的是如何在这样增q的文gpȝ之下Q如何满业务系l的需求?br />

?/span>分析pȝ有什么要?/span>呢?

能够建立集群Q分布式的保存数据文件内容(l一控制Q可配置Q?/span>

有一定的保护机制Q保证数据或节点丢失不会(x)影响pȝ使用?/span>

如果有一个Q务脚本执行框架机制就好了(jin)Q用于ƈ行计)(j)?/span>

能够q行节点间的数据均衡?/span>

能够单的查看所有的状态与日志Qweb客户端)(j)

可能主要是这些了(jin)。若自己实现Q确实是个复杂而庞大的工程Q现在我们有?jin)Hadoop?br />

pȝ物理架构Q?/span>

我们使用?jin)一台服务器Q利用虚拟化Q安装了(jin)7?4x位的CentOS。一个NamenodeQ?个DatanodeQ复制数讄?。每个系l分配到一个cpuQ?G内存QDatanode挂蝲?00G的存储空间?/span>

理想的Hadoop的搭建环境,参照《Best Practices for Selecting Apache Hadoop Hardware》(http://hortonworks.com/blog/best-practices-for-selecting-apache-hadoop-hardware/Q一文,以及(qing)一些其他的文章?/span>

CPUQ最好是双CPUQ?核左叟뀂不用太高了(jin)?/span>

内存Q推?8GQ但?G应该可以运行Hadoop?jin)?/span>

盘Q?200转的SATA盘卛_QHadoop很占I间Q所以尽量加?/span>

|络Q内部的数据交换要求非常高,内网最好是千兆|卡Q带宽ؓ(f)1GB?br />

理想与现实,有钱与没钱,呵呵?br />

pȝ软g架构Q?/span>

HadoopQ版本用的?.0.3Q再下来是2?jin),Z(jin)量化应用,所以不考虑2的新Ҏ(gu)。对Hadoop没有做太多的讄Q基本基于默认?0为NamenodeQ?1-76为Datanode?/span>

JDKQ?.6.0_33 Q?4xQ?br />

pȝ实施q程Q?/span>

HDFS部分Q?/span>

爬虫抓取数据Q整理后存放?0文g服务器,70以外部挂载的形式d。网|件比较小Q假如直接写入Hadoop对Namenode负蝲q大Q所以入库前合ƈQ将每小时网|合成Z个文件写入HDFSQ由于区分类别,所以每时基本写入10个文件左叻I总量?-8GQ耗时?0-50分钟。(q个q程中,׃爬虫的IOq于频繁Q导致文件读取困难,所以做?jin)定时Q务,每小时启动一ơ,需要处理的文g先拷贝到临时区域Q合q入库之后再删除。此处应该是受到单核cpu的限Ӟ所有操作均是串行,包括拯QcpQ和合ƈ入库QjavaQ,所以Namenode严重配置E高。)(j)

此处没有太多问题?/span>

MapReduce部分Q?/span>

写入完成后,q行分析工作QMapReduce。此处的工作q程为:(x)数据库定时生成关键词列表文g。Job执行时会(x)d列表文gQ匹配指定范围内的HDFS文gQ过M时Q,匚w出对应的表达式与HTMLQMapq程l束。在Reduce阶段Q会(x)Map的所有数据入数据库(MysqlQ?/span>

此处出现q一些问题,记录下来?/span>

1. Reduce阶段需要加载Mysql的第三方驱动包。我在三个环境测试过Q公司、家里、发布环境)(j)Q?nbsp;-libjars 一定可以,有的地方不需要也可以。不明确Q怀疑与HADOOP_HOME环境变量有关?/span>

2. MRq程中用log4j打印日志Q在Hadoop临时目录Q如果你没有配置dfs.name.dirQdfs.data.dir,mapred.local.dir.mapred.system.dir{目录,q些都会(x)在hadoop.tmp.dir当中Q我偷懒都没配|)(j)mapred文g夹中查看一下?/span>

整个q程实际上还是比较简单的Q基本编码量在Job的部分,但是一个Java文g够?jin)。在目前初阶段应该q是比较好用的。现在还没有试Job的执行效率。完成后?x)l记录下来。有什么问题可以提出。我惛_什么也?x)在本文l箋更新?/span>



一酌散千忧 2012-08-08 20:21 发表评论
]]>
Hadoop完整分布式配|方?Fully distributed mode)http://www.tkk7.com/matuobasyouca/archive/2012/07/04/382150.html一酌散千忧一酌散千忧Tue, 03 Jul 2012 23:38:00 GMThttp://www.tkk7.com/matuobasyouca/archive/2012/07/04/382150.htmlhttp://www.tkk7.com/matuobasyouca/comments/382150.htmlhttp://www.tkk7.com/matuobasyouca/archive/2012/07/04/382150.html#Feedback0http://www.tkk7.com/matuobasyouca/comments/commentRss/382150.htmlhttp://www.tkk7.com/matuobasyouca/services/trackbacks/382150.html

g资源Q?/span>

三台CentOS5.6虚拟机(VmwareQ?/span>

本机 windows7 64x

 

基本资源配置Q?/span>

三台虚拟机均是克隆自同一个镜?/span>

已经安装?/span>Java环境Q?/span>jdk1.6.0_25Q?/span>

Hadoop路径?/span>/usr/hadoop/hadoop-0.20.205.0

 

操作步骤Q?/span>

1、机器名U规?/span>

ip分别?/span>128?/span>129?/span>130Q将128讄?/span>masterQ其他设|ؓ(f)slave

修改

/etc/sysconfig/network

/etc/hosts

两处配置Q名U分别ؓ(f)hadoop-master\hadoop-slave01\hadoop-slave02

注意Q此处名U最好不用用下划线Q有可能引发namenode的启动异常?/span>

 

2、修?/span>Hadoop配置 

?/span>master节点?/span>conf中修?/span>master?/span>slave文gQ分别ؓ(f)机器?/span>ip地址

 

修改master节点?/span>conf中:(x)

core-site.xml

<property>

<name>fs.default.name</name>

<value>hdfs://ip-master:9000</value>

</property>

 

mapred-site.xml

<property>

<name>mapred.job.tracker</name>                                   

<value>master:9001</value>                                

</property>

 

hdfs-site.xm

<property>

<name>dfs.replication</name>

<value>2</value>

</property>

注意此处的端口号均ؓ(f)默认?/span>

 

 

3、徏?/span>m-s之间?/span>sshq接

首先master?/span>slave机器都需要进?/span>ssh信Q文g生成Q执行如下命令:(x)

$ ssh-keygen -t rsa

中间需要输入的地方直接回RQ接受缺省值即?/span>

 

׃使用root用户dQ所以密钥文件生成在 /root/.ssh/文g夹下Q存有一对密?/span>id_dsa?/span>id_dsa.pub?/span>

此处id_dsaQ私钥)(j)必须为其他用户不可读Q所以文件属性应当是600

 

?/span>master机器执行Q?/span>

?/span>id_dsa.pubQ公钥)(j)复制?/span> authorized_keys

$ cp id_dsa.pub authorized_keys

如果是多台机器需?/span>,无密码登?/span>,则各自机器生公钥追加到authorized_keys卛_.

 

使用scp协议覆盖slave端的密钥文g夹,使得slave机器信Q来自master的连?/span>:

$ scp /root/.ssh/* ip-slave:/root/.ssh

 

 

4、启动服?/span> 

?/span>$HADOOP_HOME/bin下的所有文件给与执行权限:(x)

$ chmod 777 bin

 

master作ؓ(f)namenod需要执行如下脚本:(x)

$HADOOP_HOME/bin/hadoop namenode –format

 

完成后执?/span> $HADOOP_HOME/bin/start-all.sh

 

5、问题检?/span>

?/span>Hadoop根目录下?/span>logs文g中,(g)查各个服务日志的启动情况

 

 

6、其他情况说明:(x)

Q: $HADOOP_HOME is deprecated

A: 基本不会(x)产生M影响。由于脚本启动时讄?jin)该环境变量Q就?x)提C用户原有环境变量失效。可以取消环境变量设|,或者直接去bin/hadoop中找到这句话Q去掉即?/span>

 

Q: 无效的选项 -jvm / Unrecognized option: -jvm

A: 在?/span>root用户d?/span> bin/hadoop 脚本׃(x)q行判断Q加?/span>-jvm参数。此处是Z(jin)q入jsvcQ?/span>http://commons.apache.org/daemon/jsvc.htmlQ,此处q不定是否bugQ也不再q行详细的追溯,解决Ҏ(gu)是q入 bin/hadoop 脚本?/span> 扑ֈ jvm 参数q去掉?/span>

 

 

 

 

 

 

 



一酌散千忧 2012-07-04 07:38 发表评论
]]>
我们的用户体验真的这么差Q?/title><link>http://www.tkk7.com/matuobasyouca/archive/2012/05/22/378752.html</link><dc:creator>一酌散千忧</dc:creator><author>一酌散千忧</author><pubDate>Mon, 21 May 2012 21:02:00 GMT</pubDate><guid>http://www.tkk7.com/matuobasyouca/archive/2012/05/22/378752.html</guid><wfw:comment>http://www.tkk7.com/matuobasyouca/comments/378752.html</wfw:comment><comments>http://www.tkk7.com/matuobasyouca/archive/2012/05/22/378752.html#Feedback</comments><slash:comments>0</slash:comments><wfw:commentRss>http://www.tkk7.com/matuobasyouca/comments/commentRss/378752.html</wfw:commentRss><trackback:ping>http://www.tkk7.com/matuobasyouca/services/trackbacks/378752.html</trackback:ping><description><![CDATA[<p><span style="font-family:宋体;Times New Roman";Times New Roman"">公司里有同事时常抱怨,目的用户体验太差,常常挨领导的骂。大安认ؓ(f)是在用户体验的设计方面,公司人员的能力和l验都不_L(fng)。发牢骚(d)的时候也?x)说Q如果公司能够请得v“淘宝”?/span>UI<span style="font-family:宋体;Times New Roman";Times New Roman"">设计师,׃的系l肯定会(x)更上一层楼。我之前也一直认为如此,x们的设计是媄(jing)响项目体验的重要原因。最q被领导调动d助一个项目,产生?jin)一些不一L(fng)体会(x)?/span></p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">目背景Q一个新的品,型目Q纯开发h?/span>3-4<span style="font-family:宋体;Times New Roman";Times New Roman"">人,</span>2<span style="font-family:宋体;Times New Roman";Times New Roman"">名熟l开发h员,</span>1<span style="font-family:宋体;Times New Roman";Times New Roman"">名新手,偶尔?x)有协助人员。没有技术经理,目l理w负多个目Q对目q度兛_(j)不Q部门经理会(x)协助q行工作和进度管理。可以看到管理还是比较؜乱?/span></p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">׃目q度太慢Q领D求从我这边调一个熟lh员协助开发。我也基本了(jin)解他们的目状况Qؓ(f)?jin)不让我的hq去抓瞎Q我和他一起去?jin)解目情况?/span></p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">目状况比较p糕Q介入项目时已经开发了(jin)一D|_(d)保留的文档只有两份,一副数据库说明Q一份非常粗略的需求说明,而且q与开发进度不同步Q就是没有维护?/span></p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">我了(jin)解了(jin)一下项目目前的隑ֺQ开发h员和我反映一个是人员熟练E度的问题,二是需求变更的问题。我整体?jin)解了(jin)一下项目目前的需求和设计Q以?qing)进度。就挑了(jin)一个模块询问他们的变更情况Q这个模块是一个关键词匚w功能。结果是领导看了(jin)他们的页面之后,嫌信息量太少Q就要求提供一些更l化的数据展C。开发h员问我有什么意见,我就单讲?jin)一下页面大概怎么构徏。其中有一个点Q是用于变更数据范围Q即查询的表变更Q我一开始觉得用下拉框可以,产生?jin)一些意见。有人徏议分Z同子模块Q或?/span>tab<span style="font-family:宋体;Times New Roman";Times New Roman"">,或者分为多块ƈ列展C。我想了(jin)惻Iq他们讲了(jin)我认为几U方案的优点~点?qing)适用范围?/span></p> <p> </p> <p style="margin-left:18.0pt;text-indent:-18.0pt;">1.<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">       </span><span style="font-family:宋体;Times New Roman";Times New Roman"">多块q行展示Q?/span></p> <p style="margin-left:21.0pt"><span style="font-family:宋体;Times New Roman";Times New Roman"">多个不同范围的数据在同一面中分Z同区域以相同形式展示。原因是׃多块数据之间有一定的兌因果关系Q或值得Ҏ(gu)。适用范围Q如购物|站中的多个物品比较?/span></p> <p style="margin-left:18.0pt;text-indent:-18.0pt;">2.<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">       </span>Tab<span style="font-family:宋体;Times New Roman";Times New Roman"">:(x)</span></p> <p style="margin-left:21.0pt"><span style="font-family:宋体;Times New Roman";Times New Roman"">同一个页面的多个</span>tab<span style="font-family:宋体;Times New Roman";Times New Roman"">,表示多个</span>tab<span style="font-family:宋体;Times New Roman";Times New Roman"">中的数据可能在一定的领域概念之下有一定的兌Q但兌度不强。因?/span>tab<span style="font-family:宋体;Times New Roman";Times New Roman"">|重要的是一个同步工作的状态,?/span>A tab<span style="font-family:宋体;Times New Roman";Times New Roman"">|看一定信息,?x)打开</span>B tab<span style="font-family:宋体;Times New Roman";Times New Roman"">|看其他信息,中途还?x)切?/span>A tab<span style="font-family:宋体;Times New Roman";Times New Roman"">c(din)适用范围Q如邮箱中,收g和草稿?/span></p> <p style="margin-left:18.0pt;text-indent:-18.0pt;">3.<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">       </span><span style="font-family:宋体;Times New Roman";Times New Roman"">下拉?/span></p> <p style="margin-left:21.0pt"><span style="font-family:宋体;Times New Roman";Times New Roman"">下拉框作为查询条件的一部分Q常用于有着常规或固定的可选择内容中(如性别Q月份)(j)Q更多是以过滤的形态出玎ͼ即下拉框更适合针对某表的某个字D过滤,如果针对的是数据范围或是对用户需要直观了(jin)解的重要业务条g则不太合适。适用范围Q如在考试成W中用下拉框qo(h)“男女”?#8220;?qing)格不?qing)?#8221;?/span></p> <p style="margin-left:18.0pt;text-indent:-18.0pt;">4.<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">       </span><span style="font-family:宋体;Times New Roman";Times New Roman"">单选框</span></p> <p style="margin-left:21.0pt"><span style="font-family:宋体;Times New Roman";Times New Roman"">单选框与下拉框的作用范围相|但是不同之处在于被选项全部展示Q目的在于能够让用户清楚的了(jin)解当前数据显C的实际范围或条Ӟ以及(qing)备选的其他范围或条件。更适用于选项与实际业务及(qing)当前展示数据关系重要Q不同选项可能?x)引发用L(fng)不同行ؓ(f)。适用范围Q如银行pȝ昄?jin)当前用户下l定多个帐号Ӟ使用单选框?/span></p> <p style="margin-left:21.0pt"> </p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">l过上述讨论Q我们仔l分析了(jin)q个模块中用L(fng)实际需求,以及(qing)可能后箋操作Q最l选择的单选框的方案?/span></p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">目前q没有后l,但是我想我们Z用户真是需求的挖掘和后l操作的认真分析Q会(x)让我们在与领D行需求讨论的时候有更加充分合理的依据?/span></p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">回来之后我又看了(jin)看淘宝的搜烦(ch)面Q比如就搜烦(ch)“鞋子”来讲Q将品牌q栏讄为单选和下拉是完全不同的效果,而确定方案的理由则是对于用户的需求和实际行ؓ(f)的深入研I。这个应该是需求分析和调研的结果。将搜烦(ch)条g?/span>tag<span style="font-family:宋体;Times New Roman";Times New Roman"">的Ş式标注于面上,q且可以直接点击</span>X<span style="font-family:宋体;Times New Roman";Times New Roman"">按钮q行删除Q我觉得更加可以們֐为用户体验。满_ƈ充分考虑?jin)用户实际需求的是好的需求分析,能够化ƈ引导用户行ؓ(f)的是好的用户体验?/span></p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">当我们面临的pȝ感觉非常隄的时候,往往q时候ƈ非是用户体验差,我们应该(g)讨的是我们对用户需求有没有好好挖掘Q做出来的是不是用户惌、用戯用的pȝ?/span></p><img src ="http://www.tkk7.com/matuobasyouca/aggbug/378752.html" width = "1" height = "1" /><br><br><div align=right><a style="text-decoration:none;" href="http://www.tkk7.com/matuobasyouca/" target="_blank">一酌散千忧</a> 2012-05-22 05:02 <a href="http://www.tkk7.com/matuobasyouca/archive/2012/05/22/378752.html#Feedback" target="_blank" style="text-decoration:none;">发表评论</a></div>]]></description></item><item><title>Zookeeper的学?fn)ȝhttp://www.tkk7.com/matuobasyouca/archive/2012/05/15/378164.html一酌散千忧一酌散千忧Tue, 15 May 2012 03:02:00 GMThttp://www.tkk7.com/matuobasyouca/archive/2012/05/15/378164.htmlhttp://www.tkk7.com/matuobasyouca/comments/378164.htmlhttp://www.tkk7.com/matuobasyouca/archive/2012/05/15/378164.html#Feedback0http://www.tkk7.com/matuobasyouca/comments/commentRss/378164.htmlhttp://www.tkk7.com/matuobasyouca/services/trackbacks/378164.html

Zookeeper的核?j)概念?x)

ZNode

Znode是核心(j)l构QZookeeper服务中是由大量的Znode构成。Znode一般是由客L(fng)建立和修改,作ؓ(f)信息或标志的载体Q甚xw就是标志?/span>

Znode可以讄为持久(PERSISTENTQ或临时QEPHEMERALQ,区别在于临时的节点若断开q接后就自动删除。徏立节Ҏ(gu)可选择是否使用序列号命名(SEQUENTIALQ,若启用则?x)自动在节点名后加入唯一序列~号?/span>

Session

作ؓ(f)客户端和Zookeeper服务之间交互的凭证?/span>

Watch

当客L(fng)对节点信息进行查询操作之后,可以选择是否讄一个Watch。其作用是当本ơ查询的数据在服务器端发生变化之后,?x)对讄Watch的客L(fng)发送通知。一ơ发送之后,将删除该WatchQ以后的变更或不再设|Watch则不?x)通知?/span>

ACLs

节点的权限限制用ACLQ如增删Ҏ(gu)操作?/span>

Zookeeper的服务器安装Q?/span>

1?/span>下蝲对应版本L(fng)tar.gz文g

2?/span>使用 tar xzvf zookeeper-3.4.2.tar.gz -C ./ 解压

3?/span>讄Q将conf/zoo.example.cfg复制到conf/zoo.cfg或者手动徏立一个新的?/span>

4?/span>启动Zookeeper服务Qbin/zkServer.sh start

5?/span>启动客户端连接:(x)bin/zkCli.sh -server 127.0.0.1:2181Q此处在本机Q且使用?jin)默认端口,且在Java环境中)(j)

6?/span>使用命o(h)Qls、get、set{?/span>

7?/span>关闭Zookeeper服务Qbin/zkServer.sh stop

Zookeeper代码~写Q?/span>

代码~写部分比较单,因ؓ(f)暴露的接口很,主要复杂在于目如何使用节点以及(qing)节点信息?/span>

启动Zookeeper服务之后Q客L(fng)代码q行节点的增删,W(xu)atch的设|,内容的改查等?/span>

此处查看官方的《Programming with ZooKeeper - A basic tutorial》部分,当中举了(jin)两个例子来模拟分布式pȝ的应用?/span>

代码基本没有问题Q唯一需要注意的是Q若之间按照原版q行调试Ӟ有可能在调用

 Stat s = zk.exists(root, false);

q句代码时会(x)出现一个异常,当中包括“KeeperErrorCode = ConnectionLoss for”?/span>

q个问题引v的原因可以看一下代?/span>

                System.out.println("Starting ZK:");
                zk 
= new ZooKeeper(address, 3000this);
                mutex 
= new Integer(-1);
                System.out.println(
"Finished starting ZK: " + zk);

最后一行有打印出Zookeeper目前的信息,若未修改的原代码Q此处的State应当是CONECTING。连接中的时候去验证是否存在节点?x)报错。解决的Ҏ(gu)也很单,是{到Zookeeper客户端以?qing)完全连接上服务器,State为CONECTED之后再进行其他操作。给Z码示例:(x)

// 使用?jin)倒数计数Q只需要计C?/span>
private CountDownLatch connectedSignal = new CountDownLatch(1); 
SyncPrimitive(String address) {
    
if(zk == null){
        
try {
            System.out.println(
"Starting ZK:");
            zk 
= new ZooKeeper(address, 3000this);
            mutex 
= new Integer(-1);
            connectedSignal.await(); 
// {待q接完成
            System.out.println("Finished starting ZK: " + zk);
        } 
catch (IOException e) {
            System.out.println(e.toString());
            zk 
= null;
        } 
catch (InterruptedException e) {
            
// TODO Auto-generated catch block
            e.printStackTrace();
        }
    }
    
//else mutex = new Integer(-1);
}
synchronized public void process(WatchedEvent event) {
    
// 此处讄在Watch中会(x)在状态变化后触发事g
    if (event.getState() == KeeperState.SyncConnected) {
        connectedSignal.countDown();
// 倒数-1
    }
    
        
synchronized (mutex) {
            
//System.out.println("Process: " + event.getType());
            mutex.notify();
        }
}

q样可以正运行代码了(jin)?/span>

Zookeeper的应用场景及(qing)方式Q?/span>

此处是ؓ(f)引用Q原地址为(http://rdc.taobao.com/team/jm/archives/1232 Q?/span>

ZooKeeper是一个高可用的分布式数据理与系l协调框架。基于对Paxos法的实玎ͼ使该框架保证?jin)分布式环境中数据的Z致性,也正是基于这L(fng)Ҏ(gu),使得zookeeper能够应用于很多场景。网上对zk的用场景也有不介l,本文结合作者n边的目例子Q系l的对zk的用场景进行归cMl?nbsp;值得注意的是Qzkq不是生来就些场景设计,都是后来众多开发者根据框架的Ҏ(gu),摸烦(ch)出来的典型用方法。因此,也非常欢q你分n你在ZK使用上的奇技淫y?/span>

场景cd

典型场景描述QZKҎ(gu),使用Ҏ(gu)Q?/span>

应用中的具体使用

数据发布与订?/span>

发布与订阅即所谓的配置理Q顾名思义是数据发布到zk节点上,供订阅者动态获取数据,实现配置信息的集中式理和动态更新。例如全局的配|信息,地址列表{就非常适合使用?/span>

1. 索引信息和集中机器节点状态存攑֜zk的一些指定节点,供各个客L(fng)订阅使用?. pȝ日志Q经q处理后的)(j)存储Q这些日志通常2-3天后被清除?nbsp;

3. 应用中用到的一些配|信息集中管理,在应用启动的时候主动来获取一ơ,q且在节点上注册一个WatcherQ以后每ơ配|有更新Q实旉知到应用,获取最新配|信息?/span>

4. 业务逻辑中需要用到的一些全局变量Q比如一些消息中间g的消息队列通常有个offsetQ这个offset存放在zk上,q样集群中每个发送者都能知道当前的发送进度?/span>

5. pȝ中有些信息需要动态获取,q且q会(x)存在人工手动M改这个信息。以前通常是暴露出接口Q例如JMX接口Q有?jin)zk后,只要这些信息存攑ֈzk节点上即可?/span>

Name Service

q个主要是作为分布式命名服务Q通过调用zk的create node apiQ能够很Ҏ(gu)创徏一个全局唯一的pathQ这个path可以作Z个名U?/span>

 

分布通知/协调

ZooKeeper中特有watcher注册与异步通知机制Q能够很好的实现分布式环境下不同pȝ之间的通知与协调,实现Ҏ(gu)据变更的实时处理。用方法通常是不同系l都对ZK上同一个znodeq行注册Q监听znode的变化(包括znode本n内容?qing)子节点的?j)Q其中一个系lupdate?jin)znodeQ那么另一个系l能够收到通知Qƈ作出相应处理?/span>

1. 另一U心(j)x机Ӟ(x)(g)系l和被检系l之间ƈ不直接关联v来,而是通过zk上某个节点关联,大大减少pȝ耦合?. 另一U系l调度模式:(x)某系l有控制台和推送系l两部分l成Q控制台的职责是控制推送系l进行相应的推送工作。管理h员在控制C的一些操作,实际上是修改?jin)ZK上某些节点的状态,而zk把q些变化通知l他们注册W(xu)atcher的客L(fng)Q即推送系l,于是Q作出相应的推送Q务?nbsp;

3. 另一U工作汇报模式:(x)一些类gd分发pȝQ子d启动后,到zk来注册一个(f)时节点,q且定时自qq度q行汇报Q将q度写回q个临时节点Q,q样d理者就能够实时知道dq度?/span>

MQ用zookeeper来进行分布式通知和协调能够大大降低系l之间的耦合?/span>

分布式锁

分布式锁Q这个主要得益于ZooKeeper为我们保证了(jin)数据的强一致性,即用户只要完全相信每时每刻,zk集群中Q意节点(一个zk serverQ上的相同znode的数据是一定是相同的。锁服务可以分ؓ(f)两类Q?/span>一个是保持独占Q另一个是控制时序?/span> 

所谓保持独占,是所有试图来获取q个锁的客户端,最l只有一个可以成功获得这把锁。通常的做法是把zk上的一个znode看作是一把锁Q通过create znode的方式来实现。所有客L(fng)都去创徏 /distribute_lock 节点Q最l成功创建的那个客户端也x有了(jin)q把锁?/span>

控制时序Q就是所有视图来获取q个锁的客户端,最l都是会(x)被安排执行,只是有个全局时序?jin)。做法和上面基本cMQ只是这?nbsp;/distribute_lock 已经预先存在Q客L(fng)在它下面创徏临时有序节点Q这个可以通过节点的属性控Ӟ(x)CreateMode.EPHEMERAL_SEQUENTIAL来指定)(j)。Zk的父节点Q?distribute_lockQ维持一份sequence,保证子节点创建的时序性,从而也形成?jin)每个客L(fng)的全局时序?/span>

 

集群理

1. 集群机器监控Q这通常用于那种寚w中机器状态,机器在线率有较高要求的场景,能够快速对集群中机器变化作出响应。这L(fng)场景中,往往有一个监控系l,实时(g)集机器是否存?gu)zR过ȝ做法通常是:(x)监控pȝ通过某种手段Q比如pingQ定时检每个机器,或者每个机器自己定时向监控pȝ汇报“我还?gu)zȝ”?nbsp;q种做法可行Q但是存在两个比较明昄问题Q?. 集群中机器有变动的时候,牵连修改的东西比较多?. 有一定的延时?nbsp;

利用ZooKeeper有两个特性,可以实时另一U集机器存?gu)zL监控系l:(x)a. 客户端在节点 x 上注册一个WatcherQ那么如?nbsp;x 的子节点变化?jin),会(x)通知该客L(fng)。b. 创徏EPHEMERALcd的节点,一旦客L(fng)和服务器的会(x)话结束或q期Q那么该节点׃(x)消失?/span>

例如Q监控系l在 /clusterServers 节点上注册一个WatcherQ以后每动态加机器Q那么就往 /clusterServers 下创Z?nbsp;EPHEMERALcd的节点:(x)/clusterServers/{hostname}. q样Q监控系l就能够实时知道机器的增减情况,至于后箋处理是监控pȝ的业务了(jin)?/span>
2. Master选D则是zookeeper中最为经典的使用场景?jin)?/span>

在分布式环境中,相同的业务应用分布在不同的机器上Q有些业务逻辑Q例如一些耗时的计,|络I/O处理Q,往往只需要让整个集群中的某一台机器进行执行,其余机器可以׃nq个l果Q这样可以大大减重复劳动,提高性能Q于是这个master选D便是q种场景下的到的主要问题?/span>

利用ZooKeeper的强一致性,能够保证在分布式高ƈ发情况下节点创徏的全局唯一性,卻I(x)同时有多个客L(fng)h创徏 /currentMaster 节点Q最l一定只有一个客L(fng)h能够创徏成功?/span>

利用q个Ҏ(gu),p很轻易的在分布式环境中进行集选取?jin)?/span>

另外Q这U场景演化一下,是动态Master选D。这p用到 EPHEMERAL_SEQUENTIALcd节点的特性了(jin)?/span>

上文中提刎ͼ所有客L(fng)创徏hQ最l只有一个能够创建成功。在q里E微变化下,是允许所有请求都能够创徏成功Q但是得有个创徏序Q于是所有的h最l在ZK上创建结果的一U可能情冉|q样Q?nbsp;/currentMaster/{sessionId}-1 , /currentMaster/{sessionId}-2 , /currentMaster/{sessionId}-3 ….. 每次选取序列h的那个机器作ؓ(f)MasterQ如果这个机器挂?jin),׃他创建的节点会(x)马上小Ӟ那么之后最的那个机器是Master?jin)?/span>

1. 在搜索系l中Q如果集中每个机器都生成一份全量烦(ch)引,不仅耗时Q而且不能保证彼此之间索引数据一致。因此让集群中的Master来进行全量烦(ch)引的生成Q然后同步到集群中其它机器?. 另外QMaster选D的容灾措施是Q可以随时进行手动指定masterQ就是说应用在zk在无法获取master信息Ӟ可以通过比如http方式Q向一个地方获取master?/span>

分布式队?/span>

队列斚wQ我目前感觉有两U,一U是常规的先q先出队列,另一U是要等到队列成员聚齐之后的才统一按序执行。对于第二种先进先出队列Q和分布式锁服务中的控制时序场景基本原理一_(d)q里不再赘述?nbsp;

W二U队列其实是在FIFO队列的基上作?jin)一个增强。通常可以?nbsp;/queue q个znode下预先徏立一?queue/num 节点Qƈ且赋gؓ(f)nQ或者直接给/queue赋值nQ,表示队列大小Q之后每ơ有队列成员加入后,判断下是否已经到达队列大小Q决定是否可以开始执行了(jin)。这U用法的典型场景是,分布式环境中Q一个大dTask AQ需要在很多子Q务完成(或条件就l)(j)情况下才能进行。这个时候,凡是其中一个子d完成Q就l)(j)Q那么就?nbsp;/taskList 下徏立自q临时时序节点QCreateMode.EPHEMERAL_SEQUENTIALQ,?nbsp;/taskList 发现自己下面的子节点满指定个数Q就可以q行下一步按序进行处理了(jin)?/span>

 



一酌散千忧 2012-05-15 11:02 发表评论
]]>
MongoDB, Java 与对象关pL?/title><link>http://www.tkk7.com/matuobasyouca/archive/2012/05/09/377698.html</link><dc:creator>一酌散千忧</dc:creator><author>一酌散千忧</author><pubDate>Wed, 09 May 2012 05:46:00 GMT</pubDate><guid>http://www.tkk7.com/matuobasyouca/archive/2012/05/09/377698.html</guid><wfw:comment>http://www.tkk7.com/matuobasyouca/comments/377698.html</wfw:comment><comments>http://www.tkk7.com/matuobasyouca/archive/2012/05/09/377698.html#Feedback</comments><slash:comments>0</slash:comments><wfw:commentRss>http://www.tkk7.com/matuobasyouca/comments/commentRss/377698.html</wfw:commentRss><trackback:ping>http://www.tkk7.com/matuobasyouca/services/trackbacks/377698.html</trackback:ping><description><![CDATA[<div> <h2><span style="font-size:14.5pt;font-family:Arial;color:black;letter-spacing:-.75pt">MongoDB</span><span><span style="font-size:14.5pt;color:black;letter-spacing:-.75pt">介绍</span></span><span></span></h2> <p><span>当今<span>NoSQL领域中有很多有力的竞争者通过多种方式来处理v量数据问题。其中重要的解决Ҏ(gu)之一是MongoDB。MongoDB是面向文档的q构化存储Ҏ(gu)Q用JSON格式来展现、查询和修改数据?/span></span></p> <p><span><span>MongoDB文档相当完备Q扩展规模与安装一L(fng)单。它提供冗余、切片、烦(ch)引以?qing)map/reduce{概忉|持。MongoDB的开源社区非常大且非常活跃。MongoDB在很多大型品中被实际运用,如:(x)Disney, Craigslist, Foursquare, Github 和SourceForge。MongoDB是一个开源项目,?/span></span><a ><span><span style="color:#0B59B2">10gen.com</span></span></a><span>建立q维护,该公司由<span>DoubleClick的前L行h员创立。同Ӟ10gen也提供了(jin)极好的商业支持与参与?/span></span></p> <h2><span><span><span style="font-size:14.5pt; font-family:Arial;color:black;letter-spacing:-.75pt">MongoDB </span></span></span><span><span style="font-size:14.5pt;color:black;letter-spacing:-.75pt">?/span></span><span><span><span style="font-size:14.5pt;font-family:Arial;color:black;letter-spacing: -.75pt"> NoSQL: </span></span></span><span><span><span style="font-size:14.5pt;color:black; letter-spacing:-.75pt">~陷与优?/span></span></span><span></span></h2> <p><span><span>MongoDB作ؓ(f)一个可用NoSQLҎ(gu)h很多优势。我刚开始接触NoSQL数据库了(jin)解了(jin)一pdZJava的方案,q且׃(jin)大量的时间来弄懂什么是列家族,Hadoop与HBase的关p,ZooKeeper到底是什么。当我终于全部清楚之后,发现Cassandra与HBase实是对于NoSQL领域非常可靠、可信赖的解x案。但与其他的解决Ҏ(gu)相比QMongoDB让我在能够开始写代码之前Q不用理解那么多的概c(din)?/span></span></p> <p><span>与其他Y件相|<span>MongoDB也存在缺陗经q一D|间用MongoDBQ我列Dl历qƈ需要注意的一些事情,我成?#8220;Gotchas”Q?/span></span></p> <ul type="disc"><li style=" text-align:left;"><span><span style="font-family:宋体;">不要按照关系型数据库来思考。这很明显,MongoDB</span></span><span><span style="font-family:宋体;">使得构徏和执行复杂查询变得非常容易。当实际使用的时候,你会(x)主要x于效率问题(像我一P(j)?/span></span></li><li style=" text-align:left;"><span><span>MongoDB</span></span><span><span style="font-family:宋体;">的烦(ch)引是二进制的?wi)。如果你不是很熟(zhn)B-tree</span></span><span><span style="font-family:宋体;">Q可能需要了(jin)解一下。这些都涉及(qing)到构建符合提供查询条仉求的建立索引的方式?/span></span></li><li style=" text-align:left;"><span><span style="font-family:宋体;">心(j)的设计烦(ch)引结构。这涉及(qing)C面提到的B-tree</span></span><span><span style="font-family:宋体;">。刚开始我的烦(ch)引包含文档中的很多字D,以防我会(x)使用C们。不要犯同样的错误。我有一个很集合的索引Q大U?</span></span><span><span style="font-family:宋体;">千万记录Q增长到过17GB</span></span><span><span style="font-family:宋体;">的空_(d)比集合本w还大。你应该不会(x)惌索引一个包含成百上千个实体的列表字Dc(din)?/span></span></li><li style=" text-align:left;"><span><span>MongoDB</span></span><span><span style="font-family:宋体;">采用?jin)非常有意思的方式来实现NoSQL</span></span><span><span style="font-family:宋体;">Q采用BSON</span></span><span><span style="font-family:宋体;">作ؓ(f)存储QJSON</span></span><span><span style="font-family:宋体;">作ؓ(f)展示QJavaScript</span></span><span><span style="font-family:宋体;">用于理和Map/Reduce</span></span><span><span style="font-family:宋体;">。因此也引v?jin)一些小问题比如</span></span><a ><span><span><span style="font-family: 宋体;color:#0B59B2">q个</span></span></span></a><span> </span><span><span style="font-family:宋体;">Q破坏了(jin)Number</span></span><span><span style="font-family:宋体;">和Long</span></span><span><span style="font-family:宋体;">的相{操作)(j)Q在MongoDB</span></span><span><span style="font-family:宋体;">逐渐行之后Q可能会(x)不断的展C出来?/span></span></li></ul> <p><span> </span></p> <h2><span><span><span style="font-size:14.5pt; font-family:Arial;color:black;letter-spacing:-.75pt">MongoDB, </span></span></span><span><span style="font-size:14.5pt;color:black;letter-spacing:-.75pt">命o(h)行与驱动</span></span><span></span></h2> <p><span><span>MongoDB基本是用JavaScript客户端命令行E序来进行复杂Q务管理的Q如数据整合和简单信息处理,~程都是完全使用JavaScript语言来的。本文中Q我们会(x)展示命o(h)行的使用CZ。现在有大量的MongoDB客户端品提供,q且由MongoDBC֌来支持驱动。通常每种~程语言都有驱动Qƈ且所有流行的语言都有包括Q一些不那么行的也包含在内。这文章展CZ(jin)使用MongoDB的Java驱动Qƈ使用一个ORM库(MJORMQ与之进行比较?/span></span></p> <h2><span><span><span style="font-size:14.5pt;color:black; letter-spacing:-.75pt">介绍</span></span></span><span><span><span style="font-size:14.5pt; font-family:Arial;color:black;letter-spacing:-.75pt"> MJORM: MongoDB</span></span></span><span><span style="font-size:14.5pt;color:black;letter-spacing:-.75pt">?/span></span><span><span><span style="font-size:14.5pt;font-family:Arial;color:black;letter-spacing: -.75pt">ORM</span></span></span><span><span><span style="font-size:14.5pt;color:black; letter-spacing:-.75pt">Ҏ(gu)</span></span></span><span></span></h2> <p><span>在解决的众多有意思的问题中,最q?span>NoSQL数据存储在开发者中主要的问题趋势就是对象关pL。对象关pL就是将传统中保存在关系型数据库中的持久化数据映ؓ(f)在应用程序中使用的对象。这使得~程语言使用h更加畅和自然?/span></span></p> <p><span><span>MongoDB面向文档的架构得它非常适合对象关系映射Q因为文档本w就是以对象形式存储的。可惜没有太多的MongoDB的Java对象关系映射库,但是q是有一些,?/span></span><a ><span><span><span style="color:#0B59B2">morphia-(A type-safe Java library for MongoDB)</span></span></span></a><span>Q?span> </span></span><a ><span><span style="color:#0B59B2">spring-data</span></span></a><span><span>(SpringData目的MongoDB实现)</span></span><span></span></p> <p><span>q些<span>ORM库大量用了(jin)注解Q因Z些原因对我不适合Q其中最重要的就是这些被注解的对象在多个目中的兼容性问题。这让我开始了(jin)mongo-Java-orm 或?"MJORM" (发音 me-yorm)目Q一个MongoDB的Java对象关系映射目。MJORM是在MIT许可之下Qƈ且在发布在了(jin)</span></span><a ><span><span><span style="color:#0B59B2">google code project</span></span></span></a><span>。项目采?span>maven构徏Qƈ且maven构g仓库托管于google code版本控制服务器。MJORM的最新可用发布版本ؓ(f)0.15Q已l由一些项目用与生环境中?/span></span></p> <h2><span><span><span style="font-size:14.5pt;color:black; letter-spacing:-.75pt">开始?/span></span></span><span><span><span style="font-size:14.5pt; font-family:Arial;color:black;letter-spacing:-.75pt">ORM</span></span></span></h2> <h3><span><span><span style="font-size:11.5pt;line-height:173%;font-family:宋体;letter-spacing: -.75pt">加入</span></span></span><span><span><span style="font-size:11.5pt; line-height:173%;font-family:Arial;letter-spacing:-.75pt">MJORM </span></span></span><span><span><span style="font-size:11.5pt;line-height:173%;font-family:宋体;letter-spacing: -.75pt">?/span></span></span><span></span></h3> <p><span><span>Maven的用者首先应当在pom.xml中加入MJORM的maven仓库Q得MJORM构g可用?/span></span></p> <div style="border:solid #EFEFEF 1.5pt;padding: 0cm 0cm 4.0pt 0cm;background:#FAFAFA"><pre style="line-height:11.25pt; background:#FAFAFA;border:none;padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222"><repository></span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         <id>mjorm-webdav-maven-repo</id></span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         <name>mjorm maven repository</name></span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         <url>http://mongo-Java-orm.googlecode.com/svn/maven/repo/</url></span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         <layout>default</layout></span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222"></repository></span></span></span></pre></div> <p><span>然后加入依赖<span>:</span></span></p> <div style="border:solid #EFEFEF 1.5pt;padding: 0cm 0cm 4.0pt 0cm;background:#FAFAFA"><pre style="line-height:11.25pt; background:#FAFAFA;border:none;padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222"><dependency></span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         <groupId>com.googlecode</groupId></span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         <artifactId>mongo-Java-orm</artifactId></span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         <version>0.15</version></span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222"></dependency></span></span></span></pre></div> <p><span>q样可以在应用中引?span>MJORM代码。假如没有用mavenQ则你需要手动下载MJORM的pom.xml中列丄所有依赖?/span></span></p> <h3><span><span><span style="font-size:11.5pt;line-height:173%;font-family:宋体;letter-spacing: -.75pt">建立</span></span></span><span><span><span style="font-size:11.5pt; line-height:173%;font-family:Arial;letter-spacing:-.75pt"> POJOs</span></span></span></h3> <p><span>依赖已经导入Q可以开始编码了(jin)。我们从<span>POJO开?</span></span><span></span></p> <div style="border:solid #EFEFEF 1.5pt;padding: 0cm 0cm 4.0pt 0cm;background:#FAFAFA"><pre style="line-height:11.25pt; background:#FAFAFA;border:none;padding:0cm;"><span> </span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">class Author {</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         private String firstName;</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         private String lastName;</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         // ... setters and getters ...</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">}</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span> </span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">class Book {</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         private String id;</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         private String isbn;</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         private String title;</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         private String description;</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         private Author author;</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         // ... setters and getters ...</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">}</span></span></span></pre></div> <p><span>我们在这个对象模型中的描q是Q作者有<span>ID、姓和名Q书有ID、ISNB、标题、描q和作者?/span></span></p> <p><span>你可能注意到书的<span>id属性是一个字W串Q这是ؓ(f)?jin)适应MongoDB的对象IDcd。MongoDB的ID是一?2字节的二q制值显CZؓ(f)一个十六进制的字符丌ӀMongoDB要求集合中的每个文档都必L一个唯一idQ但不要求一定要是ObjectId。目前MJORM只支持ObjectIdQƈ且显CZؓ(f)字符丌Ӏ?/span></span></p> <p><span>你也可能注意C(jin)<span>Author没有id字段。这是因为Book是它的父文档Q因此不需要有id。记住,MongoDB只要求集合中的文档在根别的id?/span></span></p> <h3><span><span><span style="font-size:11.5pt;line-height:173%;font-family:宋体;letter-spacing: -.75pt">创徏</span></span></span><span><span><span style="font-size:11.5pt; line-height:173%;font-family:Arial;letter-spacing:-.75pt">XML</span></span></span><span><span><span style="font-size:11.5pt;line-height:173%;font-family:宋体;letter-spacing: -.75pt">映射文g</span></span></span><span></span></h3> <p><span>下一个步骤就是徏?span>XML映射文gQMJORM能够MongoDB文档转换为对象。我们ؓ(f)每个文档创徏一个对象作为示范,无论所有的映射攑֜一个XML文g中还是分开都是可以的?/span></span></p> <p><span><code><span style="font-size:9.0pt;font-family:"Courier New"">Author.mjorm.xml</span></code>:</span></p> <div style="border:solid #EFEFEF 1.5pt;padding: 0cm 0cm 4.0pt 0cm;background:#FAFAFA"><pre style="line-height:11.25pt; background:#FAFAFA;border:none;padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222"><?xml version="1.0"?></span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222"><descriptors></span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         <object class="Author"></span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">                 <property name="firstName" /></span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">                 <property name="lastName" /></span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         </object></span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222"></descriptors></span></span></span></pre></div> <p><span><code><span style="font-size:9.0pt;font-family:"Courier New"">Book.mjorm.xml</span></code>:</span></p> <div style="border:solid #EFEFEF 1.5pt;padding: 0cm 0cm 4.0pt 0cm;background:#FAFAFA"><pre style="line-height:11.25pt; background:#FAFAFA;border:none;padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222"><?xml version="1.0"?></span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222"><descriptors></span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         <object class="Book"></span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">                 <property name="id" id="true" auto="true" /></span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">                 <property name="isbn" /></span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">                 <property name="title" /></span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">                 <property name="description" /></span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">                 <property name="author" /></span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         </object></span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222"></descriptors></span></span></span></pre></div> <p><span> </span></p> <p><span>q些映射文g能够很好的自解释?/span><span><code><span style="font-size:9.0pt;font-family:"Courier New"">descriptors</span></code> 元素是根元素Q必d含在每个映射文g中。在它下面是</span><span><code><span style="font-size:9.0pt;font-family:"Courier New"">object</span></code>元素定义?jin)文档与之对应的cR?/span><span><code><span style="font-size:9.0pt;font-family:"Courier New"">Object</span></code></span><span><code><span style="font-size:9.0pt;Courier New";Courier New";Courier New"">包含?/span></code></span><span><code><span style="font-size:9.0pt;font-family:"Courier New"">property</span></code> 元素主要用于描述<span>POJO中的属性以?qing)这些属性如何与MongoDB中的文档惛_应?/span></span><span><code><span style="font-size:9.0pt;font-family:"Courier New"">property</span></code> 元素臛_必须包含一?/span><span><code><span style="font-size:9.0pt;font-family:"Courier New"">name</span></code> 属性,q个元素是POJO和MongoDB的文档中的属性名U?/span><span><code><span style="font-size:9.0pt;font-family:"Courier New"">column</span></code> 属性则是可选的Q用于特定一个在MongoDB文档中的可选属性名U?/span></p> <p><span><code><span style="font-size:9.0pt;font-family:"Courier New"">property</span></code> 元素当中?span>id属性应该是对象的唯一识别。一个对象只能有一?/span></span><span><code><span style="font-size:9.0pt;font-family:"Courier New"">property</span></code> 元素包含<span>id属性?/span></span><span><code><span style="font-size:9.0pt;font-family:"Courier New"">auto</span></code> 的设|会(x)使得MJORM在持久化时ؓ(f)该属性自动生成一个倹{?/span></p> <p><span>可以?span>google code的MJORM目主页中查看XML映射文g的更多细节描q?/span></span></p> <h3><span><span><span style="font-size:11.5pt;line-height:173%;font-family:宋体;letter-spacing: -.75pt">整合</span></span></span><span><span><span style="font-size:11.5pt; line-height:173%;font-family:Arial;letter-spacing:-.75pt">POJO</span></span></span><span><span><span style="font-size:11.5pt;line-height:173%;font-family:宋体;letter-spacing: -.75pt">?/span></span></span><span><span><span style="font-size:11.5pt; line-height:173%;font-family:Arial;letter-spacing:-.75pt">XML</span></span></span></h3> <p><span>我们创徏?jin)数据模型以及(qing)映文Ӟ使?span>MJORM可以从MongoDB序列号以?qing)反序列号POJO。我们可以进行一些有意思的事情?jin),首先打开MongoDB的链接:(x)</span></span><span></span></p> <div style="border:solid #EFEFEF 1.5pt;padding: 0cm 0cm 4.0pt 0cm;background:#FAFAFA"><pre style="line-height:11.25pt; background:#FAFAFA;border:none;padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">Mongo mongo = new Mongo(</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         new MongoURI("mongodb://localhost/mjormIsFun")); // 10gen driver</span></span></span></pre></div> <p><span><code><span style="font-size:9.0pt;font-family:"Courier New"">Mongo</span></code> 对象是由10gen~写的Java驱动提供的。示例中q接?jin)一个本地的MongoDB实例中的mjormIsFun数据库。接下来我们创徏MJORM </span><span><span><code><span style="font-size:9.0pt; font-family:"Courier New"">ObjectMapper </span></code>。目?/span></span><span><code><span style="font-size:9.0pt;font-family:"Courier New"">ObjectMapper </span></code>在MJORM中的唯一实现是</span><span><span><code><span style="font-size:9.0pt; font-family:"Courier New"">XmlDescriptorObjectMapper</span></code>Q用XMLl构描述信息。可能之后会(x)增加Ҏ(gu)解或其他l构定义的支持?/span></span></p> <div style="border:solid #EFEFEF 1.5pt;padding: 0cm 0cm 4.0pt 0cm;background:#FAFAFA"><pre style="line-height:11.25pt; background:#FAFAFA;border:none;padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">XmlDescriptorObjectMapper objectMapper = new XmlDescriptorObjectMapper();</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">mapper.addXmlObjectDescriptor(new File("Book.mjorm.xml"));</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">mapper.addXmlObjectDescriptor(new File("Author.mjorm.xml"));</span></span></span></pre></div> <p><span>建立好了(jin)</span><span><code><span style="font-size:9.0pt;font-family:"Courier New"">XmlDescriptorObjectMapper</span></code> q且加入?jin)映文件。接下来建立由MJORM提供?/span><span><span><code><span style="font-size:9.0pt; font-family:"Courier New"">MongoDao</span></code> 对象的实例?/span></span></p> <div style="border:solid #EFEFEF 1.5pt;padding: 0cm 0cm 4.0pt 0cm;background:#FAFAFA"><pre style="line-height:11.25pt; background:#FAFAFA;border:none;padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">DB db = mongo.getDB("mjormIsFun"); // 10gen driver</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">MongoDao dao = new MongoDaoImpl(db, objectMapper);</span></span></span></pre></div> <p><span>首先我们要获?span>10gen驱动提供的DB对象实例。然后用DB?/span></span><span><span><code><span style="font-size:9.0pt; font-family:"Courier New"">ObjectMapper</span></code> 建立</span></span><span><code><span style="font-size:9.0pt;font-family:"Courier New"">MongoDao</span></code> 。我们准备开始持久化数据Q徏立一?/span><span><code><span style="font-size:9.0pt;font-family:"Courier New"">Book</span></code> 然后保存到MongoDB中?/span></p> <div style="border:solid #EFEFEF 1.5pt;padding: 0cm 0cm 4.0pt 0cm;background:#FAFAFA"><pre style="line-height:11.25pt; background:#FAFAFA;border:none;padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">Book book = new Book();</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">book.setIsbn("1594743061");</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">book.setTitle("MongoDB is fun");</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">book.setDescription("...");</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span> </span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">book = dao.createObject("books", book);</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">System.out.println(book.getId()); // 4f96309f762dd76ece5a9595</span></span></span></pre></div> <p><span>首先建立</span><span><code><span style="font-size:9.0pt;font-family:"Courier New"">Book</span></code> 对象q且填|然后调用</span><span><code><span style="font-size:9.0pt;font-family:"Courier New"">MongoDao</span></code> ?/span><span><code><span style="font-size:9.0pt;font-family:"Courier New""> createObject</span></code> Ҏ(gu)Q将</span><span><code><span style="font-size:9.0pt;font-family:"Courier New"">Book</span></code> 对象传入"</span><span><span><code><span style="font-size:9.0pt; font-family:"Courier New"">books</span></code>" 的集合中。MJORM?x)按照之前的xml映射文g?/span></span><span><code><span style="font-size:9.0pt;font-family:"Courier New"">Book</span></code> 转换?/span><span><code><span style="font-size:9.0pt;font-family:"Courier New"">DBObject</span></code> (q是10gen的Java驱动使用的基本类?Qƈ保存一个新的文档进"</span><span><code><span style="font-size:9.0pt;font-family:"Courier New"">books</span></code>" 集合。MJORMq回Book对象Ӟid属性会(x)被填充。请注意QMongoDB默认是不需要在使用前徏立数据库或集合的Q系l会(x)在需要时自动创徏Q这可能?x)造成某些困扰。在MongoDB的命令行中查看Book对象大概如下Q?/span></p> <div style="border:solid #EFEFEF 1.5pt;padding: 0cm 0cm 4.0pt 0cm;background:#FAFAFA"><pre style="line-height:11.25pt; background:#FAFAFA;border:none;padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">> db.books.find({_id:ObjectId("4f96309f762dd76ece5a9595")}).pretty()</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">{</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         "_id":          ObjectId("4f96309f762dd76ece5a9595"),</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         "isbn":         "1594743061",</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         "title":        "MongoDB is fun",</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         "description": "..."</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">}</span></span></span></pre></div> <p><span> </span></p> <p><span>我们来看看假如不?span>MJORM而直接?0gen的Java驱动Q如何?/span></span><span><code><span style="font-size:9.0pt;font-family:"Courier New"">createObject</span></code> Ҏ(gu)Q?/span></p> <div style="border:solid #EFEFEF 1.5pt;padding: 0cm 0cm 4.0pt 0cm;background:#FAFAFA"><pre style="line-height:11.25pt; background:#FAFAFA;border:none;padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">Book book = new Book();</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">book.setIsbn("1594743061");</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">book.setTitle("MongoDB is fun");</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">book.setDescription("...");</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span> </span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">DBObject bookObj = BasicDBObjectBuilder.start()</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         .add("isbn",              book.getIsbn())</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         .add("title",             book.getTitle())</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         .add("description",       book.getDescription())</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         .get();</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span> </span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">// 'db' is our DB object from earlier</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">DBCollection col = db.getCollection("books");</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">col.insert(bookObj);</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span> </span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">ObjectId id = ObjectId.class.cast(bookObj.get("_id"));</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">System.out.println(id.toStringMongod()); // 4f96309f762dd76ece5a9595</span></span></span></pre></div> <p><span> </span></p> <p><span>下面q行对象的查?span>:</span></span></p> <div style="border:solid #EFEFEF 1.5pt;padding: 0cm 0cm 4.0pt 0cm;background:#FAFAFA"><pre style="line-height:11.25pt; background:#FAFAFA;border:none;padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">Book book = dao.readObject("books", "4f96309f762dd76ece5a9595", Book.class);</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">System.out.println(book.getTitle()); // "MongoDB is fun"</span></span></span></pre></div> <p><span><code><span style="font-size:9.0pt;font-family:"Courier New"">readObject</span></code> Ҏ(gu)Ҏ(gu)l定文档的id从指定的集合中读取文档,转换为对象(再次使用映射文gQƈq回?/span></p> <p><span>敏锐的读者会(x)注意?span>Bookq没有指定AuthorQ仍然保存了(jin)。这归咎于MongoDB的结构不敏感的特性。我们不能要求集合中的文档包含所有属性(id属性是必须的)(j)Q所有在MongoDB中没有Author的Book是可以的。我们现在ؓ(f)Bookd一个Authorq且更新一下:(x)</span></span></p> <div style="border:solid #EFEFEF 1.5pt;padding: 0cm 0cm 4.0pt 0cm;background:#FAFAFA"><pre style="line-height:11.25pt; background:#FAFAFA;border:none;padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">Author author = new Author();</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">author.setFirstName("Brian");</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">author.setLastName("Dilley");</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span> </span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">book.setAuthor(author);</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span> </span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">dao.updateObject("books", "4f96309f762dd76ece5a9595", book);</span></span></span></pre></div> <p><span>现在<span>Book包含了(jin)AuthorQƈ且在MongoDB中持久化?jin)。现在在命o(h)行查看了(jin)BookQ?/span></span></p> <div style="border:solid #EFEFEF 1.5pt;padding: 0cm 0cm 4.0pt 0cm;background:#FAFAFA"><pre style="line-height:11.25pt; background:#FAFAFA;border:none;padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">> db.books.find({_id:ObjectId("4f96309f762dd76ece5a9595")}).pretty()</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">{</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         "_id":          ObjectId("4f96309f762dd76ece5a9595"),</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         "isbn":         "1594743061",</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         "title":        "MongoDB is fun",</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         "description": "..."</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         "author": {</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">             "firstName": "Brian",</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">             "lastName": "Dilley"</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         }</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">}</span></span></span></pre></div> <p><span>可以看到持久化的<span>Book中已l包含了(jin)author。不使用MJORM来操作一遍:(x)</span></span></p> <div style="border:solid #EFEFEF 1.5pt;padding: 0cm 0cm 4.0pt 0cm;background:#FAFAFA"><pre style="line-height:11.25pt; background:#FAFAFA;border:none;padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">Author author = new Author();</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">author.setFirstName("Brian");</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">author.setLastName("Dilley");</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span> </span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">book.setAuthor(author);</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span> </span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">DBObject bookObj = BasicDBObjectBuilder.start()</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         .add("isbn",              book.getIsbn())</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         .add("title",             book.getTitle())</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         .add("description",       book.getDescription())</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         .push("author")</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">                 .add("firstName",         author.getFirstName())</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">                 .add("lastName", author.getLastName())</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">                 .pop()</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">         .get();</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span> </span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">DBCollection col = db.getCollection("books");</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span><span><span style="font-size:9.0pt; font-family:"Courier New";color:#222222">col.update(new BasicDBObject("_id", bookObj.get("_id")), bookObj);</span></span></span></pre><pre style="line-height:11.25pt;background:#FAFAFA;border:none; padding:0cm;"><span> </span></pre></div> <p><span> </span></p> <p><span>对于</span><span><code><span style="font-size:9.0pt;font-family:"Courier New"">MongoDao</span></code> Ҏ(gu)的深入讨论已l超Z(jin)本文的范围。对于将MJORM有兴用于实际项目中的用户强烈徏议了(jin)解一下MJORM目提供的相x档,或?/span><span><code><span style="font-size:9.0pt;font-family:"Courier New"">MongoDao</span></code> 接口提供的相关用法?/span></p> <h2><span><span><span style="font-size:14.5pt;color:black; letter-spacing:-.75pt">ȝ</span></span></span><span></span></h2> <p><span>希望q篇文章?span>MongoDB和MJORM的亮Ҏ(gu)所展示。MongDB是一个优U的呃NoSQL数据存储Q有着大量优秀的特性,?x)是NoSQL?jng)场中长期竞争者。若你会(x)在一个Java目中用MongoDBQ希望你也能够考虑使用MJORM作ؓ(f)你的ORM框架。十分欢q大家提交特性需求、错误异常报告、文档和源码修正?/span></span></p> <p style="line-height:12.0pt;background:white"><span> </span></p> <h2><span><span style="font-size:14.5pt;color:black;letter-spacing:-.75pt">作?/span></span><span><span><span style="font-size:14.5pt;font-family:Arial;color:black;letter-spacing: -.75pt"> Bio</span></span></span></h2> <p style="line-height:12.0pt;background:white"><span><span><strong><span style="font-size:10.0pt; font-family:Tahoma;color:black">Brian Dilley</span></strong></span></span><span><span> </span></span><span><span><span style="font-size:10.0pt;color:black">是一个经验丰富的高工程师以?qing)项目领|?/span></span></span><span><span><span style="font-size:10.0pt;font-family:Tahoma;color:black">Java/Java EE /Spring Framework/Linux</span></span></span><span><span style="font-size:10.0pt;color:black">内部l构理解和管理有着过</span></span><span><span style="font-size:10.0pt;font-family:Tahoma;color:black">13</span></span><span><span style="font-size:10.0pt;color:black">q的l验?/span></span><span><span style="font-size:10.0pt;font-family:Tahoma;color:black">Brian</span></span><span><span style="font-size:10.0pt;color:black">对于创业公司有很多经验,推向?jng)场Q构?/span></span><span><span style="font-size:10.0pt;font-family:Tahoma;color:black">/</span></span><span><span style="font-size:10.0pt;color:black">l护产品{。他?/span></span><span><span style="font-size:10.0pt;font-family:Tahoma;color:black">Iaas</span></span><span><span style="font-size:10.0pt;color:black">?/span></span><span><span style="font-size:10.0pt;font-family:Tahoma;color:black">cloud</span></span><span><span style="font-size:10.0pt;color:black">?/span></span><span><span style="font-size:10.0pt;font-family:Tahoma;color:black">PHP</span></span><span><span style="font-size:10.0pt;color:black">?/span></span><span><span style="font-size:10.0pt;font-family:Tahoma;color:black">Linux</span></span><span><span style="font-size:10.0pt;color:black">的专Ӟ熟?zhn)产品的采购、安装及(qing)配置定义Q以?qing)公司的软硬件架构包括负载均衡、数据库、微博等。可?/span></span><span><span style="font-size:10.0pt;font-family:Tahoma;color:black">follow Brian</span></span><span><span style="font-size:10.0pt;color:black">?/span></span><span><span> </span></span><a ><span><span><span style="font-size:10.0pt; font-family:Tahoma;color:#0B59B2">Twitter</span></span></span></a><span> </span><span><span style="font-size:10.0pt;color:black">?/span></span><span></span></p> </div><img src ="http://www.tkk7.com/matuobasyouca/aggbug/377698.html" width = "1" height = "1" /><br><br><div align=right><a style="text-decoration:none;" href="http://www.tkk7.com/matuobasyouca/" target="_blank">一酌散千忧</a> 2012-05-09 13:46 <a href="http://www.tkk7.com/matuobasyouca/archive/2012/05/09/377698.html#Feedback" target="_blank" style="text-decoration:none;">发表评论</a></div>]]></description></item><item><title>数据集成的演化:(x)从EII到Big Datahttp://www.tkk7.com/matuobasyouca/archive/2012/04/19/375213.html一酌散千忧一酌散千忧Wed, 18 Apr 2012 23:15:00 GMThttp://www.tkk7.com/matuobasyouca/archive/2012/04/19/375213.htmlhttp://www.tkk7.com/matuobasyouca/comments/375213.htmlhttp://www.tkk7.com/matuobasyouca/archive/2012/04/19/375213.html#Feedback0http://www.tkk7.com/matuobasyouca/comments/commentRss/375213.htmlhttp://www.tkk7.com/matuobasyouca/services/trackbacks/375213.html“企业信息集成Q?/span>EIIQ:(x)实用方式”?/span>2005q发布,描述?jin)一套集成不同数据源的方法论Q利用了(jin)当时的先q技术,如面向服务架构(SOAQ?/span>Web Services?/span>XML、资源描q架构(RDFQ、基?/span>XML的元数据格式以及(qing)数据提取、{换和加蝲Q?/span>ETLQ?/span>EII能够基本为关pd数据元素提供l一视角Q但在性能效率上缺乏能够替代数据仓库和多维数据库的能力。五q之后技术已l得C(jin)显著提升Q不仅在于对于分散数据的操作Q还有简化了(jin)单一容器下不同数据的整合Q以?qing)对数据深入挖掘的能力?/span>

转变?jin)数据管理方式的技术正是虚拟化。低成本存储、云计算?/span>NoSQL数据库以?/span>Hadoop。当我们提v虚拟化时Q已l远q超Zؓ(f)一台物理机器提供一套Y件实例这一概念。时至今日,我们可以虚拟化服务器、存储以?qing)网l。所有这些虚拟化意味着我们不再被这些物理条件所限制Q能够迅速构建物理环境以支持我们特定时刻的特定需求。当面对Gb?/span>Tb?/span>Pb{数据量的处理需求时Q我们基本能摆脱l构化的数据仓库。我们不在需要仅仅ؓ(f)?jin)发掘业务的某一斚w而徏立一个特D的环境?jin)?/span>

低成本存储在业务的数据存储方面节省了(jin)开支。高昂的存储成本?x)得企业寻扑֜限定规模的数据之上进行关键业务分析的?gu)Q这样得如何选择最重要的数据变得十分关键,而且q限制了(jin)pȝ能够处理的数据的质量?/span>

负面影响便是业务最l可能面临很的选择Q因为没有够的历史数据提供从而识别一个有效关键模式。或者因为高昂的投入使得业务被停止,而用常规惯例来识别模式?/span>

云计ؓ(f)那些需要通过量数据源在合理旉范围内生结果的需求提供了(jin)一个可用的方式。v量数据处理需要两点:(x)Ҏ(gu)存储,CPU。高速网l很有帮助,但是待会(x)我们?x)看到在发掘软g在处理v量数据时Q它q是系l的瓉。弹性存储意味着企业不会(x)在期望操作的数据规模或类型上受到限制Q降低了(jin)使用数据仓库无法获取最佳结果的风险。更多的CPU使得l果能够在期望的旉范围内更快的被交付?/span>

NoSQL提供?jin)v量数据的支持Q但与传l的关系型数据库没有兌。而且大部?/span>NoSQL数据库是开源的Q无L付购买证书等费用?/span>NoSQL对于表结构有着惊h的灵zL,无须随着pȝ的改q而不断修改完善定义?/span>NoSQL可以支持不同数据源的合ƈ查看Q从而成?/span>EII之后另一个备选方案,q或许是NoSQL最重要的方面了(jin)?/span>

NoSQL内置?jin)数据冗余与分布式数据存储机制。v量数据的最大问题之一是盘dQ?/span>NoSQL通过数据分布至一pd节点来缓解这个问题。当一个查询请求发出时Q这些节点能够ƈ行查询自w节点,而不是仅仅依靠一块磁盘,一个磁盘阵列或一条网l连接等Q数据查询能够在节省?jin)读写开支之后变得更加迅速?/span>

最l,我们来讨?/span>HadoopQ集合了(jin)上述所有技术力量与一w的用于(g)和分析数据的框架。有些h可能认ؓ(f)Hadoop是一?/span>NoSQL技术,实际?/span>Hadoop是一个分布组件的java框架Q用于分?#8220;吃大?#8221;Q此处也双关Hadoop是以创立者的儿子l自q一个大象玩兯v的名字)(j)的工?#8212;—每次一口?/span>

Hadoop自n实际上与待处理数据是各自独立的。它?yu)大型查询Q务分解ؓ(f)的q行查询dQ然后收集结果,q整合出{案q回l用戗?/span>Hadoop相对?/span>NoSQL来说是一个ƈ行查询框Ӟ通过云计驱动节点,q行在低成本存储?qing)虚拟化技术之上?/span>

Kicking的知识回?/span>

?/span>EIIW一ơ作为最?jng)_践出C2003-2004q_(d)关键要素是无需再移动数据了(jin)。当时大部分的数据中?j)仍然运行于低速网l中Q有限的I间用于复制数据。之后,EII成ؓ(f)?jin)当时可用技术和问题域中最优秀的解x案?/span>EII的某些方面的优秀即在v量数据中也是很显著的?/span>

EII的优点之一是处理过E{Ud数据所在地。v量数据方案的关键架构要素之一是处理过E{Ud数据所在地Q而不是{UL据?/span>EII中的一个重要原则就是用数据归属地的查询功能。这实践就是构建靠q数据源|络?/span>Web ServiceQ能够徏立v通用查询接口Q但只针Ҏ(gu)地数据库q行查询。我们通过开攄ZWeb的接口解决了(jin)数据的专有格式的问题Q从而得多个数据子集能够迅速的整合q以l一模式展示?/span>

有了(jin)低成本存储和10G|络之后Q我们就不必那么担心(j)数据冗余与数据迁U,但还是有其他问题存在的,数据仓库无法保数据的原始性便是其中之一。在EII中,我们从原始数据源获取数据视?#8220;黄金准则”Q这样就能够保证信息未被修改q,且是准确的?/span>

Big Data要求数据必须转移到新的物理位|,q样可信d又成Z(jin)问题?/span>EII的那些获取基U数据的最?jng)_践仍然是相关而且重要的。实际上Q那些ؓ(f)EII设计开发的Web Services接口最l在Big Data的启用中扮演主要角色?/span>

当然Q讨论数据管理不能不涉及(qing)到安全问题?/span>EII在安全领域中q是过?/span>Big Data。技术上来说Q?/span>Big Data在数据集成方面更加高效与敏捷Q但是大部分~少?jin)固有的安全性,因ؓ(f)在设计上?x)加大处理的隑ֺ。所以,可能要由源系l来担Qh据访问安全方面的责Q。因?/span>EII直接在源pȝ中查询数据,所以必要求有适当的授权,否则查询将p|?/span>

上述关于安全讨论描述的是内在的安全控制情c(din)将讉K权限控制列表集成q数据库是非常合理的Q这确保安全能够作为查询的一部分q行l护。然后,一旦能够直接查?/span>NoSQL数据源,意味着能够自由的访问你所有的数据?/span>

ȝ

引用老的Virginia Slims的广告中的台词:(x)“我们已经历很长的路途了(jin)Q宝贝儿Q?#8221;文中讨论到的技术的发展已经?/span>21世纪W二?/span>10q中的的数据解决Ҏ(gu)产生?jin)巨大的影响。商业化与小型化扫除?jin)一些思想体系上的障碍Q得架构师能够专注于问题本w,而不是寻找一些实用及(qing)可实现的问题解决Ҏ(gu)。构?/span>10000个节点的处理引擎Q能够在数秒内处?/span>PbU别的数据量Q却只消耗每时几便士,q就是数据处理的好前景?/span>

有了(jin)q些新工P我们p重新考虑如何推进数据理。ؓ(f)何数据无法被很好地被l护整合Qƈ且需要花Ҏ(gu)万美元。数据管理几乎是每个大中型企业的?j)病。数据管理曾l在存储、管理、访问、整合以?qing)查询上p巨大Q但是今后不再会(x)是这样了(jin)?/span>

关于作?/span>

JP Morgenthal 是在IT{略与云计算斚w的世界专家之一。他在企业复杂问题域的解x案实施上有着25q的l验?/span>JP Morgenthal以其在技术方面的深度和广度,有利的支持他在企业问题域中的敏感度。他在集成、Y件开发和云计是一位让人尊敬的作者,同时也是InfoQ在引领云计算斚w的编辑,q且参与?#8220;云计:(x)评估风险”目?/span> 

 

原文?/span>接:(x)http://www.infoq.com/articles/DataIntegrationFromEIItoBigData



一酌散千忧 2012-04-19 07:15 发表评论
]]>
Maven目理理解与?/title><link>http://www.tkk7.com/matuobasyouca/archive/2012/04/06/373470.html</link><dc:creator>一酌散千忧</dc:creator><author>一酌散千忧</author><pubDate>Fri, 06 Apr 2012 06:48:00 GMT</pubDate><guid>http://www.tkk7.com/matuobasyouca/archive/2012/04/06/373470.html</guid><wfw:comment>http://www.tkk7.com/matuobasyouca/comments/373470.html</wfw:comment><comments>http://www.tkk7.com/matuobasyouca/archive/2012/04/06/373470.html#Feedback</comments><slash:comments>0</slash:comments><wfw:commentRss>http://www.tkk7.com/matuobasyouca/comments/commentRss/373470.html</wfw:commentRss><trackback:ping>http://www.tkk7.com/matuobasyouca/services/trackbacks/373470.html</trackback:ping><description><![CDATA[<p><span style="font-family:宋体;Times New Roman";Times New Roman"">目l理Q?/span>Project manager<span style="font-family:宋体;Times New Roman";Times New Roman"">Q是目的支׃核心(j)Q维基百U的定义Q项目经理是目理斚w的专Ӟ负责目的策划、执行和l束Q即整个生命周期q程。项目经理手中的“q将莫邪”便是软g开发过E方法(</span>software development process/software development life cycle (SDLC)<span style="font-family: 宋体;Times New Roman";Times New Roman"">Q,可能采取的有</span>RUP<span style="font-family:宋体;Times New Roman";Times New Roman"">Q?/span>Rational Unified Process<span style="font-family:宋体;Times New Roman";Times New Roman"">Q,敏捷{?/span></p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">其实软g开发区分阶D已l广为大家接受,普遍的概念即需要区分ؓ(f)分析、设计、实施、测试、发布,q程中会(x)产生若干产物Q如需求说明书、概要设计、详l设计等。若提及(qing)q程Ҏ(gu)Q如</span>RUP<span style="font-family:宋体;Times New Roman";Times New Roman"">的话Q主要分为四大阶D,先启Q?/span>Inception<span style="font-family:宋体;Times New Roman";Times New Roman"">Q、精华(</span>Elaboration<span style="font-family:宋体;Times New Roman";Times New Roman"">Q、构建(</span>Construction<span style="font-family:宋体;Times New Roman";Times New Roman"">Q、交付(</span>Transition<span style="font-family:宋体;Times New Roman";Times New Roman"">QP代的开发方式,?/span>Scrum<span style="font-family:宋体;Times New Roman";Times New Roman"">的核?j)概念则?/span>Sprint<span style="font-family:宋体;Times New Roman";Times New Roman"">?/span></p> <p> </p> <p>Maven<span style="font-family:宋体;Times New Roman";Times New Roman"">在项目管理中有那些帮助呢Q?/span>Maven<span style="font-family:宋体;Times New Roman";Times New Roman"">能够从一个信息中?j)?f)目提供构徏Q报告,文档~制{工作。在</span>Maven<span style="font-family:宋体;Times New Roman";Times New Roman"">官方介绍?/span>What is maven<span style="font-family:宋体;Times New Roman";Times New Roman"">》中介绍?/span>maven<span style="font-family:宋体;Times New Roman";Times New Roman"">的项目目标(</span>Objectives<span style="font-family:宋体;Times New Roman";Times New Roman"">Q(</span><a >http://maven.apache.org/what-is-maven.html</a><span style="font-family:宋体;Times New Roman";Times New Roman"">Q,如下Q?/span></p> <p style="margin-left:21.0pt;text-indent:-21.0pt;"><span style="font-family: Wingdings;">l<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">         </span></span><span style="font-family:宋体;Times New Roman";Times New Roman"">化构E?/span></p> <p style="margin-left:21.0pt;text-indent:-21.0pt;"><span style="font-family: Wingdings;">l<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">         </span></span><span style="font-family:宋体;Times New Roman";Times New Roman"">提供l一的构建系l?/span></p> <p style="margin-left:21.0pt;text-indent:-21.0pt;"><span style="font-family: Wingdings;">l<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">         </span></span><span style="font-family:宋体;Times New Roman";Times New Roman"">提供目质量信息</span></p> <p style="margin-left:21.0pt;text-indent:-21.0pt;"><span style="font-family: Wingdings;">l<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">         </span></span><span style="font-family:宋体;Times New Roman";Times New Roman"">提供对于开发最?jng)_늚指导</span></p> <p style="margin-left:21.0pt;text-indent:-21.0pt;"><span style="font-family: Wingdings;">l<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">         </span></span><span style="font-family:宋体;Times New Roman";Times New Roman"">允许对于新特性的透明整合</span></p> <p> </p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">对于</span>Maven<span style="font-family:宋体;Times New Roman";Times New Roman"">影响最为深ȝ是它的构徏pȝQ几乎诏I了(jin)整个实施阶段。作为对比我们参考一?/span>RUP<span style="font-family:宋体;Times New Roman";Times New Roman"">?/span>Construction<span style="font-family:宋体;Times New Roman";Times New Roman"">阶段Q以?/span>Scrum<span style="font-family:宋体;Times New Roman";Times New Roman"">的单?/span>Sprint<span style="font-family:宋体;Times New Roman";Times New Roman"">q程?/span></p> <p> </p> <p>RUP<span style="font-family:宋体;Times New Roman";Times New Roman"">?/span>Construction<span style="font-family:宋体;Times New Roman";Times New Roman"">阶段的目标:(x)</span></p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">q个阶段的目标是澄清需求ƈZ架构基线完成开发?/span></p> <p style="margin-left:21.0pt;text-indent:-21.0pt;"><span style="font-size: 9.0pt;font-family:Wingdings;">l<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">          </span></span><span style="font-size:9.0pt;font-family:宋体;Times New Roman";Times New Roman"">通过优化资源来羃减开支,q免无意义的争执与q工?/span></p> <p style="margin-left:21.0pt;text-indent:-21.0pt;"><span style="font-size: 9.0pt;font-family:Wingdings;color:red">l<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">          </span></span><span style="font-size:9.0pt;font-family:宋体;Times New Roman";Times New Roman"; color:red">实用性与质量兼具?/span></p> <p style="margin-left:21.0pt;text-indent:-21.0pt;"><span style="font-size: 9.0pt;font-family:Wingdings;color:red">l<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">          </span></span><span style="font-size:9.0pt;font-family:宋体;Times New Roman";Times New Roman"; color:red">快发布可用版本?/span></p> <p style="margin-left:21.0pt;text-indent:-21.0pt;"><span style="font-size: 9.0pt;font-family:Wingdings;">l<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">          </span></span><span style="font-size:9.0pt;font-family:宋体;Times New Roman";Times New Roman"">完成Ҏ(gu)有功能的分析、设计、开发、测试?/span></p> <p style="margin-left:21.0pt;text-indent:-21.0pt;"><span style="font-size: 9.0pt;font-family:Wingdings;">l<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">          </span></span><span style="font-size:9.0pt;font-family:宋体;Times New Roman";Times New Roman"">采用增量q代的模式完成开发ƈ准备交付?/span></p> <p style="margin-left:21.0pt;text-indent:-21.0pt;"><span style="font-size: 9.0pt;font-family:Wingdings;color:red">l<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">          </span></span><span style="font-size:9.0pt;font-family:宋体;Times New Roman";Times New Roman"; color:red">(g)查项目发布的所有资源是否已l准备完全?/span></p> <p style="margin-left:21.0pt;text-indent:-21.0pt;"><span style="font-size: 9.0pt;font-family:Wingdings;">l<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">          </span></span><span style="font-size:9.0pt;font-family:宋体;Times New Roman";Times New Roman"">形成目l之间的q行开发?/span></p> <p> </p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">在《硝烟中?/span>Scrum<span style="font-family:宋体;Times New Roman";Times New Roman"">?/span>XP<span style="font-family: 宋体;Times New Roman";Times New Roman"">》一书中Q介l了(jin)作者实?/span>Scrum<span style="font-family:宋体;Times New Roman";Times New Roman"">的过E。在一?/span>Sprint<span style="font-family:宋体;Times New Roman";Times New Roman"">中,不是只有</span>Sprint backlog<span style="font-family:宋体;Times New Roman";Times New Roman"">?/span>burn down chat<span style="font-family:宋体;Times New Roman";Times New Roman"">{,实施q程中的敏捷思想也是其中的核?j),我们来看看敏捷信奉的一部分最?jng)_践:(x)</span></p> <p style="margin-left:21.0pt;text-indent:-21.0pt;"><span style="font-size: 9.0pt;font-family:Wingdings;">l<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">          </span></span><span style="font-size:9.0pt;font-family:宋体;Times New Roman";Times New Roman"">单设计(</span><span style="font-size:9.0pt">Simple Design</span><span style="font-size: 9.0pt;font-family:宋体;Times New Roman";Times New Roman"">Q?/span></p> <p style="margin-left:21.0pt;text-indent:-21.0pt;"><span style="font-size: 9.0pt;font-family:Wingdings;">l<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">          </span></span><span style="font-size:9.0pt;font-family:宋体;Times New Roman";Times New Roman"">l对~程Q?/span><span style="font-size:9.0pt">Pair Programming</span><span style="font-size:9.0pt;font-family:宋体;Times New Roman";Times New Roman"">Q?/span></p> <p style="margin-left:21.0pt;text-indent:-21.0pt;"><span style="font-size: 9.0pt;font-family:Wingdings;color:red">l<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">          </span></span><span style="font-size:9.0pt;font-family:宋体;Times New Roman";Times New Roman"; color:red">试驱动Q?/span><span style="font-size:9.0pt;color:red">Test-Driven Development</span><span style="font-size:9.0pt;font-family:宋体;Times New Roman";Times New Roman";color:red">Q?/span></p> <p style="margin-left:21.0pt;text-indent:-21.0pt;"><span style="font-size: 9.0pt;font-family:Wingdings;color:red">l<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">          </span></span><span style="font-size:9.0pt;font-family:宋体;Times New Roman";Times New Roman"; color:red">规模发布(</span><span style="font-size:9.0pt;color:red">Small Releases</span><span style="font-size:9.0pt;font-family:宋体;Times New Roman";Times New Roman";color:red">Q?/span></p> <p style="margin-left:21.0pt;text-indent:-21.0pt;"><span style="font-size: 9.0pt;font-family:Wingdings;color:red">l<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">          </span></span><span style="font-size:9.0pt;font-family:宋体;Times New Roman";Times New Roman"; color:red">持箋集成Q?/span><span style="font-size:9.0pt;color:red">Continuous Integration</span><span style="font-size:9.0pt;font-family:宋体;Times New Roman";Times New Roman";color:red">Q?/span></p> <p style="margin-left:21.0pt;text-indent:-21.0pt;"><span style="font-size: 9.0pt;font-family:Wingdings;">l<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">          </span></span><span style="font-size:9.0pt;font-family:宋体;Times New Roman";Times New Roman"">集体拥有代码Q?/span><span style="font-size:9.0pt">Collective Code Ownership</span><span style="font-size:9.0pt;font-family:宋体;Times New Roman";Times New Roman"">Q?/span></p> <p style="margin-left:21.0pt;text-indent:-21.0pt;"><span style="font-size: 9.0pt;font-family:Wingdings;">l<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">          </span></span><span style="font-size:9.0pt;font-family:宋体;Times New Roman";Times New Roman"">~码标准Q?/span><span style="font-size:9.0pt">Coding Standard</span><span style="font-size:9.0pt;font-family:宋体;Times New Roman";Times New Roman"">Q?/span></p> <p> </p> <p>Maven<span style="font-family:宋体;Times New Roman";Times New Roman"">对于上述目标中的质量Q实用性与质量Q以试驱动Q、可用(可用版本Q小规模Q、资源管理等均能发挥较大的作用。主要是其定义了(jin)一套完整优U的构建生命周期机Ӟ其基本阶D如下:(x)</span></p> <p style="margin-left:21.0pt;text-indent:-21.0pt;"><span style="font-family: Wingdings; color:red">l<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">         </span></span><strong><span style="color:red">validate</span></strong><span style="color:red"> – </span><span style="font-family:宋体;Times New Roman";Times New Roman";color:red">验证目正确性及(qing)依赖有效?/span></p> <p style="margin-left:21.0pt;text-indent:-21.0pt;"><span style="font-family: Wingdings;">l<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">         </span></span><strong>compile</strong> – <span style="font-family: 宋体;Times New Roman";Times New Roman"">~译目源码</span></p> <p style="margin-left:21.0pt;text-indent:-21.0pt;"><span style="font-family: Wingdings; color:red">l<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">         </span></span><strong><span style="color:red">test</span></strong><span style="color:red"> – </span><span style="font-family:宋体;Times New Roman";Times New Roman";color:red">使用合适的单元试框架对编译后的源码进行测试,试代码不会(x)被打包或发布</span></p> <p style="margin-left:21.0pt;text-indent:-21.0pt;"><span style="font-family: Wingdings;">l<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">         </span></span><strong>package</strong> –<span style="font-family: 宋体;Times New Roman";Times New Roman"">编译后的代码以规定格式打包Q如</span>Jar</p> <p style="margin-left:21.0pt;text-indent:-21.0pt;"><span style="font-family: Wingdings; color:red">l<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">         </span></span><strong><span style="color:red">integration-test</span></strong><span style="color:red"> – </span><span style="font-family:宋体;Times New Roman";Times New Roman";color:red">打包后的代码放|于环境中进行集成测?/span></p> <p style="margin-left:21.0pt;text-indent:-21.0pt;"><span style="font-family: Wingdings;">l<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">         </span></span><strong>verify</strong> – <span style="font-family: 宋体;Times New Roman";Times New Roman"">(g)查打包的有效性ƈ验证质量标准</span></p> <p style="margin-left:21.0pt;text-indent:-21.0pt;"><span style="font-family: Wingdings;">l<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">         </span></span><strong>install</strong> – <span style="font-family: 宋体;Times New Roman";Times New Roman"">包装蝲入本C库,以提供与其他目的依?/span></p> <p style="margin-left:21.0pt;text-indent:-21.0pt;"><span style="font-family: Wingdings;">l<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal; ">         </span></span><strong>deploy</strong> – <span style="font-family: 宋体;Times New Roman";Times New Roman"">包发布臌E仓库中</span></p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">其上每一个阶D实际都分ؓ(f)前中后三个阶D,用户可以定义在每一个阶D前后进行自定义的操作,打造自q构徏程Q如在某个阶D|行前制定Ҏ(gu)的配|文Ӟ完成后再改回默认Q。对于阶D늚实际使用方式Q如Q?/span>validate<span style="font-family:宋体;Times New Roman";Times New Roman"">可以目所有依赖有效,</span>test<span style="font-family:宋体;Times New Roman";Times New Roman"">可以针对</span>dao<span style="font-family:宋体;Times New Roman";Times New Roman"">层进行单元测试,</span>intergration-test<span style="font-family:宋体;Times New Roman";Times New Roman"">可以对完整业务流E或服务层等q行集成试?/span></p> <p> </p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">在项目中实际使用的经验,对于标签的用心(j)得:(x)</span></p> <p>1.<dependency> - <span style="font-family:宋体;Times New Roman";Times New Roman"">依赖标签Q最重要的标{,也是</span>Maven<span style="font-family:宋体;Times New Roman";Times New Roman"">的基功能?/span></p> <p>              <dependency></p> <p>                     <groupId>junit</groupId></p> <p>                     <artifactId>junit</artifactId></p> <p>                     <version>4.8.1</version></p> <p>                     <scope>test</scope></p> <p>              </dependency></p> <p> </p> <p>2. <repository> - <span style="font-family:宋体;Times New Roman";Times New Roman"">资源仓库Q可以包含多个,常用的有</span>Maven<span style="font-family:宋体;Times New Roman";Times New Roman"">Q?/span>Jboss<span style="font-family: 宋体;Times New Roman";Times New Roman"">{,如下是公司内建的</span>Nexus<span style="font-family:宋体;Times New Roman";Times New Roman"">资源库?/span></p> <p style="margin-left:21.0pt;text-indent:21.0pt"><repository></p> <p>                     <id>Suntang's Maven Repository</id></p> <p>                     <name>Suntang Nexus Repository</name></p> <p>                     <url>http://10.10.10.33:8081/nexus/content/groups/public</url></p> <p>              </repository></p> <p>3. <profile> - <span style="font-family:宋体;Times New Roman";Times New Roman"">解释为情景模式可能较为合适。可以有多个Q在何种场景下会(x)使用哪些属性、插件等。如下例子便是当~失某个文g时激zR?/span></p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">Q感觉写的不错的一,</span><a >http://blog.csdn.net/turkeyzhou/article/details/4894657</a><span style="font-family:宋体;Times New Roman";Times New Roman"">Q?/span></p> <p style="margin-left:42.0pt;"><profile></p> <p style="margin-left:63.0pt;"> <activation></p> <p style="margin-left:84.0pt;"><file></p> <p style="margin-left:105.0pt;"><missing>target/generated-sources/axistools/wsdl2java/org/apache/maven</missing></p> <p style="margin-left:84.0pt;"></file></p> <p style="margin-left:63.0pt;"></activation></p> <p style="margin-left:42.0pt;"> </profile></p> <p>4.<build> - <span style="font-family:宋体;Times New Roman";Times New Roman"">构徏q程。是q行整个目理的核?j)标{。重炚w要掌握的知识是生命周期?/span></p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">Q?/span><a >http://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html#Lifecycle_Reference</a><span style="font-family:宋体;Times New Roman";Times New Roman"">Q?/span></p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">下面的例子就是制定了(jin)打包时的资源路径Qƈ且定义了(jin)最l打包的名称?/span></p> <p> </p> <p>       <build></p> <p>              <resources></p> <p>                     <resource></p> <p>                            <directory>src/main/resources</directory></p> <p>                            <includes></p> <p>                                   <include>**/*</include></p> <p>                            </includes></p> <p>                     </resource></p> <p>                     <resource></p> <p>                            <directory>src/main/assembly</directory></p> <p>                            <includes></p> <p>                                   <include>**/*</include></p> <p>                            </includes></p> <p>                     </resource></p> <p>              </resources></p> <p> </p> <p>              <finalName>po</finalName></p> <p>       </build></p> <p> </p> <p>4.< plugin > - <span style="font-family:宋体;Times New Roman";Times New Roman"">支持插g。如单元试自动化,之前提到?/span>Ant<span style="font-family:宋体;Times New Roman";Times New Roman"">的插件等。若有某些功能觉得不手Q可以尝试官|找一下有没有合适的插gQ?/span><a >http://maven.apache.org/plugins/index.html</a><span style="font-family:宋体;Times New Roman";Times New Roman"">Q?/span></p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">下面的例子就是在集成试中,只运行后~?/span>TestSuitex.java<span style="font-family:宋体;Times New Roman";Times New Roman"">的测试类</span></p> <p> </p> <p>                     <plugin></p> <p>                            <groupId>org.apache.maven.plugins</groupId></p> <p>                            <artifactId>maven-surefire-plugin</artifactId></p> <p>                            <version>2.4.3</version></p> <p>                            <configuration></p> <p>                                   <junitArtifactName>junit:junit</junitArtifactName></p> <p>                                   <forkMode>once</forkMode></p> <p>                            </configuration></p> <p>                            <executions></p> <p>                                   <execution></p> <p>                                          <id>default-test</id></p> <p>                                          <phase>integration-test</phase></p> <p>                                          <goals></p> <p>                                                 <goal>test</goal></p> <p>                                          </goals></p> <p>                                          <configuration></p> <p>                                                 <skip>false</skip></p> <p>                                                 <includes></p> <p>                                                        <include>**/*TestSuitex.java</include></p> <p>                                                 </includes></p> <p>                                          </configuration></p> <p>                                   </execution></p> <p>                            </executions></p> <p>                     </plugin></p> <p> </p><img src ="http://www.tkk7.com/matuobasyouca/aggbug/373470.html" width = "1" height = "1" /><br><br><div align=right><a style="text-decoration:none;" href="http://www.tkk7.com/matuobasyouca/" target="_blank">一酌散千忧</a> 2012-04-06 14:48 <a href="http://www.tkk7.com/matuobasyouca/archive/2012/04/06/373470.html#Feedback" target="_blank" style="text-decoration:none;">发表评论</a></div>]]></description></item><item><title>maven自动化测试环境搭?/title><link>http://www.tkk7.com/matuobasyouca/archive/2012/04/05/373359.html</link><dc:creator>一酌散千忧</dc:creator><author>一酌散千忧</author><pubDate>Thu, 05 Apr 2012 03:26:00 GMT</pubDate><guid>http://www.tkk7.com/matuobasyouca/archive/2012/04/05/373359.html</guid><wfw:comment>http://www.tkk7.com/matuobasyouca/comments/373359.html</wfw:comment><comments>http://www.tkk7.com/matuobasyouca/archive/2012/04/05/373359.html#Feedback</comments><slash:comments>0</slash:comments><wfw:commentRss>http://www.tkk7.com/matuobasyouca/comments/commentRss/373359.html</wfw:commentRss><trackback:ping>http://www.tkk7.com/matuobasyouca/services/trackbacks/373359.html</trackback:ping><description><![CDATA[<p><span style="font-family: 宋体; ">环境背景Q?/span></p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">我作为项目经理和技术架构管理h员负责公怸条生产线。讨Z后,首席架构师希望我们能够实?/span>TDD<span style="font-family:宋体;Times New Roman";Times New Roman"">。在实施</span>TDD<span style="font-family:宋体;Times New Roman";Times New Roman"">的过E中Q设计实施过E的整体思\是Q单元测试用例文?/span> - <span style="font-family:宋体;Times New Roman";Times New Roman"">实施单元试</span> - <span style="font-family:宋体;Times New Roman";Times New Roman"">实施业务代码</span> – <span style="font-family:宋体;Times New Roman";Times New Roman"">修改业务代码逻辑。实施h员需要参与每个环节,按照规范~写单元试用例文档。单元测试我们按照模块(模块与h员基本没有重合)(j)划分包(</span>suite<span style="font-family:宋体;Times New Roman";Times New Roman"">Q,保证实施h不会(x)产生q扰。。技术架构决定采用:(x)</span>maven<span style="font-family:宋体;Times New Roman";Times New Roman"">Q?/span>junit<span style="font-family:宋体;Times New Roman";Times New Roman"">Q?/span>svn<span style="font-family: 宋体;Times New Roman";Times New Roman"">?/span></p> <p> </p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">技术背景:(x)</span></p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">技术架构设计上Q我们封装了(jin)</span>dao<span style="font-family:宋体;Times New Roman";Times New Roman"">层的实现Q所以实施h员基本无需涉及(qing)</span>dao<span style="font-family:宋体;Times New Roman";Times New Roman"">层的开发。服务层我们采用?/span>JAX-RS<span style="font-family:宋体;Times New Roman";Times New Roman"">的服务规范,对外开发服务接口?/span></p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">在测试覆盖率斚wQ我们基本不要求?/span>dao<span style="font-family:宋体;Times New Roman";Times New Roman"">层的单元试Q但要求在服务层的单元测试达?/span>100%<span style="font-family:宋体;Times New Roman";Times New Roman"">。由于服务层?/span>Restful WS<span style="font-family:宋体;Times New Roman";Times New Roman"">的模式,所以我们采用了(jin)模拟</span>HTTP<span style="font-family:宋体;Times New Roman";Times New Roman"">h的方式在试服务层?/span></p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">׃需要模?/span>HTTP<span style="font-family:宋体;Times New Roman";Times New Roman"">的请求,所以我们在单元试中采用了(jin)</span>jetty<span style="font-family:宋体;Times New Roman";Times New Roman"">作ؓ(f)内嵌服务器,单元试开始时同一启动Q完成后关闭?/span></p> <p> </p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">实施q程Q?/span></p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">开发过E中Q实际实施的时候发C个问题,对于试数据的管理问题。即试当中需要一定的数据环境来验证业务逻辑。这个数据环境如何徏立?</span></p> <p> </p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">Ҏ(gu)一Q?/span>dbunit<span style="font-family:宋体;Times New Roman";Times New Roman"">?/span>hsqldb<span style="font-family:宋体;Times New Roman";Times New Roman"">。在试启动旉建数据环境?/span></p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">否决Q原因:(x)</span></p> <p>1.<span style="font-family:宋体;Times New Roman";Times New Roman"">与实际运行环境差异较大?/span></p> <p>2.<span style="font-family:宋体;Times New Roman";Times New Roman"">反复重徏数据环境Q效率上有缺失?/span></p> <p>3.<span style="font-family:宋体;Times New Roman";Times New Roman"">技术架构增加,学习(fn)和维护曲U较大?/span></p> <p> </p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">讨论后决定?/span></p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">Ҏ(gu)二,独立Z套测试数据库Q完整数据环境。考虑到增删改与查询的冲突Q制定默认规则,?/span>id<span style="font-family:宋体;Times New Roman";Times New Roman"">?/span>20<span style="font-family:宋体;Times New Roman";Times New Roman"">之内的不允许q行M改动。以量隔离增删改的影响?/span></p> <p> </p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">针对Ҏ(gu)二,有一个较大的问题Q如何在开发过E中自由的切换数据库配置呢?׃我们q是用了(jin)</span>Hudson<span style="font-family:宋体;Times New Roman";Times New Roman"">作ؓ(f)</span>CI<span style="font-family:宋体;Times New Roman";Times New Roman"">服务器,q要考虑到打包的q程。整体考虑之后Q有两个步骤需要注意:(x)</span></p> <p style="margin-left:21.0pt;text-indent:-21.0pt;">一?span style="font-family:宋体;Times New Roman";Times New Roman"">开发过E。开发过E中Q我们将配置直接指向试数据库?/span></p> <p style="margin-left:21.0pt;text-indent:-21.0pt;">二?span style="font-family:宋体;Times New Roman";Times New Roman"">打包q程。用了(jin)</span>maven<span style="font-family:宋体;Times New Roman";Times New Roman"">Q存在单元测试配|与最l品配|的冲突?/span></p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">所以最l问题的焦点集中在打包过E的</span>maven<span style="font-family:宋体;Times New Roman";Times New Roman"">配置Ҏ(gu)?/span></p> <p> </p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">搜烦(ch)之后比较好的资料?/span></p> <p>MAVEN:<span style="font-family:宋体;Times New Roman";Times New Roman"">如何为开发和生环境建立不同的配|文?/span> --<span style="font-family:宋体;Times New Roman";Times New Roman"">我的z方?/span></p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">Q?/span><a href="http://www.tkk7.com/scud/archive/2010/10/27/336326.html">http://www.tkk7.com/scud/archive/2010/10/27/336326.html</a><span style="font-family:宋体;Times New Roman";Times New Roman"">Q?/span></p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">q篇博客是介l在</span>maven <span style="font-family:宋体;Times New Roman";Times New Roman"">中?/span>mvn package -P test <span style="font-family:宋体;Times New Roman";Times New Roman"">q样的自定义</span>profile<span style="font-family:宋体;Times New Roman";Times New Roman"">来实现的。这h可行的,但是?/span>Hudson<span style="font-family:宋体;Times New Roman";Times New Roman"">中无法实C条命令切换两套配|?/span></p> <p><span style="font-family:宋体;Times New Roman";Times New Roman"">于是l箋LQ最l在</span>maven<span style="font-family:宋体;Times New Roman";Times New Roman"">的官方网站找到?/span>Building For Different Environments with Maven 2<span style="font-family:宋体;Times New Roman";Times New Roman"">》(</span><a >http://maven.apache.org/guides/mini/guide-building-for-different-environments.html</a><span style="font-family:宋体;Times New Roman";Times New Roman"">Q看完文章之后发玎ͼ实际</span>maven<span style="font-family:宋体;Times New Roman";Times New Roman"">提供?jin)一个非常好的插?/span>maven-antrun-plugin<span style="font-family:宋体;Times New Roman";Times New Roman"">Q以实现某些</span>ant<span style="font-family:宋体;Times New Roman";Times New Roman"">的功能。此处还需要了(jin)解的知识是</span>maven<span style="font-family:宋体;Times New Roman";Times New Roman"">的构建生命周期标准(</span>http://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html<span style="font-family:宋体;Times New Roman";Times New Roman"">Q。基于上qC个知识点Q我们制定出如下Ҏ(gu)Q?strong>在项目中建立试配置目录?qing)品配|目录,?/strong></span><strong>maven</strong><strong><span style="font-family:宋体;Times New Roman";Times New Roman"">?/span>package</strong><strong><span style="font-family:宋体;Times New Roman";Times New Roman"">阶段开始前Q都使用试配置Q运行集成测试,完成?/span>package</strong><strong><span style="font-family:宋体;Times New Roman";Times New Roman"">阶段前将产品配置覆盖x包文件夹内,然后q行打包</span></strong><span style="font-family:宋体;Times New Roman";Times New Roman"">。思\是q样Q下面脓(chung)?/span>pom<span style="font-family:宋体;Times New Roman";Times New Roman"">文g的关键部分?/span></p> <p> </p> <p> </p> <p>POM.xml</p> <p><!—profile <span style="font-family:宋体;Times New Roman";Times New Roman"">节点定义覆盖文g的方式内?/span> --></p> <p>       <profiles></p> <p>              <profile></p> <p>                     <id>product</id></p> <p>                     <build></p> <p>                            <plugins></p> <p>                                   <plugin></p> <p>                                          <artifactId>maven-antrun-plugin</artifactId></p> <p>                                          <executions></p> <p>                                                 <execution></p> <p>                                                        <id>pre_product</id></p> <p>                                                        <phase>prepare-package</phase></p> <p>                                                        <goals></p> <p>                                                               <goal>run</goal></p> <p>                                                        </goals></p> <p>                                                        <configuration></p> <p><!—<span style="font-family:宋体;Times New Roman";Times New Roman"">此处?/span>ant<span style="font-family:宋体;Times New Roman";Times New Roman"">的Q务相?/span> --></p> <p>                                                               <tasks></p> <p>                                                                      <delete file="${project.build.outputDirectory}/spring/dataSourceContext.xml" /></p> <p>                                                                      <delete file="${project.build.outputDirectory}/log4j.properties" /></p> <p>                                                                      <copy file="src/product/assembly/log4j.properties" tofile="${project.build.outputDirectory}/log4j.properties" /></p> <p>                                                                      <copy file="src/product/assembly/spring/dataSourceContext.xml" tofile="${project.build.outputDirectory}/spring/dataSourceContext.xml" /></p> <p>                                                               </tasks></p> <p>                                                        </configuration></p> <p>                                                 </execution></p> <p>                                          </executions></p> <p>                                   </plugin></p> <p>                            </plugins></p> <p>                     </build></p> <p>              </profile></p> <p>       </profiles></p> <p> </p> <p><!—<span style="font-family:宋体;Times New Roman";Times New Roman"">构徏q程</span> --></p> <p>       <build></p> <p style="text-indent:21.0pt"><!—<span style="font-family:宋体;Times New Roman";Times New Roman"">指定资源目录</span> --></p> <p>              <resources></p> <p>                     <resource></p> <p>                            <directory>src/test/resources</directory></p> <p>                            <includes></p> <p>                                   <include>**/*</include></p> <p>                            </includes></p> <p>                     </resource></p> <p>                     <resource></p> <p>                            <directory>src/test/assembly</directory></p> <p>                            <includes></p> <p>                                   <include>**/*</include></p> <p>                            </includes></p> <p>                     </resource></p> <p>              </resources></p> <p> </p> <p>              <finalName>po</finalName></p> <p> </p> <p style="margin-left:21.0pt;text-indent:21.0pt"><!—<span style="font-family:宋体;Times New Roman";Times New Roman"">指定集成试配置</span> --></p> <p>              <plugins></p> <p>                     <plugin></p> <p>                            <groupId>org.apache.maven.plugins</groupId></p> <p>                            <artifactId>maven-surefire-plugin</artifactId></p> <p>                            <version>2.4.3</version></p> <p>                            <configuration></p> <p>                                   <junitArtifactName>junit:junit</junitArtifactName></p> <p>                                   <forkMode>once</forkMode></p> <p>                            </configuration></p> <p>                            <executions></p> <p>                                   <execution></p> <p>                                          <id>default-test</id></p> <p>                                          <phase>integration-test</phase></p> <p>                                          <goals></p> <p>                                                 <goal>test</goal></p> <p>                                          </goals></p> <p>                                          <configuration></p> <p>                                                 <skip>false</skip></p> <p>                                                 <includes></p> <p>                                                        <include>**/*TestSuitex.java</include></p> <p>                                                 </includes></p> <p>                                          </configuration></p> <p>                                   </execution></p> <p>                            </executions></p> <p>                     </plugin></p> <p>              </plugins></p> <p>       </build></p><img src ="http://www.tkk7.com/matuobasyouca/aggbug/373359.html" width = "1" height = "1" /><br><br><div align=right><a style="text-decoration:none;" href="http://www.tkk7.com/matuobasyouca/" target="_blank">一酌散千忧</a> 2012-04-05 11:26 <a href="http://www.tkk7.com/matuobasyouca/archive/2012/04/05/373359.html#Feedback" target="_blank" style="text-decoration:none;">发表评论</a></div>]]></description></item><item><title>Nosql企业之道http://www.tkk7.com/matuobasyouca/archive/2012/04/04/373338.html一酌散千忧一酌散千忧Wed, 04 Apr 2012 13:41:00 GMThttp://www.tkk7.com/matuobasyouca/archive/2012/04/04/373338.htmlhttp://www.tkk7.com/matuobasyouca/comments/373338.htmlhttp://www.tkk7.com/matuobasyouca/archive/2012/04/04/373338.html#Feedback0http://www.tkk7.com/matuobasyouca/comments/commentRss/373338.htmlhttp://www.tkk7.com/matuobasyouca/services/trackbacks/373338.html阅读全文

一酌散千忧 2012-04-04 21:41 发表评论
]]>
Hadoop in action 实践(伪分布式)http://www.tkk7.com/matuobasyouca/archive/2012/04/01/373194.html一酌散千忧一酌散千忧Sun, 01 Apr 2012 07:00:00 GMThttp://www.tkk7.com/matuobasyouca/archive/2012/04/01/373194.htmlhttp://www.tkk7.com/matuobasyouca/comments/373194.htmlhttp://www.tkk7.com/matuobasyouca/archive/2012/04/01/373194.html#Feedback0http://www.tkk7.com/matuobasyouca/comments/commentRss/373194.htmlhttp://www.tkk7.com/matuobasyouca/services/trackbacks/373194.html《Hadoop in action?span style="font-family: 宋体; ">?/span>Manning出版Q磕绊lȝ是看完了(jin)。书的内容就不做介绍Q主要讲一下实늚q程。ƈ且在实践q程中参考的书籍的部分也?x)简单介l?br />
灰色背景部分Z些介l,或过E中出现问题的描qͼ可以直接忽略?/span>

׃公司的业务需要,要在|络攉|页之后对网进行结构化的解析,q个l构化过E希望能够基?/span>HDFSq且使用MR法实现?/span>

我虚拟了(jin)一个需求,针对http://hadoop.apache.org/common/releases.html 面Q假讑ַl下载了(jin)面q入库。要求最l体现的数据?/span> “版本?/span>+完整链接Q即a标签全部内容Q?#8221; 的结构?/span>

 

伪分布式环境搭徏在虚拟机上,操作pȝ?/span>centos5.5Q?/span>hadoop的版本是1.0.0.

 

插入书中的一些环境搭建的介绍

书中?/span>2.1节中介绍?jin)每个进E的作用?/span>2.2节中?/span>ssh讄也比较重要,否则好像?x)一直提CZ输入密码?/span>2.3.2节介l了(jin)伪分布式的配|方式,?/span>core-site.xmlQ?/span>mapred-site.xmlQ?/span>hdfs-site.xmlq行配置之后Q需要对namenode节点q行格式化?/span>

bin/hadoop namenode –format

 

hadoop的根目录?/span>/usr/local/hadoop-1.0.0Q直接启?/span>start-all.sh

 

[root@localhost hadoop-1.0.0]# ./bin/start-all.sh

 

启动成功后?/span>jps命o(h)

jdk工?/span>jps介绍 

jps(Java Virtual Machine Process Status Tool)?/span>JDK 1.5提供的一个显C当前所?/span>javaq程pid的命令,单实用,非常适合?/span>linux/unixq_上简单察看当?/span>javaq程的一些简单情c(din)?/span> jps存放?/span>JAVA_HOME/bin/jps

 

[root@localhost hadoop-1.0.0]# jps

5694 SecondaryNameNode

5461 NameNode

5578 DataNode

6027 Jps

5784 JobTracker

5905 TaskTracker

q几个进E是非常重要的。很多时候出现意外就是因为某Ҏ(gu)务未启动或异常。可以看C面的命o(h)上打印出日志位置。出现异常后可以在日志中查看详细的堆栈信息?/span>

 

xQ?/span>hadoop已经启动Q环境已l准备就l?/span>

 

下面准备我们的测试数据,目标页面的html保存?/span>news.txtQ伪分布式也同样支持hdfsQ所以我们?/span> fs –put ?/span>news.txt存入hdfs中?/span>

[root@localhost hadoop-1.0.0]# ./bin/hadoop fs -put /mnt/hgfs/shared/news.txt /user/root

[root@localhost hadoop-1.0.0]# ./bin/hadoop fs -lsr /userdrwxr-xr-x   - root supergroup          0 2012-04-01 11:22 /user/root

-rw-r--r--   1 root supergroup       3935 2012-04-01 11:22 /user/root/news.txt

 

实现的代码在eclipse中?/span>maven打包Q上传至虚拟机?/span>

文g?/span>com.suntang.analyse.hadoop-0.0.1.jar?br />使用hadoop的中的jar命o(h)调用该jar文g?br />


[root@localhost hadoop-1.0.0]# ./bin/hadoop jar com.suntang.analyse.hadoop-0.0.1.jar com.suntang.analyse.hadoop.AnalyseJob /user/root/news.txt output_root_test

12/04/01 14:40:04 INFO input.FileInputFormat: Total input paths to process : 1

12/04/01 14:40:05 INFO mapred.JobClient: Running job: job_201204011420_0001

12/04/01 14:40:06 INFO mapred.JobClient:  map 0% reduce 0%

12/04/01 14:40:19 INFO mapred.JobClient:  map 100% reduce 0%

12/04/01 14:40:31 INFO mapred.JobClient:  map 100% reduce 100%

12/04/01 14:40:37 INFO mapred.JobClient: Job complete: job_201204011420_0001

 

 

此处注意我犯的一个错误:(x)

[root@localhost hadoop-1.0.0]# ./bin/hadoop jar com.suntang.analyse.hadoop-0.0.1.jar AnalyseJob -libjars hadoop-core-1.0.0.jar /user/root/news.txt output_root_test

Exception in thread "main" java.lang.ClassNotFoundException: AnalyseJob

提示找不到类Q因为我忘了(jin)写完整类名,命o(h)应该改ؓ(f)

./bin/hadoop jar com.suntang.analyse.hadoop-0.0.1.jar com.suntang.analyse.hadoop.AnalyseJob -libjars hadoop-core-1.0.0.jar /user/root/news.txt output_root_test 卛_?/span>

 

此处q行可能出现另外一个错误。在命o(h)行中出现

12/04/01 14:01:38 INFO mapred.JobClient: Task Id : attempt_201204011356_0001_m_000001_0, Status : FAILED

java.lang.Throwable: Child Error

        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)

Caused by: java.io.IOException: Creation of symlink from /mnt/hgfs/shared/hadoop-1.0.0/libexec/../logs/userlogs/job_201204011356_0001/attempt_201204011356_0001_m_000001_0 to 。。?/span>

׃打全?jin),重点在?/span>

Creation of symlinkQ看详细日志?/span>hadoop-root-tasktracker-localhost.localdomain.log中提C?/span>org.apache.hadoop.fs.FileUtil: Command 'ln -s ....': Operation not supportedQ即ln操作不支持?/span>google可知q个是由?/span>vm中的׃n区域的问题,解决Ҏ(gu)是?/span>hadoop完全转移?/span>linux目录中。本例中?/span>/mnt/hgfs/shared/hadoop-1.0.0转移?/span>/usr/local/hadoop-1.0.0?/span>

 

执行完成后可?/span>hdfs中查看结果,查看目录l构?/span>

-rw-r--r--   1 root supergroup          0 2012-04-01 14:40 /user/root/output_root_test/_SUCCESS

drwxr-xr-x   - root supergroup          0 2012-04-01 14:40 /user/root/output_root_test/_logs

drwxr-xr-x   - root supergroup          0 2012-04-01 14:40 /user/root/output_root_test/_logs/history

-rw-r--r--   1 root supergroup      13634 2012-04-01 14:40 /user/root/output_root_test/_logs/history/job_201204011420_0001_1333262405103_root_ccAnalyseJob

-rw-r--r--   1 root supergroup      20478 2012-04-01 14:40 /user/root/output_root_test/_logs/history/job_201204011420_0001_conf.xml

-rw-r--r--   1 root supergroup       3580 2012-04-01 14:40 /user/root/output_root_test/part-r-00000

 

/user/root/output_root_test/part-r-00000即ؓ(f)最l结果文件?/span>

 

 =======================================================================

附加AnalyseJob代码

package com.suntang.analyse.hadoop;

 

import java.io.IOException;

 

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.conf.Configured;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.Mapper;

import org.apache.hadoop.mapreduce.Reducer;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

import org.apache.hadoop.util.Tool;

import org.apache.hadoop.util.ToolRunner;

 

public class AnalyseJob extends Configured implements Tool {

 

       public static class MapClass extends Mapper<LongWritable, Text, Text, Text> {

 

              @Override

              protected void map(LongWritable key, Text value, Context context)

                            throws IOException, InterruptedException {

                    

                    

                     // TODO Auto-generated method stub

                     // super.map(key, value, context);

                     if (value.toString().matches("<a[^>]*>.*?release.*?</a>"))

                            context.write(

                                          new Text(value.toString().substring(

                                                        value.toString().indexOf("release") + 8,

                                                        value.toString().indexOf("available") - 1)),

                                          value);

              }

 

       }

 

       public static class ReduceClass extends Reducer<Text, Text, Text, Text> {

 

              @Override

              protected void reduce(Text arg0, Iterable<Text> arg1, Context arg2)

                            throws IOException, InterruptedException {

                     // TODO Auto-generated method stub

                     // super.reduce(arg0, arg1, arg2);

                     arg2.write(arg0, arg1.iterator().next());

              }

 

       }

 

       public int run(String[] args) throws Exception {

              Configuration conf = getConf();

 

              Job job = new Job(conf, "myAnalyseJob");

              job.setJarByClass(getClass());

 

              Path in = new Path(args[0]);

              Path out = new Path(args[1]);

              FileInputFormat.setInputPaths(job, in);

              FileOutputFormat.setOutputPath(job, out);

 

              job.setMapperClass(MapClass.class);

              job.setReducerClass(ReduceClass.class);

 

              job.setInputFormatClass(TextInputFormat.class);

              job.setOutputFormatClass(TextOutputFormat.class);

              job.setOutputKeyClass(Text.class);

              job.setOutputValueClass(Text.class);

 

              System.exit(job.waitForCompletion(true) ? 0 : 1);

 

              return 0;

       }

 

       public static void main(String[] args) throws Exception {

              int res = ToolRunner.run(new Configuration(), new AnalyseJob(), args);

              System.exit(res);

       }

}



一酌散千忧 2012-04-01 15:00 发表评论
]]>
վ֩ģ壺 wwwƵ߹ۿ| ۺϹһ| ŷպۺϾþþ| ѵҰսƵ| ۺһ| ޷츾| һƵѹۿ| ɫAV| þþþŮʦһ| ۺҹ| Ļ߹ۿƬ| þþþŮۺ| Ʒվ| ޳aƬ߹ۿ| Ļȫ| ޾Ʒ벻߲HE | jizzjizzƵѿ| ëƬAV뾫Ʒҹ| ƷѵĻվ | 2019ĻѴȫ5| ҹߵӰ| 91ƷѸ| ˾þô߽| һëƬѹۿ| ۺ߳һ| AAƬٸAAƬֱ| 츾AVպ츾| ѹҹ˽Ӱ| һaƬþëƬѿ | ˾þƵ| ޳ҹӰ| ˬָ߳Ƶ| ˳ͼƬվ| Ƶ| һƵѹۿ| þ޹ŷ޾Ʒһ| 100018Ƶ | ޹Ʒþþϼ1| ĻӰ߹ۿ| ߳þѹۿ| ޹|