无码专区一va亚洲v专区在线,国产亚洲?V无码?V男人的天堂,国产亚洲综合成人91精品

HADOOP各种框架应用领域

paulwong — Sun, 04 Jan 2015 04:57:00 GMT

***** Data Analytics : Technology Area *****
1. Real Time Analytics : Apache Storm
2. In-memory Analytics : Apache Spark
3. Search Analytics : Apache Elastic search, SOLR
4. Log Analytics : Apache ELK Stack,ESK Stack(Elastic Search, Log
Stash, Spark Streaming, Kibana)
5. Batch Analytics : Apache MapReduce

***** NO SQL DB *****
1. MongoDB
2. Hbase
3. Cassandra

***** SOA *****
1. Oracle SOA
2. JBoss SOA
3. TiBco SOA
4. SOAP, RESTful Webservices

paulwong 2015-01-04 12:57 发表评论

最火爆的开源流式系�l�Storm vs 新星Samza

paulwong — Tue, 02 Dec 2014 07:03:00 GMT

分布计算�pȝ��框架�Q�按照数据集的特�Ҏ(gu��)��_(d��)��主要分�ؓ(f��)data-flow和streaming两种。data-flow主要是以数据块�ؓ(f��)数据源来处理数据�Q�代表有�Q�MR、Spark�{�，我称作它们�ؓ(f��)大数据，而streaming主要是处理单位内得到的数据，�q�种方式�Q�更注重于实时性，主要包括Strom、JStorm和Samza�{�，我称作它们�ؓ(f��)快数据�?/p>

在这��文章中�Q�我主要谈论streaming相关的框架�?/p>

�W�一个是Storm�Q�一个实时计��系�l�，它假定数据源是动态的�Q�可以向��水一样处理数据�?/p>

它的特点是：(x��)低�g�q�、高性能、分布式、可扩展和容错性�?/p>

架构如下图所�C��?/p>

Storm的具体概念可以参照：(x��)http://blog.csdn.net/hljlzc2007/article/details/12976211�Q�这里不做具体介�l��?/p>

Storm目前��是最最�E�_��的开源流式处理框�Ӟ��但是个�h认�ؓ(f��)它有两个问题�?/p>

1. Storm虽然支持多个语言�~�写spout和bolt端的代码�Q�但是它的主要技术实现是clojure�Q�这�l�玩大数据、开源的朋友带来�?ji��n)极大的不变�Q�因为大家会(x��)的语�a�不是以java和C++�{�大众语�a��Z��Q�这��L(f��ng)��话，变得不可控了(ji��n)�Q�难以深入了(ji��n)解、修改其�l�节�?/p>

2. Storm可以支持在Yarn(Hadoop 2.0)上，可以和其他开源框架共享Hadoop集群的资源，但是性能不佳�Q�这个有待Storm改善

当然无论如何�Q�Storm依然是目前开源流式处理框架的王者�?/p>

�W�二个我惌��的是JStorm�Q�这个是阉K��做的�Q�算是Storm的另一个实玎ͼ�它用的语�a�是Java.

特点�Q?/p>

1. 客户端的API与Storm基本上是一致的�Q�如果从Storm�q�移�q�来�Q�不需要修改bolt和spout的代�?/p>

2. Jstrom比Strom�E�_��Q�速度更快

3. 提供�?ji��n)一些新的特�?/p>

大家有兴��可以去玩玩�Q�项目地址https://github.com/alibaba/jstorm

�W�三个是Samza

Samza是由LinkedIn开源的一个技术，它是一个开源的分布式流处理�pȝ��Q�非常类��g��Storm。不同的是它�q�行在Hadoop之上�Q��ƈ且��用了(ji��n)自己开发的Kafka分布式消息处理系�l��?/p>

�q�是Linkin开发的一个小而美的项目，如何��呢�Q?/p>

1. 只有几千行代码，完成的功能就可以和Storm媲美�Q�当然目前还有很多的不��

2. 和Kafka�l�合紧密�Q�更方便的处理数�?/p>

3. �q�行在Yarn�?/p>

之前我做�q�的一个项目，是Kafka + Storm + ElasticSearch�Q�将来完全可以将Storm替换成Samza�Q�这��L(f��ng)��话，�q�可以利用Hadoop集群的资源，做一些存储、离�U�分析的功能。将实时处理和离�U�分析都�q�行在Hadoop上，不得不说Samza是一个伟大的��目�Q�这样可以减��项目的增长复杂度，利于�l�护�Q�还是那句话�Q�小而美的东西，更受�Ƣ迎一些�?/p>

架构�Q?/p>

Samza主要包含三层�Q?/p>

1. ��处理层 --> Kafka

2. 执行�? --> YARN

3. 处理�? --> Samza API

Samza的流处理层和执行层都是可插拔式的�Q�开发�h员可以��用其他框架来替代�Q�不局限于上述两种技术�?/p>

Samza提供�?ji��n)一个YARN ApplicationMaster�Q�和YARN job�Q�运行在集群之外�Q�下图中不同颜色代表不同的主机�?/p>

Samza客户端告诉YARN的Resouce Manager�Q�它惛_��动一个Samza job�Q?YARN RM 告诉YARN Node manager�Q�分配空间给YARN ApplicationMaster�Q�NM指定完空间后�Q�YARN container�?x��)运行Samza Task Runner�?/p>

Samza状态管�?/p>

��式处理数据对状态的��理是很隄��Q�由于数据是��动的，本��n没有状态，�q�样��需要靠历史数据来记录应用的场合�Q�Samza提供�?ji��n)一个内部的key-value数据库，它是��Z��LevelDB�Q�运行的JVM之外的，使用它来存储历史数据。这��L(f��ng)��做的好处是：(x��)

1. 减少JVM的开销

2. 使用内部存储�Q�极大提高的吞吐�?/p>

3. 减少�q�发操作

Samza处理��程.

下图是Samza官方�l�的一例子�Q�根据Member ID分组�Q�计��页面访问次数。入口消息分别来自Machine1�?�Q�出口是Machine3�Q�我们可以这��L(f��ng)��解，消息分散在不同的消息�pȝ��中（Kafka�Q�，Samza从不同的Kafka中读取topic�Q�在��topic�q�行处理后，发送到Machine3�Q�这里不做过多分解，具体可以参照官方文档�?/p>

��目地址�Q?a target="_blank" style="color: #336699; text-decoration: none;">https://github.com/apache/incubator-samza

官方文�g�Q?a target="_blank" style="color: #336699; text-decoration: none;">http://samza.incubator.apache.org/

以上�l�了(ji��n)我们无限遐想�Q�Storm是否�?x��)保持领先地位，Samza能否取而代之呢�Q�无论如何，作�ؓ(f��)开发者来��_(d��)��几千行代码，我都�q�不�?qi��ng)待去要��M��下了(ji��n)�?/p>

paulwong 2014-12-02 15:03 发表评论

Auto rebalance Storm

paulwong — Fri, 09 May 2014 15:48:00 GMT

http://stackoverflow.com/questions/15010420/storm-topology-rebalance-using-java-code

使用Nimbus获取STORM的信�?br />http://www.andys-sundaypink.com/i/retrieve-storm-cluster-statistic-from-nimbus-java-mode/

TSocket tsocket = new TSocket("localhost", 6627);
TFramedTransport tTransport = new TFramedTransport(tsocket);
TBinaryProtocol tBinaryProtocol = new TBinaryProtocol(tTransport);
Nimbus.Client client = new Nimbus.Client(tBinaryProtocol);
String topologyId = "test-1-234232567";

try {

tTransport.open();
ClusterSummary clusterSummary = client.getClusterInfo();
StormTopology stormTopology = client.getTopology(topologyId);
TopologyInfo topologyInfo = client.getTopologyInfo(topologyId);
List executorSummaries = topologyInfo.get_executors();

List topologies = clusterSummary.get_topologies();
for(ExecutorSummary executorSummary : executorSummaries){

String id = executorSummary.get_component_id();
ExecutorInfo executorInfo = executorSummary.get_executor_info();
ExecutorStats executorStats = executorSummary.get_stats();
System.out.println("executorSummary :: " + id + " emit size :: " + executorStats.get_emitted_size());
}
} catch (TTransportException e) {
e.printStackTrace();
} catch (TException e) {
e.printStackTrace();
} catch (NotAliveException e) {
e.printStackTrace();
}

paulwong 2014-05-09 23:48 发表评论

��释STORM

paulwong — Fri, 09 May 2014 14:56:00 GMT

STORM是一个消息处理引擎，可以处理源源不断的进来的消息�Q�这些消息的处理是可以按步骤的�?br />
处理的方式有各种自定义：(x��)

可自定义消息处理的步�?br />
可自定义每种�c�d��的消息需要多��个�q�程来处�?br />
每个步骤里的消息是在某个�q�程里的�U�程来做处理�?br />
可自定义每个步骤里的消息的线�E�数
可以增加和删除要处理的消息类�?

如果要处理某�U�消息了(ji��n)�Q�要怎么办呢�Q?br />

定义数据来源�l��g(SPOUT)
定义处理步骤(BOLT)
�l�合成一个消息处理流�E�框架TOPOLOGY
定义处理消息的进�E�的数量、定义每个步骤�ƈ发时可用的线�E�数
部��vTOPOLOGY

当一个TOPOLOGY被部�|�到STORM�Ӟ��STORM�?x��)查��N��|�对象的WORKER数量�Q�根据这个数量相应的启动N个JVM�Q�然后根据每个步骤配�|�的NUMTASKS生成相应个数的线�E�，然后每个步骤中配�|�的数量实例化相应个数的对象�Q�然后就启动一个线�E�不断的执行SPOUT中的nextTuple()�Ҏ(gu��)��Q�如果这个方法中有输出结果，��启动另一�U�程�Q��ƈ在此�U�程中将�q�个�l�果作�ؓ(f��)参数传到下一个对象的excue�Ҏ(gu��)��中�?br />
如果此时又有一个步骤BOLT需要执行的话，也是新取一个线�E�去执行BOLT中的�Ҏ(gu��)��启动的线�E�不�?x��)越�q�NUMTASKS的数量�?br />

paulwong 2014-05-09 22:56 发表评论

Storm performance

paulwong — Thu, 08 May 2014 01:19:00 GMT

The configuration is used to tune various aspects of the running topology. The two configurations specified here are very common:

TOPOLOGY_WORKERS (set with setNumWorkers) specifies how many processes you want allocated around the cluster to execute the topology. Each component in the topology will execute as many threads. The number of threads allocated to a given component is configured through the setBolt and setSpout methods. Those threadsexist within worker processes. Each worker process contains within it some number of threads for some number of components. For instance, you may have 300 threads specified across all your components and 50 worker processes specified in your config. Each worker process will execute 6 threads, each of which of could belong to a different component. You tune the performance of Storm topologies by tweaking the parallelism for each component and the number of worker processes those threads should run within.
TOPOLOGY_DEBUG (set with setDebug), when set to true, tells Storm to log every message every emitted by a component. This is useful in local mode when testing topologies, but you probably want to keep this turned off when running topologies on the cluster.

There's many other configurations you can set for the topology. The various configurations are detailed on the Javadoc for Config.

Common configurations

There are a variety of configurations you can set per topology. A list of all the configurations you can set can be found here. The ones prefixed with "TOPOLOGY" can be overridden on a topology-specific basis (the other ones are cluster configurations and cannot be overridden). Here are some common ones that are set for a topology:

Config.TOPOLOGY_WORKERS: This sets the number of worker processes to use to execute the topology. For example, if you set this to 25, there will be 25 Java processes across the cluster executing all the tasks. If you had a combined 150 parallelism across all components in the topology, each worker process will have 6 tasks running within it as threads.
Config.TOPOLOGY_ACKERS: This sets the number of tasks that will track tuple trees and detect when a spout tuple has been fully processed. Ackers are an integral part of Storm's reliability model and you can read more about them onGuaranteeing message processing.
Config.TOPOLOGY_MAX_SPOUT_PENDING: This sets the maximum number of spout tuples that can be pending on a single spout task at once (pending means the tuple has not been acked or failed yet). It is highly recommended you set this config to prevent queue explosion.
Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS: This is the maximum amount of time a spout tuple has to be fully completed before it is considered failed. This value defaults to 30 seconds, which is sufficient for most topologies. SeeGuaranteeing message processing for more information on how Storm's reliability model works.
Config.TOPOLOGY_SERIALIZATIONS: You can register more serializers to Storm using this config so that you can use custom types within tuples.

Reference:
http://storm.incubator.apache.org/documentation/Running-topologies-on-a-production-cluster.html

storm rebalance 命��o(h��)调整topology�q�行数及(qi��ng)问题分析

http://blog.csdn.net/jmppok/article/details/17243857

flume+kafka+storm+mysql 数据?hu��)?br />http://blog.csdn.net/jmppok/article/details/17259145

http://storm.incubator.apache.org/documentation/Tutorial.html

paulwong 2014-05-08 09:19 发表评论

安装STORM

paulwong — Sun, 04 May 2014 10:01:00 GMT

install ZeroMQ

wget http://download.zeromq.org/historic/zeromq-2.1.7.tar.gz
tar -xzf zeromq-2.1.7.tar.gz
cd zeromq-2.1.7
./configure
//在configure时可能会(x��)报缺包，安装卛_��Q�sudo apt-get install g++ uuid-dev
make
sudo make install
install JZMQ

git clone https://github.com/nathanmarz/jzmq.git
cd jzmq
./autogen.sh
./configure
make
sudo make install
下蝲�q�解压STORM
�~�辑conf/storm.yaml

storm.zookeeper.servers:
- "1.2.3.5"
- "1.2.3.6"
- "1.2.3.7"
storm.local.dir: "/opt/folder"
nimbus.host: "54.72.4.92"
supervisor.slots.ports:
- 6700
- 6701
- 6702
�~�辑/etc/profile

export JAVA_HOME=/usr/lib/jvm/java-7-oracle
export STORM_HOME=/home/ubuntu/java/storm-0.8.1
export KAFKA_HOME=/home/ubuntu/java/kafka_2.9.2-0.8.1.1
export ZOOKEEPER_HOME=/home/ubuntu/java/zookeeper-3.4.6

export PATH=$JAVA_HOME/bin:$STORM_HOME/bin:$KAFKA_HOME/bin:$ZOOKEEPER_HOME/bin:$PATH

制作启动命��o(h��): start-storm.sh
storm nimbus &
storm supervisor &
storm ui &

安装途中如果遇到问题
http://my.oschina.net/mingdongcheng/blog/43009

paulwong 2014-05-04 18:01 发表评论

STORM启动与部�|�TOPOLOGY

paulwong — Wed, 11 Sep 2013 03:00:00 GMT

启动ZOOPKEEPER
zkServer.sh start
启动NIMBUS
storm nimbus &
启动SUPERVISOR
storm supervisor &
启动UI
storm ui &
部��vTOPOLOGY
storm jar /opt/hadoop/loganalyst/storm-dependend/data/teststorm-1.0.jar teststorm.TopologyMain /opt/hadoop/loganalyst/storm-dependend/data/words.txt
删除TOPOLOGY
storm kill {toponame}
�Ȁ�z�TOPOLOGY
storm active {toponame}
不激�z�TOPOLOGY
storm deactive {toponame}
列出所有TOPOLOGY
storm list

paulwong 2013-09-11 11:00 发表评论

STORM资源

paulwong — Sun, 08 Sep 2013 11:59:00 GMT

Install Storm
http://www.jansipke.nl/installing-a-storm-cluster-on-centos-hosts/
http://www.cnblogs.com/kemaswill/archive/2012/10/24/2737833.html
http://abentotoro.blog.sohu.com/197023262.html
http://www.cnblogs.com/panfeng412/archive/2012/11/30/how-to-install-and-deploy-storm-cluster.html

使用 Twitter Storm 处理实时的大数据
http://www.ibm.com/developerworks/cn/opensource/os-twitterstorm/

Storm数据?hu��)��模型的分析及(qi��ng)讨�?br />http://www.cnblogs.com/panfeng412/archive/2012/07/29/storm-stream-model-analysis-and-discussion.html
http://www.cnblogs.com/panfeng412/tag/Storm/

storm-kafka
https://github.com/nathanmarz/storm-contrib/tree/master/storm-kafka

使用Storm实现实时大数据分析！
http://www.csdn.net/article/2012-12-24/2813117-storm-realtime-big-data-analysis

storm-deploy-aws
https://github.com/nathanmarz/storm-deploy/wiki

!!!知乎�|�站上的Twitter Storm
http://www.zhihu.com/topic/19673110

storm-elastic-search
https://github.com/hmsonline/storm-elastic-search

storm-examples
https://github.com/stormprocessor/storm-examples

kafka-aws
https://github.com/nathanmarz/kafka-deploy

Next Gen Real-time Streaming with Storm-Kafka Integration
http://blog.infochimps.com/2012/10/30/next-gen-real-time-streaming-storm-kafka-integration/

flume+kafka+storm+mysql 数据?hu��)?
http://blog.csdn.net/baiyangfu/article/details/8096088
http://blog.csdn.net/baiyangfu/article/category/1244640

Kafka学习(f��n)�W�记
http://blog.csdn.net/baiyangfu/article/details/8096084

STORM+KAFKA
https://github.com/buildlackey/cep

STORM+KETTLE
https://github.com/buildlackey/kettle-storm

paulwong 2013-09-08 19:59 发表评论

paulwong — Sun, 08 Sep 2013 11:49:00 GMT

对于一堆时��d��增长的数据，如果要统计，可以采取什么方法呢�Q?br />

�{�数据增长到一定程度的时候，跑一个统计程序进行统计。适用于实时性要求不高的场景�?br />如将数据导到HDFS�Q�再�q�行一个MAP REDUCE JOB�?br />
如果实时性要求高的，上面的方法就不行�?ji��n)。因此就带来�W�二�U�方法�?br />在数据每�ơ增长一�W�的时候，��p��行统计JOB�Q�结果放到DB或搜索引擎的INDEX中�?br />STORM��是完成�q�种工作的�?/li>

HADOOP与STORM比较

数据来源�Q�HADOOP是HDFS上某个文件夹下的可能是成TB的数据，STORM是实时新增的某一�W�数�?/li>
处理�q�程�Q�HADOOP是分MAP阶段到REDUCE阶段�Q�STORM是由用户定义处理��程�Q?br />��程中可以包含多个步骤，每个步骤可以是数据源(SPOUT)或处理逻辑(BOLT)
是否�l�束�Q�HADOOP最后是要结束的�Q�STORM是没有结束状态，到最后一步时�Q�就停在那，直到有新
数据�q�入时再从头开�?/li>
处理速度�Q�HADOOP是以处理HDFS上大量数据�ؓ(f��)目的�Q�速度慢，STORM是只要处理新增的某一�W�数据即�?br />可以做到很快�?/li>
适用场景�Q�HADOOP是在要处理一�Ҏ(gu��)��据时用的�Q�不讲究时效性，要处理就提交一个JOB�Q�STORM是要处理
某一新增数据时用的，要讲时效�?br />
与MQ�Ҏ(gu��)��Q�HADOOP没有�Ҏ(gu��)��性，STORM可以看作是有N个步骤，每个步骤处理完就向下一个MQ发送消息，
监听�q�个MQ的消费者��l�处�?br />

paulwong 2013-09-08 19:49 发表评论