亚洲va国产va天堂va久久,久久亚洲精品专区蓝色区,久久精品亚洲男人的天堂

Install Hadoop in the AWS cloud

get the Whirr tar file

wget http://www.eu.apache.org/dist/whirr/stable/whirr-0.8.2.tar.gz
untar the Whirr tar file
tar -vxf whirr-0.8.2.tar.gz
create credentials file
mkdir ~/.whirr
cp conf/credentials.sample ~/.whirr/credentials
add the following content to credentials file
# Set cloud provider connection details
PROVIDER=aws-ec2
IDENTITY=<AWS Access Key ID>
CREDENTIAL=<AWS Secret Access Key>
generate a rsa key pair
ssh-keygen -t rsa -P ''
create a hadoop.properties file and add the following content
whirr.cluster-name=whirrhadoopcluster
whirr.instance-templates=1 hadoop-jobtracker+hadoop-namenode,2 hadoop-datanode+hadoop-tasktracker
whirr.provider=aws-ec2
whirr.private-key-file=${sys:user.home}/.ssh/id_rsa
whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub
whirr.hadoop.version=1.0.2
whirr.aws-ec2-spot-price=0.08
launch hadoop
bin/whirr launch-cluster --config hadoop.properties
launch proxy
cd ~/.whirr/whirrhadoopcluster/
./hadoop-proxy.sh
add a rule to iptables
0.0.0.0/0 50030
0.0.0.0/0 50070
check the web ui in the browser
http://<aws-public-dns>:50030
add to /etc/profile
export HADOOP_CONF_DIR=~/.whirr/whirrhadoopcluster/
check if the hadoop works
hadoop fs -ls /

posted on 2013-09-08 13:45 paulwong 閱讀(406) 評論(0) 編輯收藏所屬分類: HADOOP 、AWS


只有注冊用戶登錄后才能發(fā)表評論。




網(wǎng)站導(dǎo)航: 博客園 IT新聞 Chat2DB C++博客博問管理
相關(guān)文章: HADOOP各種框架應(yīng)用領(lǐng)域編譯HADOOP源碼 Simplehbase 安裝CLOUDERA 2014年值得關(guān)注的十個(gè)Hadoop大數(shù)據(jù)創(chuàng)業(yè)公司 KMEANS PAGERANK ON HADOOP Packt celebrates International Day Against DRM, May 6th 2014 A book: Web Crawling and Data Mining with Apache Nutch 【轉(zhuǎn)載】經(jīng)典漫畫講解HDFS原理 Install Hadoop in the AWS cloud