一区二区三区亚洲视频,国产亚洲福利一区二区免费看,亚洲一级特黄特黄的大片

安装logstash+kibana+elasticsearch+redis搭徏集中式日志分析��^�?

Eric_jiang — Fri, 12 Jun 2015 03:30:00 GMT

本文是参考logstash官方文档实践的笔讎ͼ�搭徏环境和所需�l��g如下�Q?/p>

Redhat 5.7 64bit / CentOS 5.x
JDK 1.6.0_45
logstash 1.3.2 (内带kibana)
elasticsearch 0.90.10
redis 2.8.4

搭徏的集中式日志分析�q�_��程如下�Q?nbsp;

elasticsearch

1、下载elasticsearch�?/p>

wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.10.tar.gz

2、解压后�Q�进入bin目录。执行如下命令，让elasticsearch以前台方式启动：

./elasticsearch -f

[2014-01-16 16:21:31,825][INFO ][node                     ] [Saint Elmo] version[0.90.10], pid[32269], build[0a5781f/2014-01-10T10:18:37Z] [2014-01-16 16:21:31,826][INFO ][node                     ] [Saint Elmo] initializing ... [2014-01-16 16:21:31,836][INFO ][plugins                  ] [Saint Elmo] loaded [], sites [] [2014-01-16 16:21:35,425][INFO ][node                     ] [Saint Elmo] initialized [2014-01-16 16:21:35,425][INFO ][node                     ] [Saint Elmo] starting ... [2014-01-16 16:21:35,578][INFO ][transport                ] [Saint Elmo] bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/10.0.2.15:9300]}

Redis

1、其安装方式可以参考我的另一��文�?a style="color: #4183c4; line-height: 26px; text-decoration: none; box-sizing: border-box;">Redis�~�译安装�?/p>

2、进入其bin目录�Q�执行如下命令，使之在控制台输出debug信息�Q?/p>

./redis-server --loglevel verbose

[32470] 16 Jan 16:45:57.330 * The server is now ready to accept connections on port 6379 [32470] 16 Jan 16:45:57.330 - 0 clients connected (0 slaves), 283536 bytes in use

logstash日志生成器（shipper�Q?/h2>

1、新��Z��个配�|�文�Ӟ��shipper.conf�Q�其内容如下�Q?/p>

input {     stdin {         type => "example"     } }  output {     stdout {         codec => rubydebug     }     redis {         host => "127.0.0.1"         port => 6379         data_type => "list"         key => "logstash"     } }

2、启动shipper。执行如下命令：

java -jar logstash-1.3.2-flatjar.jar agent -f shipper.conf

�l�端�H�口��出现如下提�C�Z��息：

Using milestone 2 output plugin 'redis'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.3.2/plugin-milestones {:level=>:warn}

然后在终端窗口直接按回�R�Q�将出现如下信息�Q?/p>

{        "message" => "",       "@version" => "1",     "@timestamp" => "2014-01-16T08:15:19.400Z",           "type" => "example",           "host" => "redhat" }

�q�个json信息��发送给redis�Q?同时redis的终端窗口将出现�c�M��下面的提�C�Z��息：

[32470] 16 Jan 17:09:23.604 - Accepted 127.0.0.1:44640 [32470] 16 Jan 17:09:27.127 - DB 0: 1 keys (0 volatile) in 4 slots HT. [32470] 16 Jan 17:09:27.127 - 1 clients connected (0 slaves), 304752 bytes in use

logstash日志索引器（indexer�Q?/h2>

1、新��Z��个配�|�文�Ӟ��indexer.conf�Q�其内容如下�Q?/p>

input {   redis {     host => "127.0.0.1"     # these settings should match the output of the agent     data_type => "list"     key => "logstash"      # We use the 'json' codec here because we expect to read     # json events from redis.     codec => json   } }  output {   stdout { debug => true debug_format => "json"}    elasticsearch {     host => "127.0.0.1"   } }

2、启动日志烦引器。执行如下命令：

java -jar logstash-1.3.2-flatjar.jar agent -f indexer.conf

�l�端�H�口��出现如下提�C�Z��息：

Using milestone 2 input plugin 'redis'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.3.2/plugin-milestones {:level=>:warn} You are using a deprecated config setting "debug_format" set in stdout. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future.  If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"debug_format", :plugin=>, :level=>:warn}

索引器从Redis接收��C��息，在终端窗口会昄��c�M��如下的信息：

{"message":"","@version":"1","@timestamp":"2014-01-16T17:10:03.831+08:00","type":"example","host":"redhat"}{"message":"","@version":"1","@timestamp":"2014-01-16T17:13:20.545+08:00","type":"example","host":"redhat"}{

logstash WEB界面�Q�kibana�Q?/h2>

1、启动kibana。执行如下命令：

java -jar logstash-1.3.2-flatjar.jar web

2、打开��览器（��L��持HTML5�Q�，输入地址�Q�http://127.0.0.1:9292/index.html#/dashboard/file/logstash.json。界面效果如下：

参考资�?/span>

logstash-getting-started-centralized
访谈与书评：《LogStash�Q��日志��理更简单�?/a>

Eric_jiang 2015-06-12 11:30 发表评论

fluentd�l�合kibana、elasticsearch实时搜烦分析hadoop集群日志

Eric_jiang — Fri, 12 Jun 2015 03:05:00 GMT

Fluentd是一个开源收集事件和日志�pȝ��Q�它目前提供150+扩展插�g让你存储大数据用于日志搜索，数据分析和存储�?/p>

官方地址http://fluentd.org/ 插�g地址http://fluentd.org/plugin/

Kibana 是一个�ؓ ElasticSearch 提供日志分析�?Web ui工具�Q�可使用它对日志�q�行高效的搜索、可视化、分析等各种操作。官方地址http://www.elasticsearch.org/overview/kibana/

elasticsearch 是开源的�Q�Apache2协议�Q�，分布式的�Q�RESTful的，构徏在Apache Lucene之上的的搜烦引擎.

官方地址http://www.elasticsearch.org/overview/ 中文地址 http://es-cn.medcl.net/

具体的工作流�E�就是利用fluentd 监控�q�过滤hadoop集群的系�l�日志，��过滤后的日志内容发�l�全文搜索服务ElasticSearch, 然后用ElasticSearch�l�合Kibana �q�行自定义搜索web��面展示.

下面开始说部��v�Ҏ��和过�E�。以下安装步骤在centos 5 64位测试通过

一�? elasticsearch安装部��v

elasticsearch 官方提供了几�U�安装包�Q�适用于windows的zip压羃包，适用于unix/linux的tar.gz压羃包，适用于centos�pȝ��的rpm包和ubuntu的deb包。大家可以自己选择安装使用�?br />

因�ؓelasticsearch 需要java环境�q�行�Q�首先需要安装jdk,安装步骤��q��略了�?/span>

使用.tar.gz压羃包安装部�|�的话，先下载压�~�包

# wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.5.tar.gz

# tar zxvf elasticsearch-0.90.5.tar.gz

////////////如果是单机部�|?/p>

# cd elasticsearch-0.90.5

# elasticsearch-0.90.5/bin/elasticsearch -f

��可以启动搜索服务了�Q�查看端�?200是否打开�Q�如果打开说明启动正常�?/p>

////////////////如果是部�|�集��的话，需要进行配�|?/p>

例如�?92.168.0.1 192.168.0.2 两台服务器部�|Ԍ��两台服务器都安装好jdk�Q�下载elasticsearch 解压�~�，然后�~�辑配置文�g

//////////////////////192.168.0.1 服务器编辑文�?/p>

vi elasticsearch-0.90.5/config/elasticsearch.yml

删除cluster.name 前面注释�Q�修攚w��名�U?nbsp;

cluster.name: es_cluster

删除node.name前注�?�Q�修改节点名�U�ͼ�不修改的话，�pȝ��启动后会生成随即node名称�?/p>

node.name: "elastic_inst1"

node.master: true 讄��该节点�ؓ主节�?br />

/////////////////////////192.168.0.2 �~�辑文�g

vi elasticsearch-0.90.5/config/elasticsearch.yml

删除cluster.name 前面注释�Q�修攚w��名�U?nbsp;

cluster.name: es_cluster

删除node.name前注�?�Q�修改节点名�U�ͼ�不修改的话，�pȝ��启动后会生成随即node名称�?/p>

node.name: "elastic_inst2"

node.master: false 讄��该节点�ؓ主节�?/p>

分别启动两台服务器的服务后，�?92.168.0.2的日志中会看�?/p>

[elastic_inst2] detected_master [elastic_inst1] 日志信息。说明集��连接成功�?br />

二、安装部�|�fluentd

在需要监控分析的hadoop集群节点中安装fluentd�Q�安装步骤很��?/span>

curl -L http://toolbelt.treasure-data.com/sh/install-redhat.sh | sh

安装完成后，�~�辑配置文�g

# vim /etc/td-agent/td-agent.conf

[html] view plain copy print ?

<source>
type tail #### tail方式采集日志
path /var/log/hadoop/mapred/hadoop-mapred-tasktracker-node-128-70.log ### hadoop日志路径
pos_file /var/log/td-agent/task-access.log.pos
tag task.mapred
format /^(?<message>.+(WARN|ERROR).+)$/ #### 攉��error 或者warn 日志�?nbsp;
source>
<match task.**>
host 192.168.0.1 ##### <span style="font-family:Arial,Helvetica,sans-serif">elasticsearch 服务器地址span>
type elasticsearch
logstash_format true
flush_interval 5s
include_tag_key true
tag_key mapred
match>

启动fluentd 服务

# service td-agent start

三、安装部�|�kibana 3

kibana 3 �?span style="font-family: Helvetica, arial, freesans, clean, sans-serif;">使用html 和javascript 开发的web ui前端工具�?/span>

下蝲 wget http://download.elasticsearch.org/kibana/kibana/kibana-latest.zip

解压�~?unzip kibana-latest.zip

安装apache yum -y install httpd

cp -r kibana-latest /var/www/html

因�ؓ我将kibana3 安装在和elasticsearch同一台服务器中，所以不用修攚w��|�文�?/span>

启动apache service httpd start

打开��览�?http://ip/kibana ��可以看到kibana 界面

初次使用kibana 需要自己定义模�?/span>

Eric_jiang 2015-06-12 11:05 发表评论