说明
本文使用的elasticsearch logstash 都是6.1.2版本，基于centos7环境上进行测试验证。
本文测试节点的IP地址为： 192.168.5.60 。
本文不做elasticsearch logstash安装详细说明。
文中最后附下载地址链接。
一、简单安装
java环境安装
# java -version
openjdk version "1.8.0_161"
从文中最后下载链接下载下来的安装包
elasticsearch-6.1.2.rpm
logstash-6.1.2.rpm
安装
rpm -ivh elasticsearch-6.1.2.rpm
rpm -ivh logstash-6.1.2.rpm
修改 elasticsearch.yml 配置中的network.host，并关闭防火墙
# cat /etc/elasticsearch/elasticsearch.yml |grep network.host
network.host: 192.168.5.60
# systemctl stop irewalld.service
启动elasticsearch
systemctl enable elasticsearch.service
systemctl start elasticsearch.service
检测elasticsarch状态
curl '192.168.5.60:9200/_cat/health?v'
二、logstash监听本地文件
配置
# cat /etc/logstash/conf.d/log2.conf
input {
file {
path => ["/var/log/lyh/messages"]
type => "system"
start_position => "beginning"
}
}
filter {
}
output {
stdout {}
}
执行logstash
# cd /usr/share/logstash/bin/
# ./logstash -f /etc/logstash/conf.d/log2.conf --path.settings /etc/logstash
往 /var/log/lyh/messages 插入日志
echo "Jan 23 08:51:59 localhost kernel: LYH 111" >> /var/log/lyh/messages
echo "Jan 24 08:41:58 localhost systemd: Starting Session 36 of user root. " >> /var/log/lyh/messages
查看信息
可以在执行 ./logstash -f /etc/logstash/conf.d/log2.conf --path.settings /etc/logstash 的界面看到打印日志
2018-01-24T01:10:00.202Z 0.0.0.0 Jan 23 08:51:59 localhost kernel: LYH 111
....
三、logstash作为syslog-server监听syslog日志信息
配置
# cat /etc/logstash/conf.d/log3.conf
input {
tcp {
port => 514
type => syslog
}
udp {
port => 514
type => syslog
}
}
filter {
}
output {
stdout {}
}
启动logstash
# cd /usr/share/logstash/bin/
# ./logstash -f /etc/logstash/conf.d/log3.conf --path.settings /etc/logstash
模拟一条syslog的日志
# logger -T -P 514 -n 127.0.0.1 'hello world '
查看信息
可以在执行 ./logstash -f /etc/logstash/conf.d/log2.conf --path.settings /etc/logstash 的界面看到打印日志
2018-01-24T06:22:55.969Z 127.0.0.1 <5>Jan 24 14:22:55 root: hello world
四、logstash作为syslog-server监听syslog日志信息，并将日志对接到elasticsearch
配置
# cat /etc/logstash/conf.d/log4.conf
input {
tcp {
port => 514
type => syslog
}
udp {
port => 514
type => syslog
}
}
filter {
}
output {
elasticsearch {
action => "index"
hosts => "192.168.5.60:9200"
index => "lyh-test"
}
stdout {}
}
启动logstash
# cd /usr/share/logstash/bin/
# ./logstash -f /etc/logstash/conf.d/log4.conf --path.settings /etc/logstash
模拟一条syslog的日志
# logger -T -P 514 -n 127.0.0.1 'hello world '
查看信息
可以在执行 ./logstash -f /etc/logstash/conf.d/log2.conf --path.settings /etc/logstash 的界面看到打印日志
2018-01-24T06:22:55.969Z 127.0.0.1 <5>Jan 24 14:22:55 root: hello world
获取elasticsearch的索引
增加了一条lyh-test的的索引
# curl -X GET 192.168.5.60:9200/_cat/indices?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open lyh-test nWx7hdNqQOStFbEVXd8tYQ 5 1 5 0 27.7kb 27.7kb
获取该索引里面的数据
# curl -X GET -H 'Content-type: application/json' 192.168.5.60:9200/lyh-test/_search -d '{
"query": {
"match_all": {}
}
}'
# 通过size from sort进行分页排序查找
# curl -X GET -H 'Content-type: application/json' http://192.168.5.60:9200/ssp-attacklog--*/_search?size=10\&from=1\&pretty -d '{
"query": {"match_all": {}},
"sort": {
"happentime": {"order": "desc"}
}
}'
五、下载地址
logstash: https://www.elastic.co/downloads/logstash
elasticsearch: https://www.elastic.co/downloads/elasticsearch
kibana: https://www.elastic.co/downloads/kibana
更多logstash filter相关的可以参见官网
https://www.elastic.co/guide/en/logstash/current/config-examples.html
六、一些出错信息，以及解决方法
1、创建ES索引报错：FORBIDDEN/12/index read-only / allow delete (api)
1)报错 info:{"error":{"root_cause":[{"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}],"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"},"status":403}
2)解决方式：到es指定节点，执行
curl -XPUT -H "Content-Type: application/json" http://127.0.0.1:9200/_all/_settings -d '{
"index":{
"blocks.read_only_allow_delete":false
}
}'
3)说明：
_all 表示全部索引，可以指定具体出错的索引
