Elastic Stack之Redis集群使用
2024-08-27 04:29:59
Elastic Stack之Redis集群使用
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
本篇博客数据流走向：FileBeat ===》Redis ===》logstash ===> elasticsearch。
一.安装Redis服务器
1>.安装repo源
[root@node105 ~]#
[root@node105 ~]# yum -y install epel-release
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.cn99.com
* extras: mirror.lzu.edu.cn
* updates: mirror.lzu.edu.cn
Resolving Dependencies
--> Running transaction check
---> Package epel-release.noarch :- will be installed
--> Finished Dependency Resolution Dependencies Resolved ===================================================================================================================================================================================================
Package Arch Version Repository Size
===================================================================================================================================================================================================
Installing:
epel-release noarch - extras k Transaction Summary
===================================================================================================================================================================================================
Install Package Total download size: k
Installed size: k
Downloading packages:
epel-release--.noarch.rpm | kB ::
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
Installing : epel-release--.noarch /
Verifying : epel-release--.noarch / Installed:
epel-release.noarch :- Complete!
[root@node105 ~]#
[root@node105 ~]# yum -y install epel-release
2>.安装redis服务器
[root@node105 ~]#
[root@node105 ~]# yum -y install redis
Loaded plugins: fastestmirror
epel/x86_64/metalink | 7.9 kB ::
epel | 4.7 kB ::
(/): epel/x86_64/group_gz | kB ::
(/): epel/x86_64/updateinfo | kB ::
(/): epel/x86_64/primary_db | 6.6 MB ::
Loading mirror speeds from cached hostfile
* base: mirrors.cn99.com
* epel: mirrors.yun-idc.com
* extras: mirror.lzu.edu.cn
* updates: mirror.lzu.edu.cn
Resolving Dependencies
--> Running transaction check
---> Package redis.x86_64 :3.2.-.el7 will be installed
--> Processing Dependency: libjemalloc.so.()(64bit) for package: redis-3.2.-.el7.x86_64
--> Running transaction check
---> Package jemalloc.x86_64 :3.6.-.el7 will be installed
--> Finished Dependency Resolution Dependencies Resolved ===================================================================================================================================================================================================
Package Arch Version Repository Size
===================================================================================================================================================================================================
Installing:
redis x86_64 3.2.-.el7 epel k
Installing for dependencies:
jemalloc x86_64 3.6.-.el7 epel k Transaction Summary
===================================================================================================================================================================================================
Install Package (+ Dependent package) Total download size: k
Installed size: 1.7 M
Downloading packages:
warning: /var/cache/yum/x86_64//epel/packages/jemalloc-3.6.-.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY ] 0.0 B/s | B --:--:-- ETA
Public key for jemalloc-3.6.-.el7.x86_64.rpm is not installed
(/): jemalloc-3.6.-.el7.x86_64.rpm | kB ::
(/): redis-3.2.-.el7.x86_64.rpm | kB ::
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total kB/s | kB ::
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
Importing GPG key 0x352C64E5:
Userid : "Fedora EPEL (7) <epel@fedoraproject.org>"
Fingerprint: 91e9 7d7c 4a5e 96f1 7f3e 888f 6a2f aea2 352c 64e5
Package : epel-release--.noarch (@extras)
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : jemalloc-3.6.-.el7.x86_64 /
Installing : redis-3.2.-.el7.x86_64 /
Verifying : redis-3.2.-.el7.x86_64 /
Verifying : jemalloc-3.6.-.el7.x86_64 / Installed:
redis.x86_64 :3.2.-.el7 Dependency Installed:
jemalloc.x86_64 :3.6.-.el7 Complete!
[root@node105 ~]#
[root@node105 ~]# yum -y install redis
3>.修改Redis服务器的默认配置
[root@node105 ~]#
[root@node105 ~]# grep bind /etc/redis.conf | grep -v ^# #指定绑定的ip地址为服务器的所有网卡！
bind 0.0.0.0
[root@node105 ~]#
[root@node105 ~]# grep requirepass /etc/redis.conf | grep -v ^# #开启认证
requirepass yinzhengjie.org.cn
[root@node105 ~]#
[root@node105 ~]#
4>.启动redis服务器
[root@node105 ~]#
[root@node105 ~]# systemctl enable redis
Created symlink from /etc/systemd/system/multi-user.target.wants/redis.service to /usr/lib/systemd/system/redis.service.
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# systemctl start redis
[root@node105 ~]#
[root@node105 ~]# systemctl status redis
● redis.service - Redis persistent key-value database
Loaded: loaded (/usr/lib/systemd/system/redis.service; disabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/redis.service.d
└─limit.conf
Active: active (running) since Mon -- :: CST; 4s ago
Main PID: (redis-server)
CGroup: /system.slice/redis.service
└─ /usr/bin/redis-server 0.0.0.0: Mar :: node105.yinzhengjie.org.cn systemd[]: Starting Redis persistent key-value database...
Mar :: node105.yinzhengjie.org.cn systemd[]: Started Redis persistent key-value database.
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# ss -tnl | grep
LISTEN *: *:*
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# systemctl start redis
二.使用FileBeat结合redis收集日志文件案例
1>.安装FileBeat软件
详情请参考： Elastic Stack之FileBeat使用实战。
2>.修改FileBeat的配置文件并启动
[root@node105 ~]#
[root@node105 ~]# cat /etc/filebeat/filebeat.yml | egrep -v "^#|^$| #"
filebeat.prospectors:
- input_type: log
paths:
- /var/log/httpd/access_log output.redis:
enabled: true
hosts: ["node105.yinzhengjie.org.cn:6379"]
key: filebeat
password: yinzhengjie.org.cn
db:
datatype: list
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# cat /etc/filebeat/filebeat.yml | egrep -v "^#|^$| #" #配置我们可以从/etc/filebeat/filebeat.full.yml将支持的输出配置拷贝过去！注意配置文件的属性之间应该有一个空格！
3>.查看redis服务器
[root@node105 ~]#
[root@node105 ~]# redis-cli -a yinzhengjie.org.cn
127.0.0.1:>
127.0.0.1:> KEYS *
) "filebeat"
127.0.0.1:>
127.0.0.1:> LINDEX filebeat
"{\"@timestamp\":\"2019-03-11T15:34:37.114Z\",\"beat\":{\"hostname\":\"node105.yinzhengjie.org.cn\",\"name\":\"node105.yinzhengjie.org.cn\",\"version\":\"5.6.12\"},\"input_type\":\"log\",\"message\":\"80.129.1.1 - - [11/Mar/2019:23:05:48 +0800] \\\"GET /test42.html HTTP/1.1\\\" 200 8 \\\"-\\\" \\\"curl/7.29.0\\\"\",\"offset\":134141,\"source\":\"/var/log/httpd/access_log\",\"type\":\"log\"}"
127.0.0.1:>
127.0.0.1:>
127.0.0.1:>
127.0.0.1:> LLEN filebeat
(integer)
127.0.0.1:> [root@node105 ~]# redis-cli -a yinzhengjie.org.cn
[root@node105 ~]# redis-cli -a yinzhengjie.org.cn
4>.配置logstash的redis输入插件（参考链接：https://www.elastic.co/guide/en/logstash/current/plugins-inputs-redis.html）
[root@node105 ~]#
[root@node105 ~]# cat /etc/logstash/conf.d/redis-filter-elasticsearch.conf
input {
redis {
host => "node105.yinzhengjie.org.cn"
port =>
password => "yinzhengjie.org.cn"
db =>
key => "filebeat"
data_type => "list"
}
} filter {
grok {
match => { "message" => "%{HTTPD_COMBINEDLOG}" }
remove_field => ["message","beat"]
}
date {
match => ["timestamp","dd/MMM/YYYY:H:m:s Z"]
remove_field => "timestamp"
}
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/maxmind/GeoLite2-City.mmdb"
}
} output {
elasticsearch {
hosts => ["http://node101.yinzhengjie.org.cn:9200/","http://node102.yinzhengjie.org.cn:9200/","http://node103.yinzhengjie.org.cn:9200/"]
index => "logstash-%{+YYYY.MM.dd}"
document_type => "httpd_access_logs"
}
} [root@node105 ~]#
[root@node105 ~]# cat /etc/logstash/conf.d/redis-filter-elasticsearch.conf
[root@node105 ~]#
[root@node105 ~]# logstash -f /etc/logstash/conf.d/redis-filter-elasticsearch.conf -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
Configuration OK
[root@node105 ~]#
[root@node105 ~]# logstash -f /etc/logstash/conf.d/redis-filter-elasticsearch.conf -t
5>.启动logstash并查看redis服务器
[root@node103 ~]# while true; do curl -H "X-Forwarded-For:$[$RANDOM%223+1].$[$RANDOM%255].1.1" http://node105.yinzhengjie.org.cn/test$[$RANDOM%60+1].html;sleep 1;done
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test58.html was not found on this server.</p>
</body></html>
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test54.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test53.html was not found on this server.</p>
</body></html>
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test59.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test55.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test54.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test53.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test53.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test51.html was not found on this server.</p>
</body></html>
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test58.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test58.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test53.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test55.html was not found on this server.</p>
</body></html>
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test60.html was not found on this server.</p>
</body></html>
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test51.html was not found on this server.</p>
</body></html>
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test55.html was not found on this server.</p>
</body></html>
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test56.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test53.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test51.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test60.html was not found on this server.</p>
</body></html>
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test57.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test55.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test59.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test53.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test57.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test55.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test54.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test57.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test55.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test52.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test59.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test55.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test57.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test52.html was not found on this server.</p>
</body></html>
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test58.html was not found on this server.</p>
</body></html>
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test56.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test55.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test52.html was not found on this server.</p>
</body></html>
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test52.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test59.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test53.html was not found on this server.</p>
</body></html>
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test55.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test51.html was not found on this server.</p>
</body></html>
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test58.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test54.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test52.html was not found on this server.</p>
</body></html>
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test56.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test60.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test52.html was not found on this server.</p>
</body></html>
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test54.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test57.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test51.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test56.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
^C
[root@node103 ~]#
[root@node103 ~]# while true; do curl -H "X-Forwarded-For:$[$RANDOM%223+1].$[$RANDOM%255].1.1" http://node105.yinzhengjie.org.cn/test$[$RANDOM%60+1].html;sleep 1;done
[root@node105 ~]# redis-cli -a yinzhengjie.org.cn
127.0.0.1:>
127.0.0.1:> LLEN filebeat
(integer)
127.0.0.1:>
[root@node105 ~]# redis-cli -a yinzhengjie.org.cn #启动logstash之前
[root@node105 ~]# logstash -f /etc/logstash/conf.d/redis-filter-elasticsearch.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[root@node105 ~]# logstash -f /etc/logstash/conf.d/redis-filter-elasticsearch.conf #启动logstash
[root@node105 ~]# redis-cli -a yinzhengjie.org.cn
127.0.0.1:>
127.0.0.1:>
127.0.0.1:> LLEN filebeat
(integer)
127.0.0.1:>
[root@node105 ~]# redis-cli -a yinzhengjie.org.cn #启动logstash之后
[root@node101 ~]# curl -X GET http://node101.yinzhengjie.org.cn:9200/logstash-*/_search?q=response:404 | jq .
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
428k --:--:-- --:--:-- --:--:-- 441k
{
"took": ,
"timed_out": false,
"_shards": {
"total": ,
"successful": ,
"skipped": ,
"failed":
},
"hits": {
"total": ,
"max_score": 1.8790942,
"hits": [
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltN80lXxXllWpXYAEi",
"_score": 1.8790942,
"_source": {
"request": "/test55.html",
"geoip": {
"ip": "100.242.1.1",
"latitude": 37.751,
"country_name": "United States",
"country_code2": "US",
"continent_code": "NA",
"country_code3": "US",
"location": {
"lon": -97.822,
"lat": 37.751
},
"longitude": -97.822
},
"offset": ,
"auth": "-",
"ident": "-",
"input_type": "log",
"verb": "GET",
"source": "/var/log/httpd/access_log",
"type": "log",
"tags": [
"beats_input_codec_plain_applied"
],
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:02:33.000Z",
"response": "",
"bytes": "",
"clientip": "100.242.1.1",
"@version": "",
"beat": {
"name": "node105.yinzhengjie.org.cn",
"hostname": "node105.yinzhengjie.org.cn",
"version": "5.6.12"
},
"host": "node105.yinzhengjie.org.cn",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltN80lXxXllWpXYAEq",
"_score": 1.8790942,
"_source": {
"request": "/test52.html",
"geoip": {
"timezone": "Europe/Paris",
"ip": "82.234.1.1",
"latitude": 48.6942,
"continent_code": "EU",
"city_name": "Brunoy",
"country_name": "France",
"country_code2": "FR",
"country_code3": "FR",
"region_name": "Essonne",
"location": {
"lon": 2.4922,
"lat": 48.6942
},
"postal_code": "",
"region_code": "",
"longitude": 2.4922
},
"offset": ,
"auth": "-",
"ident": "-",
"input_type": "log",
"verb": "GET",
"source": "/var/log/httpd/access_log",
"type": "log",
"tags": [
"beats_input_codec_plain_applied"
],
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:02:41.000Z",
"response": "",
"bytes": "",
"clientip": "82.234.1.1",
"@version": "",
"beat": {
"name": "node105.yinzhengjie.org.cn",
"hostname": "node105.yinzhengjie.org.cn",
"version": "5.6.12"
},
"host": "node105.yinzhengjie.org.cn",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltN80lXxXllWpXYAEt",
"_score": 1.8790942,
"_source": {
"request": "/test52.html",
"geoip": {
"timezone": "Africa/Accra",
"ip": "45.209.1.1",
"latitude": ,
"country_name": "Ghana",
"country_code2": "GH",
"continent_code": "AF",
"country_code3": "GH",
"location": {
"lon": -,
"lat":
},
"longitude": -
},
"offset": ,
"auth": "-",
"ident": "-",
"input_type": "log",
"verb": "GET",
"source": "/var/log/httpd/access_log",
"type": "log",
"tags": [
"beats_input_codec_plain_applied"
],
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:02:44.000Z",
"response": "",
"bytes": "",
"clientip": "45.209.1.1",
"@version": "",
"beat": {
"name": "node105.yinzhengjie.org.cn",
"hostname": "node105.yinzhengjie.org.cn",
"version": "5.6.12"
},
"host": "node105.yinzhengjie.org.cn",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltN80lXxXllWpXYAE9",
"_score": 1.8790942,
"_source": {
"request": "/test58.html",
"geoip": {
"timezone": "Asia/Tokyo",
"ip": "111.89.1.1",
"latitude": 34.9833,
"continent_code": "AS",
"city_name": "Shobara",
"country_name": "Japan",
"country_code2": "JP",
"country_code3": "JP",
"region_name": "Hiroshima",
"location": {
"lon": 132.9833,
"lat": 34.9833
},
"postal_code": "727-0321",
"region_code": "",
"longitude": 132.9833
},
"offset": ,
"auth": "-",
"ident": "-",
"input_type": "log",
"verb": "GET",
"source": "/var/log/httpd/access_log",
"type": "log",
"tags": [
"beats_input_codec_plain_applied"
],
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:03:01.000Z",
"response": "",
"bytes": "",
"clientip": "111.89.1.1",
"@version": "",
"beat": {
"name": "node105.yinzhengjie.org.cn",
"hostname": "node105.yinzhengjie.org.cn",
"version": "5.6.12"
},
"host": "node105.yinzhengjie.org.cn",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltN80lXxXllWpXYAFH",
"_score": 1.8790942,
"_source": {
"request": "/test51.html",
"geoip": {
"timezone": "Europe/Madrid",
"ip": "83.47.1.1",
"latitude": 36.54,
"continent_code": "EU",
"city_name": "Fuengirola",
"country_name": "Spain",
"country_code2": "ES",
"country_code3": "ES",
"region_name": "Malaga",
"location": {
"lon": -4.6247,
"lat": 36.54
},
"postal_code": "",
"region_code": "MA",
"longitude": -4.6247
},
"offset": ,
"auth": "-",
"ident": "-",
"input_type": "log",
"verb": "GET",
"source": "/var/log/httpd/access_log",
"type": "log",
"tags": [
"beats_input_codec_plain_applied"
],
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:03:11.000Z",
"response": "",
"bytes": "",
"clientip": "83.47.1.1",
"@version": "",
"beat": {
"name": "node105.yinzhengjie.org.cn",
"hostname": "node105.yinzhengjie.org.cn",
"version": "5.6.12"
},
"host": "node105.yinzhengjie.org.cn",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltN80lXxXllWpXYAFY",
"_score": 1.8790942,
"_source": {
"request": "/test53.html",
"geoip": {
"timezone": "Europe/Rome",
"ip": "94.164.1.1",
"latitude": 42.5245,
"continent_code": "EU",
"city_name": "Piansano",
"country_name": "Italy",
"country_code2": "IT",
"country_code3": "IT",
"region_name": "Provincia di Viterbo",
"location": {
"lon": 11.8298,
"lat": 42.5245
},
"postal_code": "",
"region_code": "VT",
"longitude": 11.8298
},
"offset": ,
"auth": "-",
"ident": "-",
"input_type": "log",
"verb": "GET",
"source": "/var/log/httpd/access_log",
"type": "log",
"tags": [
"beats_input_codec_plain_applied"
],
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:03:28.000Z",
"response": "",
"bytes": "",
"clientip": "94.164.1.1",
"@version": "",
"beat": {
"name": "node105.yinzhengjie.org.cn",
"hostname": "node105.yinzhengjie.org.cn",
"version": "5.6.12"
},
"host": "node105.yinzhengjie.org.cn",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltN80lXxXllWpXYAFd",
"_score": 1.8790942,
"_source": {
"request": "/test55.html",
"geoip": {
"timezone": "Africa/Tunis",
"ip": "154.106.1.1",
"latitude": ,
"country_name": "Tunisia",
"country_code2": "TN",
"continent_code": "AF",
"country_code3": "TN",
"location": {
"lon": ,
"lat":
},
"longitude":
},
"offset": ,
"auth": "-",
"ident": "-",
"input_type": "log",
"verb": "GET",
"source": "/var/log/httpd/access_log",
"type": "log",
"tags": [
"beats_input_codec_plain_applied"
],
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:03:33.000Z",
"response": "",
"bytes": "",
"clientip": "154.106.1.1",
"@version": "",
"beat": {
"name": "node105.yinzhengjie.org.cn",
"hostname": "node105.yinzhengjie.org.cn",
"version": "5.6.12"
},
"host": "node105.yinzhengjie.org.cn",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltN80lXxXllWpXYAFq",
"_score": 1.8790942,
"_source": {
"request": "/test56.html",
"geoip": {
"ip": "33.65.1.1",
"latitude": 37.751,
"country_name": "United States",
"country_code2": "US",
"continent_code": "NA",
"country_code3": "US",
"location": {
"lon": -97.822,
"lat": 37.751
},
"longitude": -97.822
},
"offset": ,
"auth": "-",
"ident": "-",
"input_type": "log",
"verb": "GET",
"source": "/var/log/httpd/access_log",
"type": "log",
"tags": [
"beats_input_codec_plain_applied"
],
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:03:46.000Z",
"response": "",
"bytes": "",
"clientip": "33.65.1.1",
"@version": "",
"beat": {
"name": "node105.yinzhengjie.org.cn",
"hostname": "node105.yinzhengjie.org.cn",
"version": "5.6.12"
},
"host": "node105.yinzhengjie.org.cn",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltN85S3WCT5NaOiwHV",
"_score": 1.8790942,
"_source": {
"request": "/test57.html",
"geoip": {
"timezone": "America/Bogota",
"ip": "179.19.1.1",
"latitude": 4.5981,
"country_name": "Colombia",
"country_code2": "CO",
"continent_code": "SA",
"country_code3": "CO",
"location": {
"lon": -74.0758,
"lat": 4.5981
},
"longitude": -74.0758
},
"offset": ,
"auth": "-",
"ident": "-",
"input_type": "log",
"verb": "GET",
"source": "/var/log/httpd/access_log",
"type": "log",
"tags": [
"beats_input_codec_plain_applied"
],
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:03:54.000Z",
"response": "",
"bytes": "",
"clientip": "179.19.1.1",
"@version": "",
"beat": {
"name": "node105.yinzhengjie.org.cn",
"hostname": "node105.yinzhengjie.org.cn",
"version": "5.6.12"
},
"host": "node105.yinzhengjie.org.cn",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltN85T3WCT5NaOiwHc",
"_score": 1.8790942,
"_source": {
"request": "/test51.html",
"geoip": {
"city_name": "Central",
"timezone": "Asia/Hong_Kong",
"ip": "13.94.1.1",
"latitude": 22.2909,
"country_name": "Hong Kong",
"country_code2": "HK",
"continent_code": "AS",
"country_code3": "HK",
"region_name": "Central and Western District",
"location": {
"lon": 114.15,
"lat": 22.2909
},
"region_code": "HCW",
"longitude": 114.15
},
"offset": ,
"auth": "-",
"ident": "-",
"input_type": "log",
"verb": "GET",
"source": "/var/log/httpd/access_log",
"type": "log",
"tags": [
"beats_input_codec_plain_applied"
],
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:04:01.000Z",
"response": "",
"bytes": "",
"clientip": "13.94.1.1",
"@version": "",
"beat": {
"name": "node105.yinzhengjie.org.cn",
"hostname": "node105.yinzhengjie.org.cn",
"version": "5.6.12"
},
"host": "node105.yinzhengjie.org.cn",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
}
]
}
}
[root@node101 ~]#
[root@node101 ~]# curl -X GET http://node101.yinzhengjie.org.cn:9200/logstash-*/_search?q=response:404 | jq . #查询数据
三.使用一个FileBeat加载多种不同的日志案例
1>.编写FileBeat的配置文件
[root@node105 ~]#
[root@node105 ~]# cat /etc/filebeat/filebeat.yml | egrep -v "^#|^$| #"
filebeat.prospectors:
- input_type: log
paths:
- /var/log/httpd/access_log
fields:
log_type: access
- paths:
- /var/log/httpd/error_log
fields:
log_type: errors
output.redis:
enabled: true
hosts: ["node105.yinzhengjie.org.cn:6379"]
key: filebeat
password: yinzhengjie.org.cn
db:
datatype: list
logging.level: debug
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# cat /etc/filebeat/filebeat.yml | egrep -v "^#|^$| #"
2>.编写logstash的配置文件
[root@node105 ~]#
[root@node105 ~]# cat /etc/logstash/conf.d/redis-condition-elasticsearch.conf
input {
redis {
host => "node105.yinzhengjie.org.cn"
port =>
password => "yinzhengjie.org.cn"
db =>
key => "filebeat"
data_type => "list"
}
} filter {
if [fields][log_type] == "access" {
grok {
match => { "message" => "%{HTTPD_COMBINEDLOG}" }
remove_field => ["message","beat"]
}
date {
match => ["timestamp","dd/MMM/YYYY:H:m:s Z"]
remove_field => "timestamp"
}
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/maxmind/GeoLite2-City.mmdb"
}
}
} output {
if [fields][log_type] == "access" {
elasticsearch {
hosts => ["http://node101.yinzhengjie.org.cn:9200/","http://node102.yinzhengjie.org.cn:9200/","http://node103.yinzhengjie.org.cn:9200/"]
index => "logstash-%{+YYYY.MM.dd}"
document_type => "httpd_access_logs"
}
} else {
elasticsearch {
hosts => ["http://node101.yinzhengjie.org.cn:9200/","http://node102.yinzhengjie.org.cn:9200/","http://node103.yinzhengjie.org.cn:9200/"]
index => "logstash-%{+YYYY.MM.dd}"
document_type => "httpd_error_logs"
}
}
} [root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# cat /etc/logstash/conf.d/redis-condition-elasticsearch.conf
[root@node105 ~]#
[root@node105 ~]# logstash -f /etc/logstash/conf.d/redis-condition-elasticsearch.conf -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
Configuration OK
[root@node105 ~]#
[root@node105 ~]# logstash -f /etc/logstash/conf.d/redis-condition-elasticsearch.conf -t
3>.启动logstash服务
[root@node105 ~]#
[root@node105 ~]# mkdir -pv /yinzhengjie/logstash/conf.d
mkdir: created directory ‘/yinzhengjie/logstash’
mkdir: created directory ‘/yinzhengjie/logstash/conf.d’
[root@node105 ~]#
[root@node105 ~]# mv /etc/logstash/conf.d/* /yinzhengjie/logstash/conf.d/
[root@node105 ~]#
[root@node105 ~]# cp /yinzhengjie/logstash/conf.d/redis-condition-elasticsearch.conf /etc/logstash/conf.d/
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# ll /etc/logstash/conf.d/
total 4
-rw-r--r--. 1 root root 1112 Mar 12 21:45 redis-condition-elasticsearch.conf
[root@node105 ~]#
[root@node105 ~]# cp /yinzhengjie/logstash/conf.d/redis-condition-elasticsearch.conf /etc/logstash/conf.d/ #我们将要用的配置文件存放到默认的目录中
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# systemctl start logstash
[root@node105 ~]#
[root@node105 ~]# systemctl status logstash
● logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: disabled)
Active: active (running) since Tue -- :: CST; 5s ago
Main PID: (java)
CGroup: /system.slice/logstash.service
└─ /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction= -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-... Mar :: node105.yinzhengjie.org.cn systemd[]: Started logstash.
Mar :: node105.yinzhengjie.org.cn systemd[]: Starting logstash...
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# systemctl start logstash #启动logstash后，他会自动去默认的配置文件加载相应的配置文件！
[root@node105 ~]#
[root@node105 ~]# ps -ef | grep logstash | grep -v grep
logstash : ? :: /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction= -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF- -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.settings /etc/logstash
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# netstat -untalp | grep
tcp6 127.0.0.1: :::* LISTEN /java
tcp6 172.30.1.105: 172.30.1.101: ESTABLISHED /java
tcp6 172.30.1.105: 172.30.1.102: ESTABLISHED /java
tcp6 172.30.1.105: 172.30.1.103: ESTABLISHED /java
tcp6 172.30.1.105: 172.30.1.105: ESTABLISHED /java
tcp6 172.30.1.105: 172.30.1.103: ESTABLISHED /java
tcp6 172.30.1.105: 172.30.1.102: ESTABLISHED /java
tcp6 172.30.1.105: 172.30.1.101: ESTABLISHED /java
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# ps -ef | grep logstash | grep -v grep #检查logstash的进程
最新文章
- 使用Maven+Nexus+Jenkins+Svn+Tomcat+Sonar搭建持续集成环境（二）
- adobe air类app 接入腾讯开放平台移动游戏使用带tencent包名前缀的问题
- about_Execution_Policies
- albert1017 Linux下压缩某个文件夹（文件夹打包）
- 手机app测试之我见
- OSGI.NET 插件无法启动之情景一
- 【Weblogic】--Weblogic的部署方式和缓存
- codeforces 305E Playing with String
- pod update或者pod install很慢
- Objective-C浅拷贝和深拷贝
- SQL声明发育异常导致项目错误
- find查找命令
- 201521123060 《Java程序设计》第10周学习总结
- 02.将SDK获取到的ECS主机信息入库
- Unity实现c#热更新方案探究（二）
- Python 遇到的坑
- hibernate延迟加载org.hibernate.LazyInitializationException: failed to lazily initialize a collection of role: com.javakc.hibernate.onetomany.entity.DeptEntity.emp, could not initialize proxy - no Session
- 【转】gcc 编译使用动态链接库和静态链接库
- JAVA追加写入文本文件
- Linux跨服务器发送文件