Elasticsearch日志收集

Install pip if necessary

curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"

python get-pip.py

Install Curator for Elasticsearch

Elasticsearch Curator helps you curate, or manage, your Elasticsearch indices and snapshots by:

Obtaining the full list of indices (or snapshots) from the cluster, as the actionable list
Iterate through a list of user-defined filters to progressively remove indices (or snapshots) from this actionable list as needed.
Perform various actions on the items which remain in the actionable list.

pip install elasticsearch-curator

pip install click==6.7

Configure curator

mkdir -p /var/log/elastic

touch /var/log/elastic/curator.log

mkdir ~/.curator

vi ~/.curator/curator.yml

curator.yml

# Remember, leave a key empty if there is no value. None will be a string,

## not a Python "NoneType"

client:

hosts: [Elasticsearch Server IP]

port: 9200

url_prefix:

use_ssl: False

certificate:

client_cert:

client_key:

ssl_no_validate: False

http_auth:

timeout: 30

master_only: False

logging:

loglevel: INFO

logfile: /var/log/elastic/curator.log

logformat: default

blacklist: ['elasticsearch', 'urllib3']

Have a test, now you can get the indices list
curator_cli show_indices

Create repository

Configure elasticseach.yml default in /etc/elasticsearch/elasticsearch.yml

elasticsearch.yml

path.repo: /u01/elasticsearch/backup

http.max_header_size: 16kb

Restart elasticsearch service (service elasticsearch restart) to make the configurations work.

Create repository elasticsearch. Ensure location points to a valid path which is configured in path.repo, accesable from all nodes.

curl -XPUT http://localhost:9200/_snapshot/es_backup -H "Content-Type: application/json" -d @repository.json

repository.json

{

"type": "fs",

"settings": {

"compress": true,

"location": "/u01/elasticsearch/backup"

}

Have a test

curl -XGET 'localhost:9200/_snapshot/_all?pretty=true'

Create curator yaml action files

daily_backup.yml

Customize the snapshot name in name option
action 1: backup all indices before today to repository elasticsearch with specified snapshot name
action 2: delete indices older than 185 days

daily_backup.yml

---

actions:

1:

action: snapshot

description: >-

Snapshot selected all indices to repository 'elasticsearch' with the snapshot name

options:

repository: es_backup

name: '<c4cert-{now/d-1d}>'

wait_for_completion: True

max_wait: 4800

wait_interval: 30

filters:

- filtertype: age

source: name

direction: older

unit: days

unit_count: 1

timestring: "%Y.%m.%d"

2:

action: delete_indices

description: >-

Delete indices which is older than 185 days

filters:

- filtertype: age

source: name

direction: older

unit: days

unit_count: 185

timestring: "%Y.%m.%d"

del_snapshot.yml
action 1: Delete snapshots from repository elasticsearch which is older than 185 days

del_snapshot.yml

---

actions:

1:

action: delete_snapshots

description: >-

Delete snapshots from repository which is older than 185 days

options:

repository: es_backup

retry_interval: 120

retry_count: 3

filters:

- filtertype: age

source: creation_date

direction: older

unit: days

unit_count: 185

restore.yml
action 1: Restore all indices in the most recent snapshot with state SUCCESS.

restore.yml

---

actions:

1:

action: restore

description: >-

Restore all indices in the most recent snapshot with state SUCCESS. Wait

for the restore to complete before continuing. Do not skip the repository

filesystem access check. Use the other options to define the index/shard

settings for the restore.

options:

repository: es_backup

# If name is blank, the most recent snapshot by age will be selected

name:

# If indices is blank, all indices in the snapshot will be restored

indices:

wait_for_completion: True

max_wait: 3600

wait_interval: 10

filters:

- filtertype: state

state: SUCCESS

Note: use --dry-run option to verify your action without any change. Find the dry run results in log path.
Curator --dry-run daily_backup.yml

Shell script and crontab

run.sh

#!/bin/sh

curator /u01/curator/del_snapshot.yml

curator /u01/curator/daily_backup.yml

crontab -e

Here configured the job run on every 3 AM

crontab

0 3 * * * /bin/sh /u01/curator/run.sh

Restore

Curator restore.yml

Tested OK in CERT env.

Some useful API

# get all repositories

curl -XGET 'localhost:9200/_snapshot/_all?pretty=true'

# delete repository

curl -XDELETE 'localhost:9200/_snapshot/es-snapshot?pretty=true'

# show snapshots

curator_cli show_snapshots --repository es_backup

# show indices

curator_cli show_indices

巴特西

Elasticsearch日志收集

最新文章

热门文章