一、Elasticsearch1.1 集群部署server1本地rpm方式安装Elasticsearch 7.6.1修改系统资源软硬限制ES需要大量打开文件、创建线程系统默认限制过低运行一段时间会崩溃。[rootserver1 ~]# rpm -ivh elasticsearch-7.6.1-x86_64.rpm[rootserver1 ~]# cd /etc/elasticsearch/修改ES核心配置文件义集群名称、节点名称、集群节点地址、开放外部访问实现多台机器组建集群。[rootserver1 elasticsearch]# vim elasticsearch.yml修改系统资源软硬限制ES需要大量打开文件、创建线程系统默认限制过低运行一段时间会崩溃。[rootserver1 elasticsearch]# vim /etc/security/limits.conf修改systemd服务配置通过systemd启动的程序会忽略limits.conf必须在服务文件内单独声明资源上限。[rootserver1 elasticsearch]# vim /usr/lib/systemd/system/elasticsearch.service重载服务[rootserver1 elasticsearch]# systemctl daemon-reload临时关闭swap交换分区ES官方禁用swap内存和磁盘交换会极大降低检索性能容易引发集群卡顿、节点脱离集群。[rootserver1 elasticsearch]# swapoff -a将下方图片圈出来的一行注释掉永久关闭 swap 分区[rootserver1 elasticsearch]# vim /etc/fstab启动ES并设置开机自启[rootserver1 elasticsearch]# systemctl daemon-reload[rootserver1 elasticsearch]# systemctl enable --now elasticsearch查看是否配置成功免密登录将改过的配置文件拷贝给server2和server3记得修改node.name为本机名称如果注释掉就不用管了否则很有可能开启不了ssh-keygenssh-copy-id server2ssh-copy-id server3[rootserver1 elasticsearch]# scp elasticsearch.yml server2:/etc/elasticsearch/[rootserver1 elasticsearch]# scp /etc/security/limits.conf server2:/etc/security/[rootserver1 elasticsearch]# scp /usr/lib/systemd/system/elasticsearch.serviceserver2和server3上的操作和server1操作一样[rootserver2 ~]# swapoff -a[rootserver2 ~]# vim /etc/fstab[rootserver2 ~]# systemctl daemon-reload[rootserver2 ~]# systemctl start elasticsearch[rootserver2 ~]# systemctl enable --now elasticsearch1.2 Cerebro部署CerebroES集群Web可视化管理工具图形化查看节点、分片、索引状态替代大量curl命令。新打开一个虚拟机server4不要在ES集群上安装下载很有可能因为内存不够而崩溃。安装yum工具集提供yum-config-manager命令用来添加第三方软件源。加阿里云Docker官方yum源完整安装Docker引擎、客户端、容器运行时、compose插件yum install -y yum-utilsyum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repoyum install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin后台运行cerebro容器systemctl enable --now dockerdocker run -d --name cerebro -p 9000:9000 lmenezes/cerebro浏览器访问填写集群内任一节点的IP地址此时master为server11.3 集群节点角色划分修改配置文件/etc/elasticsearch/elasticsearch.ymlserver1server2server3重新配置后重启服务server2为master会自动选举master故意down掉server2的elasticsearch会自动选举server1为master启动服务server2自动加入集群此时为从二、logstash——数据采集重新开了个虚拟机server52.1 部署安装JDK11Logstash7.6.1 运行依赖Java环境必须预先部署JDK。Logstash负责采集→过滤→输出日志命令行临时启动一条简易管道验证Logstash能否正常启动测试基础输入输出流程。[rootserver5 ~]# yum install -y jdk-11.0.15_linux-x64_bin.rpm[rootserver5 ~]# yum install -y logstash-7.6.1.rpm[rootserver5 ~]# /usr/share/logstash/bin/logstash -e input { stdin { } } output { stdout {} }出现successfully可输入数据2.2 输出插件编写持久化配置文件test.conf[rootserver5 ~]# cd /etc/logstash/[rootserver5 logstash]# cd conf.d/[rootserver5 conf.d]# vim test.conf加载配置文件启动Logstash[rootserver5 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf此时界面有数据2.3 head插件解压phantomjs压缩包将phantomjs二进制程序复制到全局PATH目录安装字体依赖库phantomjs运行缺少字体库会直接启动报错。验证phantomjs环境是否安装成功无报错代表环境就绪CtrlC退出。yum install -y bzip2tar jxf phantomjs-2.1.1-linux-x86_64.tar.bz2cd phantomjs-2.1.1-linux-x86_64cp bin/phantomjs /usr/local/bin/yum install -y fontconfigphantomjs安装NodeJSelasticsearch-head基于Node.js开发npm打包、启动服务必须依赖node环境。使用国内淘宝镜像下载项目所有依赖包rpm -ivh nodejs-9.11.2-1nodesource.x86_64.rpmyum install -y unzipunzip elasticsearch-head-master.zipcd elasticsearch-head-master/npm install --registryhttps://registry.npmmirror.com/配置Head前端默认连接的ES集群地址[rootserver4 elasticsearch-head-master]# vim _site/app.js在第4388行修改成集群内的节点IP后台启动 head 服务head默认占用9100端口[rootserver4 elasticsearch-head-master]# npm run start [rootserver4 elasticsearch-head-master]# netstat -antlp|grep :9100tcp 0 0 0.0.0.0:9100 0.0.0.0:* LISTEN 20156/gruntES集群开启跨域配置添加下面图片圈出来的两行开启ES跨域资源共享允许所有地址跨域访问ES[rootserver1 ~]# vim /etc/elasticsearch/elasticsearch.yml重启服务[rootserver1 ~]# systemctl restart elasticsearch.service浏览器访问http://192.168.159.154:9100停止server2的elasticsearch集群健康状态变为yellow将server2再次打开由于ES集群分片自动均衡机制会自动加入集群且在页面显示server2成功加入集群ES内部自动同步分片数据集群状态自动恢复green2.4 file插件[rootserver5 conf.d]# vim test.conffile{}作用文件输入插件持续采集磁盘日志文件生产环境主流采集方式替代stdin 测试插件可以持续监控日志新增内容类似 tail -f。path /var/log/messages作用指定要监控采集的日志文件路径/var/log/messages是CentOS系统全局系统日志用来做实验数据源。start_position beginning作用首次读取文件时默认值是end首次启动只读取新增日志实验需要读取历史日志因此修改为beginning。加载 file 采集配置启动Logstash[rootserver5 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf会自动生成Logstash File插件会维护sincedb偏移量文件记录文件inode、已经读到哪个字节位置。先清理偏移量保证可以从头读取如果看不到最新日期的数据写入则是数据冗余了删掉下方文件[rootserver5 ~]# cd /usr/share/logstash/data/plugins/inputs/file[rootserver5 file]# l.. .. .sincedb_452905a167cf4509fd08acb964fdb20c[rootserver5 file]# rm -f .sincedb_452905a167cf4509fd08acb964fdb20c重新启动Logstash[rootserver5 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf就能看到新的日期数据写入2.5 syslog插件[rootserver5 conf.d]# vim rys.confinput { syslog {} }启用 syslog 输入插件默认监听UDP 514端口接收标准syslog协议报文 自动解析syslog协议格式自动拆分出程序名、优先级、设备类型、日志源 IP 等字段不需要手动正则切割。stdout {}控制台打印日志调试观察解析效果。elasticsearch { ... }将解析完成的syslog日志写入ES集群index rsyslog-%{YYYY.MM.dd}按日期生成独立索引区分文件采集日志与syslog转发日志。[rootserver5 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/rys.conf[rootserver1 ~]# vim /etc/rsyslog.conf加载新的日志转发规则[rootserver1 ~]# systemctl restart rsyslog.service[rootserver1 ~]# logger server1此时网页页面也有2.6 多行合并插件将 server1上ES日志文件拷贝至server5作为file插件监控的日志源文件。[rootserver1 ~]# cd /var/log/elasticsearch[rootserver1 elasticsearch]# scp my-es.log server5:/var/log/[rootserver5 conf.d]# vim myes.conf[rootserver5 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/myes.conf此时错误为一行输出2.7 grok过滤插件Apache访问日志是单行纯文本字符串无法直接统计IP、请求方式、状态码、URI。grok插件使用正则模板把非结构化日志自动切割成独立结构化字段方便ES检索、Kibana绘图统计。搭建Apache测试环境压力测试产生访问日志[rootserver5 ~]# yum install -y httpd[rootserver5 ~]# systemctl enable --now httpd[rootserver5 ~]# echo www.westos.org /var/www/html/index.html[rootserver1 ~]# ab -c1 -n 300 http://192.168.159.155/index.html[rootserver5 conf.d]# vim grok.conf[rootserver5 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/grok.conf三、firebeat日志分析Filebeat安装与启用Apache模块rpm -ivh filebeat-7.6.1-x86_64.rpmcd /etc/filebeat/modules.dfilebeat modules enableapache3.1 Filebeat直接输出Elasticsearchvim apache.yml日志流向Filebeat采集 → 直接写入ES。vim /etc/filebeat/filebeat.yml# 校验配置语法是否合法[rootserver5 modules.d]# filebeat test config -c /etc/filebeat/filebeat.yml# 测试输出连通性[rootserver5 modules.d]# filebeat test output -c /etc/filebeat/filebeat.yml# 测试输出连通性[rootserver5 modules.d]# systemctl enable --now filebeat3.2 Filebeat输出到Logstash修改配置文件vim/etc/logstash/conf.d/beats.conf修改filebeat.yml注释elasticsearch输出启用logstash输出四、kibana可视化下载安装kibana并配置文件修改成下方图片的模样[rootserver5 ~]# rpm -ivh kibana-7.6.1-x86_64.rpm[rootserver5 ~]# cd /etc/kibana/[rootserver5 kibana]# vim kibana.yml启动kibana服务确认端口已被监听[rootserver5 kibana]# systemctl enable --now kibana[rootserver5 kibana]# systemctl start kibana[rootserver5 kibana]# netstat -antlp |grep :5601浏览器访问5601端口进入后创建索引模式按以下图片进行新建可视化可自行进行选择创建添加可视化图形或指标添加仪表板把多张图表整合到一个大屏实现日志监控面板。五、Elasticsearch快照备份与恢复5.1 备份快照创建共享目录并配置权限ES运行用户需要读写目录放宽权限避免权限拒绝[rootserver5 ~]# yum install -y nfs-utils[rootserver5 ~]# mkdir -p /data/es-backup[rootserver5 ~]# chmod 777 /data/es-backup/[rootserver5 ~]# vim /etc/exports编辑NFS共享配置文件[rootserver5 ~]# systemctl enable --now nfs在elasticsearch三个节点同时安装nfs-utils并且配置同样的以下操作[rootserver2 ~]# mkdir -p /data/es-backup挂载远端 NFS 共享目录[rootserver2 ~]# mount 192.168.159.155:/data/es-backup /data/es-backup确认已经挂载成功修改配置文件elasticsearch.yml定义快照仓库允许使用的本地路径[rootserver2 ~]# vim /etc/elasticsearch/elasticsearch.yml重启服务[rootserver2 ~]# systemctl restart elasticsearch5.2 滚动重启点击扳手设置临时分片策略滚动重启前执行PUT _cluster/settings{transient: {cluster.routing.allocation.enable: primaries}}出现“true”表示成功集群全部节点重启完成恢复默认分片策略PUT _cluster/settings{transient: {cluster.routing.allocation.enable: null}}5.3 API创建快照仓库type: fs仓库类型为本地文件系统仓库location存储路径与path.repo配置保持一致compress: true开启快照文件压缩节省磁盘空间。PUT /_snapshot/my_backup{type: fs,settings: {location: /data/es-backup,compress: true}}验证已经有了验证仓库创建成功GET /_snapshot/my_backup5.4 创建快照数据备份操作快照支持两种模式备份集群全部索引、仅备份指定索引。5.4.1 备份集群内所有索引PUT /_snapshot/my_backup/snapshot_all_20260811 # 快照名称快照名称带上日期方便后期区分管理5.4.2 只备份指定索引indices指定需要备份的索引支持通配符“*”多个索引用逗号隔开ignore_unavailable: true如果部分索引不存在不中断备份任务include_global_state: false不备份集群全局状态索引模板、别名等。恢复时避免覆盖集群现有模板。PUT /_snapshot/my_backup/snapshot_all_20260811{indices: syslog-*,ignore_unavailable: true,include_global_state: false}5.4.3 查看仓库内所有快照GET /_snapshot/my_backup/_all5.4.4 查看单个快照GET /_snapshot/my_backup/snapshot_all_202608115.4.5 监控快照进度GET /_snapshot/my_backup/snapshot_all_20260811/_status5.5 恢复快照# 关闭待恢复索引 POST /apachelog-*/_close # 执行快照恢复 POST /_snapshot/my_backup/snapshot_apachelog_20260812/_restore { indices: apachelog-*, ignore_unavailable: true, include_global_state: false } # 恢复完成后开启索引 POST /apachelog-*/_open六、ES集群监控6.1 xpack认证server1上生成SSL证书开启集群传输层SSL加密cd /usr/share/elasticsearch/生成CA根证书bin/elasticsearch-certutil ca使用CA签发集群通信证书elastic-certificates.p12bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12将证书移动至ES配置目录cp elastic-certificates.p12 /etc/elasticsearchcd /etc/elasticsearch/修改文件属主为elasticsearch运行用户chown elasticsearch elastic-certificates.p12修改所有节点elasticsearch.yml配置三个节点都需要vim elasticsearch.ymlsystemctl restart elasticsearch证书分发至其余节点[rootserver1 elasticsearch]# scp elastic-certificates.p12 server2:/etc/elasticsearch/[rootserver1 elasticsearch]# scp elastic-certificates.p12 server3:/etc/elasticsearch/server2上执行相同操作但配置内容有所不一样[rootserver2 ~]# chown elasticsearch /etc/elasticsearch/elastic-certificates.p12[rootserver2 ~]# vim /etc/elasticsearch/elasticsearch.yml[rootserver2 ~]# systemctl restart elasticsearchserver3上和server2同样执行相同操作[rootserver3 ~]# chown elasticsearch /etc/elasticsearch/elastic-certificates.p12[rootserver3 ~]# vim /etc/elasticsearch/elasticsearch.yml[rootserver3 ~]# systemctl restart elasticsearch设置ES内置预留用户密码server1设置密码[rootserver1 elasticsearch]# bin/elasticsearch-setup-passwords interactiveES启动安全机制后必须初始化内置账号密码elastic、kibana、logstash_system、beats_system、apm_system、remote_monitoring_user配套组件适配账号密码访问 ES开启安全后Head无法直连访问地址格式需要追加认证参数http://192.168.159.154:9100/?auth_userelasticauth_passwordelastic、[rootserver4 ~]# cd elasticsearch-head-master/[rootserver4 elasticsearch-head-master]# npm run start [rootserver4 ~]# docker start cerebro修改logstash配置文件[rootserver5 ~]# vim /etc/logstash/conf.d/grok.conf[rootserver5 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/grok.conf修改kibana文件[rootserver5 kibana]# vim kibana.ymlkibana通过用户和密码正常访问6.2 Metricbeat 部署实现ES集群监控采集server1、server2、server3执行相同的操作安装metricbeat监控[rootserver1 ~]# rpm -ivh metricbeat-7.6.1-x86_64.rpm[rootserver1 ~]# cd /etc/metricbeat/modules.d/启用ES监控模块[rootserver1 modules.d]# metricbeat modules enable elasticsearch-xpack修改模块配置elasticsearch-xpack.yml一般用自己的本机IP监测的是本机状况和数据[rootserver1 modules.d]# vim elasticsearch-xpack.yml修改metricbeat.yml配置输出ES地址与账号最好统一用一个节点的[rootserver1 metricbeat]# vim metricbeat.yml[rootserver1 metricbeat]# systemctl enable --now metricbeat.service点击内部检测进入设置模式当所有步骤配置好后会全部metricbeat进行监测server2和server3的操作一样