EFK-03 开始收集日志

2025-05-12

收集日志

从上一篇文章中获取日志路径

logs
└── test
    └── test_info.log

项目启动会出现如下的文件目录。

创建索引模板

进入kibana

创建template

put _template/java_log_template
{
  "order": 1,
  //注意这里
  "index_patterns": [
    "java_log_*" 
  ],
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 2
  },
  //mapping根据自己的条件来
  "mappings": {  
    "properties": {
      "serviceName":{
        "type": "keyword"
      },
      "level":{
        "type": "keyword"
      },
      "requestTime":{
        "type": "date"
      },
      "responseTime":{
        "type": "date"
      },
      "stackTrace":{
        "type":"text",
        "analyzer": "ik_max_word",
        "search_analyzer": "ik_smart",
        "norms": false
      },
      "message":{
        "type":"text",
        "analyzer": "ik_max_word",
        "search_analyzer": "ik_smart",
        "norms": false
      },
      "@timestamp":{
        "type": "date"
      }
    }
  }
}

配置filebeat.yml

filebeat.inputs:
- type: log
  id: test
  enabled: true
  tag: ['test']
  paths:
  #替换成你的日志路劲
    - /Users/vv/logs/project/test/*.log
  encoding: utf-8
  #json数据的处理
  json.keys_under_root: true
  json.add_error_key: true
  fields:
    server-name: test
​
setup.template.settings:
  index.number_of_shards: 1
setup.template.enable: true
#这里和template名称对应
setup.template.name: 'java_log_template'
#这里和template中的index_patterns名称对应 
setup.template.pattern: java_log_*'
setup.ilm.enabled: false
​
setup.kibana:
output.elasticsearch:
  hosts: ["localhost:9200"]
  username: "elastic"
  password: "你输入的密码"
  index: "java_log_%{[fields.server-name]}"
  #如果server-name包含特定的名称则命中
  indices:
    - index: "java_log_test_%{+yyyy.MM.dd}"
      when.equals: 
        fields: 
          server-name: "test"
​
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

启动filebeat

./filebeat -e -c filebeat.yml

注意查看日志。

这里代表将日志上传到了es，每个版本不一定。

2022-07-20T16:31:55.395+0800 INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(elasticsearch(http://localhost:9200)) established

查看索引是否简历

java_log_test 已经存在了。

建立索引模式

查询数据

完成。

测试

在java demo项目中新建controller。

import com.alibaba.fastjson.JSONObject;
import lombok.extern.slf4j.Slf4j;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
​
import static net.logstash.logback.argument.StructuredArguments.entries;
@RestController
@RequestMapping("/test")
@Slf4j
public class TestController {
    @PostMapping
    public JSONObject test(@RequestBody JSONObject request){
        log.info("entries",entries(request));
        log.info("no-entries {}",request);
        return request;
    }
}
​

调用日志

调用接口

POST http://localhost:8081/test
{
  "value": "1"
}

打印的日志

{
  "value": "1",
  "date": "2022-07-20 09:05:24.622",
  "level": "INFO",
  "message": "entries",
  "traceId": "",
  "thread": "http-nio-8081-exec-1",
  "serverName": "test"
}
{
  "date": "2022-07-20 09:05:24.623",
  "level": "INFO",
  "message": "no-entries {"value":"1"}",
  "traceId": "",
  "thread": "http-nio-8081-exec-1",
  "serverName": "test"
}

区别第一条记录中多了"value": "1"节点。而第二条是在 message打印了请求的信息。

entries() 方法可以在日志中增加属性。也可以用v() 方法、如下：

java

log.info("no-entries",v("param",request));

日志

{
  "param": {
    "value": "1"
  },
  "date": "2022-07-20 09:11:41.576",
  "level": "INFO",
  "message": "no-entries",
  "traceId": "",
  "thread": "http-nio-8081-exec-2",
  "serverName": "test"
}

详细解释: https://github.com/logfellow/logstash-logback-encoder#event-specific-custom-fields

查看日志是否进入es

查看具体的信息

{
#省略其他信息
    "value": [
      "1"
    ],
    "value": [
      "1"
    ],
    "thread": [
      "http-nio-8081-exec-2"
    ],
    "message": [
      "entries"
    ],
    "@timestamp": [
      "2022-07-20T09:11:42.882Z"
    ],
    "level.keyword": [
      "INFO"
    ],
  }
}