小乐的博客

Filebeat 7.11 读取Nginx日志时，默认配置会读取失败，故作备忘

• 修改Nginx日志生成格式

      log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                              '$status $body_bytes_sent "$http_referer" '
                              '"$http_user_agent" "$http_x_forwarded_for" '
                              '"$http_x_real_ip" "$server_addr" "$host" '
                              '$request_time $upstream_response_time "$upstream_addr" '
                              '"$time_iso8601"';

• 修改Filebeat的Nginx规则

  ....filebeat/module/nginx/access/ingest/default.json
  //原格式
  - grok:
      field: message
      patterns:
      - (%{NGINX_HOST} )?"?(?:%{NGINX_ADDRESS_LIST:nginx.access.remote_ip_list}|%{NOTSPACE:source.address})
        - (-|%{DATA:user.name}) \[%{HTTPDATE:nginx.access.time}\] "%{DATA:nginx.access.info}"
        %{NUMBER:http.response.status_code:long} %{NUMBER:http.response.body.bytes:long}
        "(-|%{DATA:http.request.referrer})" "(-|%{DATA:user_agent.original})"
  //修改后的格式
  - grok:
      field: message
      patterns:
        - "\"?%{IP_LIST:nginx.access.remote_ip_list} - %{DATA:nginx.access.user_name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{GREEDYDATA:nginx.access.info}\" %{NUMBER:nginx.access.response_code} %{NUMBER:nginx.access.body_sent.bytes} \"%{DATA:nginx.access.referrer}\" \"%{DATA:nginx.access.agent}\" \"%{DATA:nginx.access.xff}\" \"%{DATA:nginx.access.x_real_ip}\" \"%{DATA:nginx.access.server_addr}\" \"%{DATA:nginx.access.host}\" %{DATA:nginx.access.request_time} %{DATA:nginx.access.upstream_response_time} \"%{DATA:nginx.access.upstream_addr}\" \"%{DATA:nginx.access.time_iso8601}\""

• 修改 fields.yml

  //找到对应的Nginx节点
  - name: agent
  .......
  //在上面节点的后面增加如下(空格等格式自己把握)
             - name: xff
               type: group
               description: >
                 http_x_forwarded_for.
             - name: x_real_ip
               type: group
               description: >
                 http_x_real_ip.
             - name: server_addr
               type: group
               description: >
                 server_addr 服务器地址.
             - name: host
               type: group
               description: >
                 host http_host http主机头.
             - name: request_time
               type: group
               description: >
                 request_time 请求时间.
             - name: upstream_response_time
               type: group
               description: >
                 upstream_response_time 后端响应时间.
             - name: upstream_addr
               type: group
               description: >
                 upstream_addr 后端地址.
             - name: time_iso8601
               type: group
               description: >
                 time_iso8601 iso8601格式时间.

• 重启filebeat

延伸

• 如果Nginx日志是Json格式

      log_format main '{"@timestamp":"$time_iso8601",'
                      '"host":"$server_addr",'
                      ' "clientip" : "$remote_addr",'
                      ' "size" : "$body_bytes_sent" ,'
                      '"respnsetime":"$request_time",'
                      '"upstremtime":"$upstream_response_time",'
                      '"upstremhost":"$upstream_addr",'
                      '"httphost":"$host",'
                      '"referer":"$http_referer",'
                      '"xff":"$http_x_forwarded_for",'
                      '"agent":"$http_user_agent",'
                      '"clientip":"$remote_addr",'
                      '"request":"$request",'
                      '"uri":"$uri",'
                      '"status":"$status"}';

• 同样需要修改增加fields.yml