Logstash 过滤NGINX中多个IP

问题

架构 WEB防火墙->SLB->Nginx

虽然Nginx配置了real_ip

set_real_ip_from 100.97.0.0/16;
real_ip_header X-Forwarded-For;
real_ip_recursive on;

但是获取到的X-Forwarded-For中却有2个IP，第一个是用户真实IP，第二个是WEB防火墙公IP，这样导致Logstash收集IP不准确

{
	"@timestamp": "2018-08-21T00:00:01+08:00",
	"@source": "10.47.24.8",
	"hostname": "*****",
	"ip": "180.172.191.175, 118.178.15.114",
	"client": "100.116.226.198",
	"request_method": "GET",
	"scheme": "http",
	"domain": "*****",
	"referer": "-",
	"request": "*******",
	"args": "-",
	"size": 2417,
	"status": 200,
	"responsetime": 0.000,
	"upstreamtime": "-",
	"upstreamaddr": "-",
	"http_user_agent": "Mozilla/5.0 (Linux; Android 7.1.1; OD105 Build/NMF26F; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/62.0.3202.84 Mobile Safari/537.36 FoApp/3.2.4 (Android)",
	"https": ""
}

解决办法

利用Logstash fileter dissect

配置文件如下

input {
    file {
        path => [ "/data/logs/*_access.log" ]
        ignore_older => 0
	codec => json
    }
}

filter {
    mutate {
      convert => [ "status","integer" ]
      convert => [ "size","integer" ]
      convert => [ "upstreatime","float" ]
      remove_field => "message"
    }
    dissect {
      mapping => {
          "ip" => "%{ip1}, %{ip2}"
      	  }
    }
    geoip {
        source => "ip1"
    }


}

参考

https://www.elastic.co/guide/en/logstash/current/plugins-filters-dissect.html