使用Logstash+Redis+ElasticSearch+Kibana分析nginx日志 2015-08-05 21:21

介绍

方法

Logstash读日志发送到Redis

  • 定制nginx的日志格式

详见此文档定制nginx的日志格式

  • 配置LogStash支持解析nginx日志格式

在logstash目录下新增文件:vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.1.10/patterns/nginx

内容如下:

NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:http_host} %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NUMBER:request_time:float} %{NUMBER:upstream_time:float}
NGINXACCESS %{IPORHOST:http_host} %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NUMBER:request_time:float}
  • 配置Logstash

创建logstash配置文件nginx_redis.conf,内容如下:

input {
    file {
        path => [ "/opt/app/blog/log/nginx_access.log" ]
    }
}

filter {
    mutate { 
        replace => { "type" => "nginx_access" } 
    }
    grok {
        match => { "message" => "%{NGINXACCESS}" }
    }
    date {
        match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
    }
    geoip {
        source => "clientip"
    }
}


output {
    redis { host => "127.0.0.1" 
            data_type => "list"
            key => "logstash_nginx_access_log" 
    }
}
bin/logstash -f nginx_redis.conf

Logstash从Redis读日志发送到ElasticSearch

创建logstash配置文件redis_es.conf,内容如下:

input {
    redis {
        host => "127.0.0.1"
        port => "6379"
        key => "logstash_nginx_access_log"
        data_type => "list"
        codec  => "json"
        type => "logstash-arthas-access"
        tags => ["arthas"]
    }
}

output {
    elasticsearch {
        host => "127.0.0.1"
        index => "logstash-arthas-access-%{+YYYY.MM.dd}"
    }
}
bin/logstash -f redis_es.conf

安装Kibana

cd /opt/
wget https://download.elastic.co/kibana/kibana/kibana-4.0.3-linux-x64.tar.gz
tar -zxvf kibana-4.0.3-linux-x64.tar.gz
mv kibana-4.0.3-linux-x64.tar.gz kibana

vi conf/kibana.yml, 修改其中的elasticsearch_url,指向正确的ES地址。

  • 访问Kibana
http://ip:5601

参考文档

  1. LOGSTASH+ELASTICSEARCH+KIBANA处理NGINX访问日志