Nginx日志中的双引号解析问题

这个问题其实是在上一篇的问题解决之后出现的又一个问题,还好测试到了,问题是这样的:

在Nginx记录的日志中会把出现在日志行中的双引号解析为x22

apache里面的正常日志应该是下面这样:

18.14.6.19 [28/Aug/2014:16:20:04 +0800] GET /server/shop/game/118 pos=0&cnt=50 200 \"clientInfo\":{\"num\":\"1375\",\"op\":\"46000\",\"dev\":\"GN708W\",\"imei\":\"862795012135838\",\"pkg\":\"com.assistor\",\"vc\":420,\"imsi\":\"460009182585103\",\"sdk\":17,\"vn\":\"4.2.0\",\"ccid\":\"5\"} 0.046

但是同样的log_format参数配置,到了Nginx里面就成了下面这样:

18.14.6.19 [28/Aug/2014:16:20:04 +0800] GET /server/shop/game/118 pos=0&cnt=50 200 \x22clientInfo\x22:{\x22num\x22:\x221375\x22,\x22op\x22:\x2246000\x22,\x22dev\x22:\x22GN708W\x22,\x22imei\x22:\x22862795012135838\x22,\x22pkg\x22:\x22com.assistor\x22,\x22vc\x22:420,\x22imsi\x22:\x22460009182585103\x22,\x22sdk\x22:17,\x22vn\x22:\x224.2.0\x22,\x22ccid\x22:\x225\x22} 0.046

查了很多资料,其中这里提到了,但是按照这办法没有测试成功,后来看到Nginx的Changelog里面是这样写的

Changes with nginx 1.1.6 17 Oct 2011

*) Change: now the 0x7F-0x1F characters are escaped as \xXX in an
   access_log.
Changes with nginx 0.7.0 19 May 2008

*) Change: now the 0x00-0x1F, '"' and '\' characters are escaped as \xXX
   in an access_log.
   Thanks to Maxim Dounin.

貌似Nginx是有意为之,只能理解为Nginx默认的log_format使用双引号作为间隔符,为了避免日志分析时候出现混乱,所以将双引号解析为x22了。

后来想了个笨办法,在每天日志切割的时候,替换日志中的x22字符为双引号,也算是绕过这个烦人的问题了。

附Nginx日志切割脚本

#!/bin/bash

#setting log path
log_files_path="/usr/local/nginx/logs/"
log_files_dir=${log_files_path}$(date -d "yesterday" +"%Y")/$(date -d "yesterday" +"%m")

#setting nginx
nginx_sbin="/usr/local/nginx/sbin/nginx"

if [ ! -d $log_files_dir ]; then
    mkdir -p $log_files_dir
fi

cd $log_files_path

#setting log name
log_files_name=(www.abc.com)
log_files_num=${#log_files_name[@]}

for((i=0;i<$log_files_num;i++));do
mv ${log_files_path}${log_files_name[i]}.log ${log_files_dir}/${log_files_name[i]}.log
/bin/sed -e 's/x22/"/g' ${log_files_dir}/${log_files_name[i]}.log > ${log_files_dir}/${log_files_name[i]}_$(date -d "yesterday" +"%Y%m%d").log
/bin/rm ${log_files_dir}/${log_files_name[i]}.log
done

kill -USR1 `cat /usr/local/nginx/nginx.pid`

 

 

在Nginx中记录自定义Header

从Apache切到Nginx需要保持日志格式统一,以便兼容之前的数据统计脚本

现在Apache的日志格式为:

LogFormat "%h %t %m %U %q %>s %{HEAD}i %D"

说明:

%h:客户端IP地址

%t:时间(标准英语格式)

%m:请求的方法(GET,POST)

%U:请求的URL路径,不包含查询字符串

%q:查询字符串

%>s:请求的最终状态

%{HEAD}i:请求头信息

%D:服务器处理请求所用时间,微秒为单位

对应的Nginx日志格式为:

log_format  main  '$remote_addr [$time_local] $request_method $uri '
                  '$query_string $status '
                  '$http_head $request_time ';

其中Apache里的%{HEAD}i这个参数里的HEAD是在程序里自定义了一个header,在Nginx里面取这个HEAD用到的是$http_head参数,head就是自定义的header名称HEAD, Nginx里面需要小写,在这里走了不少弯路。

用firebug获取http header,如下图

Snip20140828_3

有两部分内容,一个是响应头,一个是请求头,请求头是客户端发送给服务器的,包括Requst Line及HTTP Header,响应头是服务器返回给客户端的,包括Status Line及HTTP Header。关于http header的说明,这里有一篇很好的文章可以参考。

在Nginx里面可以用$upstream_http_’header名称’来获取响应头信息,比如$upstream_http_content_type将会获取到类似text/html; charset=utf-8的内容,说明返回的文件类型及编码格式。在我的需求中是要获取自定义的请求头的内容,需要使用$sent_http_’自定义header名称’来获取,但我测试过程中没有取到,用$http_’自定义header名称’才取到,也许和Nginx版本有关,我的版本是1.4.2。

另外,测试过程中用的是firefox的poster插件,可以自定义请求头,截图如下:

Snip20140828_4

自定义一个header,然后输入URL,点GET即可发送带有自定义header的请求,在服务器端查看日志可以看到取到了‘AAAAAAAAAAAAAAAAAAAAAA’.

nginx.org里关于log_format参数的说明,英文不好,看着费劲。

The ngx_http_core_module module supports embedded variables with names matching the Apache Server variables. First of all, these are variables representing client request header fields, such as $http_user_agent, $http_cookie, and so on. Also there are other variables:

$arg_name
argument name in the request line

$args
arguments in the request line

$binary_remote_addr
client address in a binary form, value’s length is always 4 bytes

$body_bytes_sent
number of bytes sent to a client, not counting the response header; this variable is compatible with the “%B” parameter of the mod_log_config Apache module

$bytes_sent
number of bytes sent to a client (1.3.8, 1.2.5)

$connection
connection serial number (1.3.8, 1.2.5)

$connection_requests
current number of requests made through a connection (1.3.8, 1.2.5)

$content_length
“Content-Length” request header field

$content_type
“Content-Type” request header field

$cookie_name
the name cookie

$document_root
root or alias directive’s value for the current request

$document_uri
same as $uri

$host
in this order of precedence: host name from the request line, or host name from the “Host” request header field, or the server name matching a request

$hostname
host name

$http_name
arbitrary request header field; the last part of a variable name is the field name converted to lower case with dashes replaced by underscores

$https
“on” if connection operates in SSL mode, or an empty string otherwise

$is_args
“?” if a request line has arguments, or an empty string otherwise

$limit_rate
setting this variable enables response rate limiting; see limit_rate

$msec
current time in seconds with the milliseconds resolution (1.3.9, 1.2.6)

$nginx_version
nginx version

$pid
PID of the worker process

$pipe
“p” if request was pipelined, “.” otherwise (1.3.12, 1.2.7)

$proxy_protocol_addr
client address from the PROXY protocol header, or an empty string otherwise (1.5.12)
The PROXY protocol must be previously enabled by setting the proxy_protocol parameter in the listen directive.

$query_string
same as $args

$realpath_root
an absolute pathname corresponding to the root or alias directive’s value for the current request, with all symbolic links resolved to real paths

$remote_addr
client address

$remote_port
client port

$remote_user
user name supplied with the Basic authentication

$request
full original request line

$request_body
request body
The variable’s value is made available in locations processed by the proxy_pass, fastcgi_pass, uwsgi_pass, and scgi_pass directives.

$request_body_file
name of a temporary file with the request body
At the end of processing, the file needs to be removed. To always write the request body to a file, client_body_in_file_only needs to be enabled. When the name of a temporary file is passed in a proxied request or in a request to a FastCGI/uwsgi/SCGI server, passing the request body should be disabled by the proxy_pass_request_body off, fastcgi_pass_request_body off, uwsgi_pass_request_body off, or scgi_pass_request_body off directives, respectively.

$request_completion
“OK” if a request has completed, or an empty string otherwise

$request_filename
file path for the current request, based on the root or alias directives, and the request URI

$request_length
request length (including request line, header, and request body) (1.3.12, 1.2.7)

$request_method
request method, usually “GET” or “POST”

$request_time
request processing time in seconds with a milliseconds resolution (1.3.9, 1.2.6); time elapsed since the first bytes were read from the client

$request_uri
full original request URI (with arguments)

$scheme
request scheme, “http” or “https”

$sent_http_name
arbitrary response header field; the last part of a variable name is the field name converted to lower case with dashes replaced by underscores

$server_addr
an address of the server which accepted a request
Computing a value of this variable usually requires one system call. To avoid a system call, the listen directives must specify addresses and use the bind parameter.

$server_name
name of the server which accepted a request

$server_port
port of the server which accepted a request

$server_protocol
request protocol, usually “HTTP/1.0” or “HTTP/1.1”

$status
response status (1.3.2, 1.2.2)

$tcpinfo_rtt, $tcpinfo_rttvar, $tcpinfo_snd_cwnd, $tcpinfo_rcv_space
information about the client TCP connection; available on systems that support the TCP_INFO socket option

$time_iso8601local
time in the ISO 8601 standard format (1.3.12, 1.2.7)

$time_local

local time in the Common Log Format (1.3.12, 1.2.7)

$uri
current URI in request, normalized
The value of $uri may change during request processing, e.g. when doing internal redirects, or when using index files.

参考:http://blog.51yip.com/apachenginx/1277.html

用Zabbix监控nginx日志里的http状态码

方法很简单,用脚本取nginx上一个小时输出的日志,然后计算几种http状态码出现的次数即可,修改这里用到的那个脚本,再次感谢krish@toonheart.com


#!/bin/bash
# Script to fetch nginx statuses for tribily monitoring systems
 
HOST=m.xyz.com
#HOST=`/sbin/ifconfig eth0 | sed -n '/inet /{s/.*addr://;s/ .*//;p}'`
PORT="80"
#webstatus=nginx-status
# Functions to return nginx stats

function active {
        /usr/bin/curl "http://$HOST:$PORT/nginx-status" 2>/dev/null| grep 'Active' | awk '{print $NF}' 
        } 

function reading {
        /usr/bin/curl "http://$HOST:$PORT/nginx-status" 2>/dev/null| grep 'Reading' | awk '{print $2}' 
        } 

function writing {
        /usr/bin/curl "http://$HOST:$PORT/nginx-status" 2>/dev/null| grep 'Writing' | awk '{print $4}' 
        } 

function waiting {
        /usr/bin/curl "http://$HOST:$PORT/nginx-status" 2>/dev/null| grep 'Waiting' | awk '{print $6}' 
        } 

function accepts {
        /usr/bin/curl "http://$HOST:$PORT/nginx-status" 2>/dev/null| awk NR==3 | awk '{print $1}'
        } 

function handled {
        /usr/bin/curl "http://$HOST:$PORT/nginx-status" 2>/dev/null| awk NR==3 | awk '{print $2}'
        } 

function requests {
        /usr/bin/curl "http://$HOST:$PORT/nginx-status" 2>/dev/null| awk NR==3 | awk '{print $3}'
        }

# Get http code status

logname='/usr/local/nginx/logs/abc.com.log'

function code_404 {
        more $logname |grep `date -d '1 hour ago' "+%d/%b/%Y:%H"`|cut -d '"' -f3|cut -d ' ' -f2|sort|uniq -c|sort -r|grep 404$|awk '{print $1}'
        }

function code_503 {
        more $logname |grep `date -d '1 hour ago' "+%d/%b/%Y:%H"`|cut -d '"' -f3|cut -d ' ' -f2|sort|uniq -c|sort -r|grep 503$|awk '{print $1}'
        }

function code_499 {
        more $logname |grep `date -d '1 hour ago' "+%d/%b/%Y:%H"`|cut -d '"' -f3|cut -d ' ' -f2|sort|uniq -c|sort -r|grep 499$|awk '{print $1}'
        }

function code_500 {
        more $logname |grep `date -d '1 hour ago' "+%d/%b/%Y:%H"`|cut -d '"' -f3|cut -d ' ' -f2|sort|uniq -c|sort -r|grep 500$|awk '{print $1}'
        }

function code_400 {
        more $logname |grep `date -d '1 hour ago' "+%d/%b/%Y:%H"`|cut -d '"' -f3|cut -d ' ' -f2|sort|uniq -c|sort -r|grep 400$|awk '{print $1}'
        }

# Run the requested function
$1

 

再修改/usr/local/zabbix_agent/conf/zabbix_agentd/userparameter_nginx.conf 文件

UserParameter=nginx.accepts,/usr/local/zabbix_agent/bin/get_nginx_status.sh accepts
UserParameter=nginx.handled,/usr/local/zabbix_agent/bin/get_nginx_status.sh handled
UserParameter=nginx.requests,/usr/local/zabbix_agent/bin/get_nginx_status.sh requests
UserParameter=nginx.connections.active,/usr/local/zabbix_agent/bin/get_nginx_status.sh active
UserParameter=nginx.connections.reading,/usr/local/zabbix_agent/bin/get_nginx_status.sh reading
UserParameter=nginx.connections.writing,/usr/local/zabbix_agent/bin/get_nginx_status.sh writing
UserParameter=nginx.connections.waiting,/usr/local/zabbix_agent/bin/get_nginx_status.sh waiting

UserParameter=nginx.code_404,/usr/local/zabbix_agent/bin/get_nginx_status.sh code_404
UserParameter=nginx.code_499,/usr/local/zabbix_agent/bin/get_nginx_status.sh code_499
UserParameter=nginx.code_500,/usr/local/zabbix_agent/bin/get_nginx_status.sh code_500
UserParameter=nginx.code_503,/usr/local/zabbix_agent/bin/get_nginx_status.sh code_503
UserParameter=nginx.code_400,/usr/local/zabbix_agent/bin/get_nginx_status.sh code_400

 

重启zabbix客户端程序,然后在服务器端测试:

root@ABC:/usr/local/zabbix/bin# ./zabbix_get  -s 192.168.0.1 -p 10050 -k "nginx.code_404"
11

 

最后,在Zabbix的web前端修改Template Nginx Status,添加几个状态码的Items即可。

 

 

Mysql二进制日志的清理

删除MySQL二进制日志的3种方法

1.RESET MASTER
可以删除列于索引文件中的所有二进制日志,把二进制日志索引文件重新设置为空,并创建一个新的二进制日志文件

2.PURGE MASTER LOGS
语法
PURGE {MASTER | BINARY} LOGS TO ‘log_name
PURGE {MASTER | BINARY} LOGS BEFORE ‘date
用于删除列于在指定的日志或日期之前的日志索引中的所有二进制日志。这些日志也会从记录在日志索引文件中的清单中被删除,这样被给定的日志成为第一个。
例如:
PURGE MASTER LOGS TO ‘mysql-bin.010’;
PURGE MASTER LOGS BEFORE ‘2003-04-02 22:46:26’;
BEFORE变量的date自变量可以为’YYYY-MM-DD hh:mm:ss’格式。MASTER和BINARY是同义词。
如果您有一个活性的从属服务器,该服务器当前正在读取您正在试图删除的日志之一,则本语句不会起作用,而是会失败,并伴随一个错误。不过,如果从属服务器是休止的,并且您碰巧清理了其想要读取的日志之一,则从属服务器启动后不能复制。当从属服务器正在复制时,本语句可以安全运行。您不需要停止它们。
要清理日志,需按照以下步骤:

1.
在每个从属服务器上,使用SHOW SLAVE STATUS来检查它正在读取哪个日志。
2.
使用SHOW MASTER LOGS获得主服务器上的一系列日志。
3.
在所有的从属服务器中判定最早的日志。这个是目标日志。如果所有的从属服务器是更新的,这是清单上的最后一个日志。
4.
制作您将要删除的所有日志的备份。(这个步骤是自选的,但是建议采用。)
5.
清理所有的日志,但是不包括目标日志。

3. expire_logs_days参数
二进制日志自动删除的天数。默认值为0,表示“没有自动删除”。启动时和二进制日志循环时可能删除
expire_logs_days numeric GLOBAL
在my.cnf配置文件【mysqld】段中加入,重启MySQL服务,例:
expire_logs_days=3
删除3天以前的日志

注:当然需要考虑有slave情况下的影响