linux系统维护及程序管理(Linux Nginx LAST_ACK 超高处理)

今天有几台下载服务器,开发那边使用程序检查发现连接超时:
登上去一看,因为我这些服务器都是做软件下载的,都是百兆独享,要的就是高效响应和快速的连接,开始查看吧！

引用

[root@leysin02 ~]# netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'
LAST_ACK 1601
SYN_RECV 164
ESTABLISHED 568
FIN_WAIT1 311
FIN_WAIT2 5
TIME_WAIT 1
CLOSING 86
发现5百多个连接,导致了LAST_ACK高达1600 简直就是连接的3倍。恐怖哦！！！！

根据自己习惯问题。使用sar 3 10 观察一段时间

引用

[root@leysin02 zhangmh]# sar 3 10
Linux 2.6.9-67.ELsmp (leysin02)         05/25/2011

01:49:15 PM       CPU     %user     %nice   %system   %iowait     %idle
01:49:18 PM       all      0.17      0.00      0.50      4.16     95.17
01:49:21 PM       all      0.00      0.00      0.50      5.68     93.82
01:49:24 PM       all      0.17      0.00      0.33      5.99     93.51
01:49:27 PM       all      0.00      0.00      0.50      4.67     94.83
01:49:30 PM       all      0.17      0.00      0.67      7.83     91.33
01:49:33 PM       all      0.17      0.00      0.50      5.83     93.50
01:49:36 PM       all      0.00      0.00      0.67      4.83     94.50
01:49:39 PM       all      0.17      0.00      0.33      4.16     95.34
01:49:42 PM       all      0.17      0.00      0.50      4.50     94.83
01:49:45 PM       all      0.00      0.00      0.50      3.83     95.67
Average:          all      0.10      0.00      0.50      5.15     94.25
系统上没有什么负载,并且IO 没有什么瓶颈.

从Web 连接方面入手:

ESTABLISHED->CLOSE_WAIT->（发送ACK）->LAST_ACK->(发送FIN+接收ACK)->CLOSED
根据TCP状态的变化过程来分析,LAST_ACK属于被动关闭连接过程中的状态

引用

(1): 排除SYN 干扰:
[root@leysin02 ~]# sysctl -w net.ipv4.tcp_syncookies=1
[root@leysin02 ~]# sysctl -w net.ipv4.tcp_synack_retries=2
[root@leysin02 ~]# sysctl -w net.ipv4.tcp_syn_retries=2
(2):查看系统tcp_keepalive 状态:
[root@leysin02 ~]# sysctl -a|grep tcp_keepalive
net.ipv4.tcp_keepalive_intvl = 10
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_time = 1800

经过调试,LAST_ACK 明显下滑了不少。

[root@leysin02 ~]# netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'
LAST_ACK 906
SYN_RECV 73
ESTABLISHED 579
FIN_WAIT1 116
FIN_WAIT2 3
TIME_WAIT 7
CLOSING 12
加速处理那些等待ACK的LAST_ACK，减少等待ACK的LAST_ACK的重试次数

tcp_retries2:在丢弃激活(已建立通讯状况)的TCP连接之前﹐需要进行多少次重试
[root@leysin02 ~]# sysctl -w net.ipv4.tcp_retries2=3
net.ipv4.tcp_retries2 = 3
[root@leysin02 ~]# netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'
LAST_ACK 1451
SYN_RECV 71
ESTABLISHED 673
FIN_WAIT1 161
FIN_WAIT2 3
TIME_WAIT 3
CLOSING 53
哦，FIN_WAIT1 也高了！！,
[root@leysin02 ~]# sysctl -a |grep tcp_fin_timeout
net.ipv4.tcp_fin_timeout = 60
[root@leysin02 ~]# sysctl -a |grep tcp_fin_timeout
[root@leysin02 ~]# sysctl -w net.ipv4.tcp_fin_timeout=30
net.ipv4.tcp_fin_timeout = 30
[root@leysin02 ~]# sysctl -w net.ipv4.tcp_orphan_retries=3
net.ipv4.tcp_orphan_retries = 3 ####这个值需要上下移动,达到合适的值
LAST_ACK 762
SYN_RECV 74
ESTABLISHED 554
FIN_WAIT1 67
FIN_WAIT2 1
CLOSING 8
TIME_WAIT 32
[root@leysin02 ~]# sysctl -w net.ipv4.tcp_orphan_retries=2
[root@leysin02 ~]# netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'
LAST_ACK 430
SYN_RECV 75
ESTABLISHED 603
FIN_WAIT1 43
FIN_WAIT2 5
CLOSING 2
TIME_WAIT 5

我调试到2 发现LAST_ACK 值小的多了,观察一段时间，发现LAST_ACK 没有增长, 而且ESTABLISHED 连接数也没有降低多少!算是稳定了！