Проблема с PF или с системой

[mnemonic]

Добрый день!

Сервер: FreeBSD 8.4 amd 64, сетевухи 2 двухпортовые 82576 - igb 2.3.10

Обязанности: PF NAT

 

Периодически начинается задержки вплоть до потери пакетов. Хотя netstat молчит.

помогает очистка таблицы состояний

pfctl -F state

но не надолго!

 

понимаю что причиной всему какой то трафик. Потому как проблема носит периодический характер.

Возможно надо тюнить PF но я в нем не силен. Вроде бы и ресурсов достаточно. Трафик не большой 500Мбит/с, 50К пакетов.

 

Загрузка при этом небольшая

#top -SPH

last pid: 15734; load averages: 1.65, 1.25, 1.13 up 0+13:54:04 16:32:23

137 processes: 7 running, 78 sleeping, 52 waiting

CPU 0: 0.0% user, 0.0% nice, 0.8% system, 12.0% interrupt, 87.2% idle

CPU 1: 0.0% user, 0.0% nice, 1.5% system, 15.0% interrupt, 83.5% idle

CPU 2: 0.0% user, 0.0% nice, 1.9% system, 15.0% interrupt, 83.1% idle

CPU 3: 0.0% user, 0.0% nice, 1.5% system, 15.0% interrupt, 83.5% idle

CPU 4: 0.0% user, 0.0% nice, 1.9% system, 15.4% interrupt, 82.8% idle

CPU 5: 0.0% user, 0.0% nice, 2.3% system, 15.8% interrupt, 82.0% idle

Mem: 23M Active, 14M Inact, 769M Wired, 200K Cache, 764M Buf, 7063M Free

Swap: 4096M Total, 4096M Free

 

PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND

11 root 171 ki31 0K 96K CPU1 1 738:46 89.26% idle{idle: cpu1}

11 root 171 ki31 0K 96K RUN 3 737:55 88.77% idle{idle: cpu3}

11 root 171 ki31 0K 96K CPU5 5 736:47 86.77% idle{idle: cpu5}

11 root 171 ki31 0K 96K CPU2 2 733:43 83.69% idle{idle: cpu2}

11 root 171 ki31 0K 96K CPU4 4 733:36 83.25% idle{idle: cpu4}

11 root 171 ki31 0K 96K CPU0 0 731:06 82.86% idle{idle: cpu0}

12 root -44 - 0K 832K WAIT 0 73:55 12.79% intr{swi1: netisr 0}

12 root -44 - 0K 832K WAIT 2 69:53 12.60% intr{swi1: netisr 2}

12 root -44 - 0K 832K WAIT 4 69:56 12.26% intr{swi1: netisr 3}

12 root -44 - 0K 832K WAIT 0 72:48 11.67% intr{swi1: netisr 5}

12 root -44 - 0K 832K WAIT 1 68:59 11.67% intr{swi1: netisr 1}

12 root -44 - 0K 832K WAIT 0 70:22 11.47% intr{swi1: netisr 4}

12 root -68 - 0K 832K WAIT 3 9:10 0.98% intr{irq271: igb2:que}

12 root -68 - 0K 832K WAIT 0 7:58 0.98% intr{irq274: igb2:que}

12 root -68 - 0K 832K WAIT 2 8:52 0.78% intr{irq270: igb2:que}

12 root -68 - 0K 832K WAIT 5 8:21 0.78% intr{irq273: igb2:que}

12 root -68 - 0K 832K WAIT 4 8:20 0.68% intr{irq272: igb2:que}

12 root -68 - 0K 832K WAIT 1 8:09 0.59% intr{irq275: igb2:que}

12 root -68 - 0K 832K WAIT 2 6:28 0.59% intr{irq258: igb0:que}

12 root -68 - 0K 832K WAIT 4 6:21 0.59% intr{irq260: igb0:que}

12 root -68 - 0K 832K WAIT 0 6:32 0.49% intr{irq256: igb0:que}

 

# netstat -w1 -I lagg0

input (lagg0) output

packets errs idrops bytes packets errs bytes colls

37770 0 0 8635338 50058 0 59097321 0

39153 0 0 9225072 50614 0 59490156 0

37727 0 0 8555634 49671 0 58950358 0

35544 0 0 8320739 46340 0 54230153 0

38637 0 0 9295857 50521 0 59966492 0

37661 0 0 9357863 49138 0 58212143 0

36987 0 0 8108765 47896 0 56676719 0

 

# netstat -m

74741/12694/87435 mbufs in use (current/cache/total)

74738/10900/85638/204800 mbuf clusters in use (current/cache/total/max)

74738/10894 mbuf+clusters out of packet secondary zone in use (current/cache)

0/167/167/102400 4k (page size) jumbo clusters in use (current/cache/total/max)

0/0/0/51200 9k jumbo clusters in use (current/cache/total/max)

0/0/0/25600 16k jumbo clusters in use (current/cache/total/max)

168161K/25641K/193803K bytes allocated to network (current/cache/total)

0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)

0/0/0 requests for jumbo clusters denied (4k/9k/16k)

0/0/0 sfbufs in use (current/peak/max)

0 requests for sfbufs denied

0 requests for sfbufs delayed

0 requests for I/O initiated by sendfile

0 calls to protocol drain routines

 

# pfctl -si

No ALTQ support in kernel

ALTQ related functions disabled

Status: Enabled for 0 days 14:01:31 Debug: Urgent

 

State Table Total Rate

current entries 91186

searches 2716465761 53801.0/s

inserts 66159928 1310.3/s

removals 66068742 1308.5/s

Counters

match 91653858 1815.3/s

bad-offset 0 0.0/s

fragment 142 0.0/s

short 55 0.0/s

normalize 863 0.0/s

memory 0 0.0/s

bad-timestamp 0 0.0/s

congestion 0 0.0/s

ip-option 0 0.0/s

proto-cksum 21325 0.4/s

state-mismatch 138951 2.8/s

state-insert 76250 1.5/s

state-limit 0 0.0/s

src-limit 0 0.0/s

synproxy 0 0.0/s

 

TIMEOUTS:

tcp.first 30s

tcp.opening 5s

tcp.established 3600s

tcp.closing 60s

tcp.finwait 30s

tcp.closed 30s

tcp.tsdiff 10s

udp.first 60s

udp.single 30s

udp.multiple 60s

icmp.first 20s

icmp.error 10s

other.first 60s

other.single 30s

other.multiple 60s

frag 10s

interval 10s

adaptive.start 0 states

adaptive.end 0 states

src.track 30s

 

LIMITS:

states hard limit 900000

src-nodes hard limit 60000

frags hard limit 300000

tables hard limit 1000

table-entries hard limit 400000

[Ivan_83]

vmstat -z

[mnemonic]

Попробовал поднять LIMITS

LIMITS:

states hard limit 1000000

src-nodes hard limit 100000

frags hard limit 1000000

tables hard limit 1000000

table-entries hard limit 1000000

 

Но толку, НОЛЬ.

 

vmstat -z

# vmstat -z

ITEM SIZE LIMIT USED FREE REQUESTS FAILURES

 

UMA Kegs: 208, 0, 93, 9, 93, 0

UMA Zones: 384, 0, 93, 7, 93, 0

UMA Slabs: 568, 0, 961, 5, 1786, 0

UMA RCntSlabs: 568, 0, 44847, 2, 44847, 0

UMA Hash: 256, 0, 2, 13, 3, 0

16 Bucket: 152, 0, 163, 12, 163, 0

32 Bucket: 280, 0, 151, 3, 151, 5

64 Bucket: 536, 0, 176, 6, 176, 174

128 Bucket: 1048, 0, 1433, 1, 1433, 67

VM OBJECT: 216, 0, 1190, 754, 189036, 0

MAP: 232, 0, 7, 25, 7, 0

KMAP ENTRY: 120, 276520, 43, 329, 4634, 0

MAP ENTRY: 120, 0, 668, 1161, 411266, 0

DP fakepg: 120, 0, 0, 0, 0, 0

SG fakepg: 120, 0, 0, 0, 0, 0

mt_zone: 2056, 0, 172, 11, 172, 0

16: 16, 0, 3314, 550, 101352, 0

32: 32, 0, 12859, 877, 48090, 0

64: 64, 0, 29850, 8230, 488451248, 0

128: 128, 0, 75684, 528, 86499, 0

256: 256, 0, 710, 400, 34650, 0

512: 512, 0, 2480, 173, 22474, 0

1024: 1024, 0, 89, 431, 14001, 0

2048: 2048, 0, 52, 134, 58058, 0

4096: 4096, 0, 210, 125, 47691, 0

Files: 80, 0, 56, 1699, 265590, 0

TURNSTILE: 136, 0, 289, 111, 289, 0

umtx pi: 96, 0, 0, 0, 0, 0

MAC labels: 40, 0, 0, 0, 0, 0

PROC: 1136, 0, 38, 76, 13919, 0

THREAD: 1128, 0, 207, 81, 207, 0

SLEEPQUEUE: 80, 0, 289, 146, 289, 0

VMSPACE: 392, 0, 22, 228, 13903, 0

cpuset: 72, 0, 115, 335, 169, 0

audit_record: 952, 0, 0, 0, 0, 0

mbuf_packet: 256, 0, 81298, 8302, 250652009, 0

mbuf: 256, 0, 26, 1514, 239399560, 0

mbuf_cluster: 2048, 204800, 89600, 6, 89600, 0

mbuf_jumbo_page: 4096, 102400, 0, 44, 30, 0

mbuf_jumbo_9k: 9216, 51200, 0, 0, 0, 0

mbuf_jumbo_16k: 16384, 25600, 0, 0, 0, 0

mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0

g_bio: 232, 0, 0, 368, 9868, 0

ttyinq: 160, 0, 135, 129, 270, 0

ttyoutq: 256, 0, 72, 48, 144, 0

ata_request: 320, 0, 0, 204, 2490, 0

ata_composite: 336, 0, 0, 0, 0, 0

VNODE: 472, 0, 869, 91, 899, 0

VNODEPOLL: 112, 0, 0, 0, 0, 0

S VFS Cache: 108, 0, 878, 178, 1923, 0

STS VFS Cache: 148, 0, 0, 0, 0, 0

L VFS Cache: 328, 0, 0, 0, 0, 0

LTS VFS Cache: 368, 0, 0, 0, 0, 0

NAMEI: 1024, 0, 0, 112, 343263, 0

DIRHASH: 1024, 0, 47, 37, 47, 0

pipe: 728, 0, 2, 78, 11289, 0

ksiginfo: 112, 0, 101, 955, 245, 0

itimer: 344, 0, 0, 22, 1, 0

KNOTE: 128, 0, 0, 145, 20, 0

socket: 680, 204804, 12, 1386, 77048, 0

unpcb: 240, 204800, 6, 122, 93, 0

ipq: 56, 6426, 0, 0, 0, 0

udp_inpcb: 336, 204809, 2, 1439, 76917, 0

udpcb: 16, 204960, 2, 2182, 76917, 0

tcp_inpcb: 336, 204809, 5, 83, 23, 0

tcpcb: 944, 204800, 4, 44, 23, 0

tcptw: 72, 27800, 1, 249, 9, 0

syncache: 144, 15366, 0, 130, 8, 0

hostcache: 136, 15372, 2, 82, 2, 0

tcpreass: 40, 12852, 0, 0, 0, 0

sackhole: 32, 0, 0, 0, 0, 0

ripcb: 336, 204809, 0, 0, 0, 0

rtentry: 200, 0, 2087, 79, 2087, 0

pfsrctrpl: 152, 100000, 0, 0, 0, 0

pfrulepl: 912, 0, 7204, 8988, 28815, 0

pfstatepl: 392, 1000000, 145083, 30407, 10009346, 0

pfaltqpl: 240, 0, 0, 0, 0, 0

pfpooladdrpl: 88, 0, 7196, 9100, 28783, 0

pfrktable: 1296, 1000002, 2, 22, 10, 0

pfrkentry: 216, 1000008, 4, 86, 16, 0

pfrkentry2: 216, 0, 0, 0, 0, 0

pffrent: 32, 1000001, 0, 707, 6153, 0

pffrag: 80, 0, 0, 315, 2415, 0

pffrcache: 80, 10035, 0, 0, 0, 0

pffrcent: 24, 50022, 0, 0, 0, 0

pfstatescrub: 40, 0, 0, 0, 0, 0

pfiaddrpl: 120, 0, 0, 0, 0, 0

pfospfen: 112, 0, 710, 346, 2850, 0

pfosfp: 40, 0, 420, 504, 1686, 0

selfd: 56, 0, 56, 700, 42148, 0

SWAPMETA: 288, 116519, 0, 0, 0, 0

Mountpoints: 752, 0, 5, 10, 5, 0

FFS inode: 168, 0, 836, 88, 865, 0

FFS1 dinode: 128, 0, 0, 0, 0, 0

FFS2 dinode: 256, 0, 836, 49, 865, 0

Edited August 31, 2013 by mnemonic

[mnemonic]

На порту коммутатора Catalyst 3550 куда подключен сервер копились ошибки PAUSE output

Оказывается проблема была во включенном flowcontrol на порту

Заметил только проверив командой

show flowcontrol

Edited August 31, 2013 by mnemonic