Проблемв с se100 после обновления bgbilling

clinkclinkclink · April 24, 2015

Обновили биллинг, после чего увелечилась загрузка cpu и начал зависать процесс aaad. Обновили прошивку на se100 с SEOS-se100-6.5.1.5.tar до SEOS-se100-12.1.1.9. Теперь процесс rib, bgp периодически скачут до 45%. А процесс stats постоянно загружен от 6 до 12%.

И уменьшается память.

Memory: Total 802716k, Used 755384k, Free 9844k, Reserved 24

NAME               PID    SPAWN    MEMORY TIME           %CPU STATE     UP/DOWN
evtmd             2761        1    38980K 00:00:01.57   0.00% run       01:06:40
pnsd              2762        1    10172K 00:00:01.15   0.00% run       01:06:40
csm               2763        1    47740K 00:00:04.18   0.00% run       01:06:38
rcm               2764        1   111428K 00:00:41.92   0.00% run       01:06:29
ism               2765        1    72916K 00:01:17.45   0.00% run       01:06:28
ped_parse         2766        1    11096K 00:00:00.77   0.00% run       01:06:28
issud             2767        1     9800K 00:00:00.62   0.00% run       01:06:28
vxxlatsvr         2768        1    13764K 00:00:00.75   0.00% run       01:06:27
rpm               2769        1    11820K 00:00:22.30   0.00% run       01:06:27
rib               2851        3   115864K 00:00:13.36  44.69% run       00:00:51
ntp               2799        1    50884K 00:00:01.50   0.00% run       01:06:04
arp               2771        1    41356K 00:00:15.57   0.00% run       01:06:25
static            2795        1     9872K 00:00:00.65   0.00% run       01:06:06
isis                 0        0        0K Not Avail     0.00% demand    01:07:03
rip                  0        0        0K Not Avail     0.00% demand    01:07:03
bgp               2794        1   107852K 00:00:54.03  17.38% run       01:06:07
igmp              2796        1    42088K 00:00:05.49   0.00% run       01:06:06
pim               2797        1    42740K 00:00:04.11   0.00% run       01:06:06
ospf                 0        0        0K Not Avail     0.00% demand    01:07:03
sysmon            2772        1    11240K 00:00:01.88   0.00% run       01:06:25
dns               2801        1     9704K 00:00:00.72   0.00% run       01:05:59
msdp                 0        0        0K Not Avail     0.00% demand    01:07:03
ppaslog           2773        1     9752K 00:00:00.61   0.00% run       01:06:25
vrrp                 0        0        0K Not Avail     0.00% demand    01:07:03
cls               2774        1    47852K 00:00:05.44   0.00% run       01:06:24
dot1q             2803        1    61088K 00:00:07.64   0.00% run       01:05:57
tunnel            2775        1    10576K 00:00:00.68   0.00% run       01:06:24
qos               2789        1    43132K 00:00:05.04   0.00% run       01:06:09
dlm               2776        1    14608K 00:00:00.93   0.00% run       01:06:23
pem               2777        1    42568K 00:00:01.00   0.00% run       01:06:23
dhcp              2808        1    56628K 00:00:10.58   0.00% run       01:05:54
fr                   0        0        0K Not Avail     0.00% demand    01:07:03
rsvp                 0        0        0K Not Avail     0.00% demand    01:07:03
mpls_static          0        0        0K Not Avail     0.00% demand    01:07:03
bridge            2805        1    40896K 00:00:01.16   0.00% run       01:05:56
atm               2804        1    53576K 00:00:01.42   0.00% run       01:05:57
lm                2778        1    41288K 00:00:01.05   0.00% run       01:06:22
nat                  0        0        0K Not Avail     0.00% demand    01:07:03
ospf3                0        0        0K Not Avail     0.00% demand    01:07:03
odd               2779        1    13324K 00:00:00.86   0.00% run       01:06:21
dhelperd          2811        1    40292K 00:00:05.89   0.00% run       01:05:53
tap                  0        0        0K Not Avail     0.00% demand    01:07:03
gsmp                 0        0        0K Not Avail     0.00% demand    01:07:03
mip                  0        0        0K Not Avail     0.00% demand    01:07:03
cspf                 0        0        0K Not Avail     0.00% demand    01:07:03
ipfix             2812        1    40608K 00:00:01.42   0.00% run       01:05:52
fsd                  0        0        0K Not Avail     0.00% demand    01:07:03
dhcpv6d           2813        1    53076K 00:00:02.87   0.00% run       01:05:52
dhelperv6d        2814        1    39176K 00:00:02.38   0.00% run       01:05:51
mcastmgr          2802        1    42032K 00:00:04.32   0.00% run       01:05:59
icrd                 0        0        0K Not Avail     0.00% demand    01:07:03
aaa_helperd       2780        1    39828K 00:00:00.83   0.00% run       01:06:21
snmp              2800        1    47628K 00:00:08.68   0.15% run       01:05:59
stats             2781        1    50940K 00:05:57.90   8.20% run       01:06:19
ppp               2798        1    46044K 00:00:42.45   0.00% run       01:06:04
xcd                  0        0        0K Not Avail     0.00% demand    01:07:03
lg                   0        0        0K Not Avail     0.00% demand    01:07:03
ldp               2782        1    40912K 00:00:01.99   0.00% run       01:06:18
netopd               0        0        0K Not Avail     0.00% demand    01:07:03
aaad              2783        1    70100K 00:00:52.14   0.05% run       01:06:15
pppoe             2806        1    41216K 00:00:21.61   0.00% run       01:05:56
l2tp              2807        1    41684K 00:00:04.05   0.00% run       01:05:55
clips             2809        1    53968K 00:00:06.12   0.00% run       01:05:54
nd                   0        0        0K Not Avail     0.00% demand    01:07:03
hr                2810        1    12568K 00:00:38.16   0.00% run       01:05:53
flowd             2784        1    50804K 00:00:01.64   0.00% run       01:06:14
l4l7d                0        0        0K Not Avail     0.00% demand    01:07:03
mipsim               0        0        0K Not Avail     0.00% demand    01:07:03
cfm                  0        0        0K Not Avail     0.00% demand    01:07:03
shm_ribd             0        0        0K Not Avail     0.00% demand    01:07:03
sctp              2785        1     9736K 00:00:00.55   0.00% run       01:06:14
mgmd                 0        0        0K Not Avail     0.00% demand    01:07:03
mgd-1                0        0        0K Not Avail     0.00% demand    01:07:03
mgd-2                0        0        0K Not Avail     0.00% demand    01:06:13
mgd-3                0        0        0K Not Avail     0.00% demand    01:06:13
ssm               2786        1    11444K 00:00:00.67   0.00% run       01:06:14
fssbcsim             0        0        0K Not Avail     0.00% demand    01:07:03
limm              2787        1    53460K 00:00:00.77   0.00% run       01:06:13
clibe             2788        1    10200K 00:00:00.58   0.00% run       01:06:13

Apr 24 15:06:33: %PM-6-INFO: pm_process_is_dead Invalidate pid rib(2850)
Apr 24 15:06:33: %PM-6-PROCDIE: rib is dying
Apr 24 15:06:33: %SYSLOG-0-EMERG: /netbsd: PID            NAME                MEMORY(Kbytes)
Apr 24 15:06:33: %SYSLOG-0-EMERG: /netbsd: 2850           ribd                138440
Apr 24 15:06:33: %SYSLOG-0-EMERG: /netbsd: 2848           exec_cli            15904
Apr 24 15:06:33: %SYSLOG-0-EMERG: /netbsd: 2847           sshd                9884
Apr 24 15:06:33: %SYSLOG-0-EMERG: /netbsd: 2821           exec_cli            16084
Apr 24 15:06:33: %SYSLOG-0-EMERG: /netbsd: 2820           sshd                9860
Apr 24 15:06:33: %SYSLOG-0-EMERG: /netbsd: 2816           exec_cli            16068
Apr 24 15:06:33: %SYSLOG-0-EMERG: /netbsd: 2815           sshd                9844
Apr 24 15:06:33: %SYSLOG-0-EMERG: /netbsd: 2814           dhelperv6d          7756
Apr 24 15:06:34: %SYSLOG-0-EMERG: /netbsd: 2813           dhcpv6d             10304
Apr 24 15:06:34: %SYSLOG-0-EMERG: /netbsd: 2812           ipfixd              9464
Apr 24 15:06:34: %SYSLOG-0-EMERG: /netbsd: 2811           dhelperd            8816
Apr 24 15:06:34: %SYSLOG-0-EMERG: /netbsd: 2810           hrd                 10148
Apr 24 15:06:34: %SYSLOG-0-EMERG: /netbsd: 2809           clipsd              11264
Apr 24 15:06:34: %SYSLOG-0-EMERG: /netbsd: 2808           dhcpd               13940
Apr 24 15:06:34: %SYSLOG-0-EMERG: /netbsd: 2807           l2tpd               10680
Apr 24 15:06:34: %SYSLOG-0-EMERG: /netbsd: 2806           pppoed              10096
Apr 24 15:06:35: %SYSLOG-0-EMERG: /netbsd: 2805           bridged             9632
Apr 24 15:06:35: %SYSLOG-0-EMERG: /netbsd: 2804           atmd                11208
Apr 24 15:06:35: %SYSLOG-0-EMERG: /netbsd: 2803           dot1qd              18600
Apr 24 15:06:35: %SYSLOG-0-EMERG: /netbsd: 2802           mcastmgrd           11912
Apr 24 15:06:35: %SYSLOG-0-EMERG: /netbsd: 2801           dnsd                7492
Apr 24 15:06:35: %SYSLOG-0-EMERG: /netbsd: 2800           snmpd               16716
Apr 24 15:06:35: %SYSLOG-0-EMERG: /netbsd: 2799           ntpd                8600
Apr 24 15:06:35: %SYSLOG-0-EMERG: /netbsd: 2798           pppd                15848
Apr 24 15:06:35: %SYSLOG-0-EMERG: /netbsd: 2797           pimd                12760
Apr 24 15:06:35: %SYSLOG-0-EMERG: /netbsd: 2796           igmpd               12228
Apr 24 15:06:35: %SYSLOG-0-EMERG: /netbsd: 2795           staticd             7784
Apr 24 15:06:35: %SYSLOG-0-EMERG: /netbsd: 2794           bgpd                106676
Apr 24 15:06:35: %SYSLOG-0-EMERG: /netbsd: 2789           qosd                12604
Apr 24 15:06:36: %SYSLOG-0-EMERG: /netbsd: 2788           clibe               8048
Apr 24 15:06:36: %SYSLOG-0-EMERG: /netbsd: 2787           limmd               10696
Apr 24 15:06:36: %SYSLOG-0-EMERG: /netbsd: 2786           ssmd                9056
Apr 24 15:06:36: %SYSLOG-0-EMERG: /netbsd: 2785           sctpd               7448
Apr 24 15:06:36: %SYSLOG-0-EMERG: /netbsd: 2784           flowd               8300
Apr 24 15:06:36: %SYSLOG-0-EMERG: /netbsd: 2783           aaad                29688
Apr 24 15:06:36: %SYSLOG-0-EMERG: /netbsd: 2782           ldpd                10588
Apr 24 15:06:36: %SYSLOG-0-EMERG: /netbsd: 2781           statd               22308
Apr 24 15:06:36: %SYSLOG-0-EMERG: /netbsd: 2780           aaa_helperd         8812
Apr 24 15:06:36: %SYSLOG-0-EMERG: /netbsd: 2779           oddd                10328
Apr 24 15:06:36: %SYSLOG-0-EMERG: /netbsd: 2778           lm                  10960
Apr 24 15:06:36: %SYSLOG-0-EMERG: /netbsd: 2777           pemd                10612
Apr 24 15:06:36: %SYSLOG-0-EMERG: /netbsd: 2776           dlmd                11516
Apr 24 15:06:36: %SYSLOG-0-EMERG: /netbsd: 2775           tunneld             8568
Apr 24 15:06:36: %SYSLOG-0-EMERG: /netbsd: 2774           clsd                17660
Apr 24 15:06:36: %SYSLOG-0-EMERG: /netbsd: 2773           ppaslogd            7416
Apr 24 15:06:37: %SYSLOG-0-EMERG: /netbsd: 2772           sysmond             8944
Apr 24 15:06:37: %SYSLOG-0-EMERG: /netbsd: 2771           arpd                10908
Apr 24 15:06:37: %SYSLOG-0-EMERG: /netbsd: 2769           rpmd                9684
Apr 24 15:06:37: %SYSLOG-0-EMERG: /netbsd: 2768           vxxlatsvr           10204
Apr 24 15:06:37: %SYSLOG-0-EMERG: /netbsd: 2767           issud               7640
Apr 24 15:06:37: %SYSLOG-0-EMERG: /netbsd: 2766           ped_parse           8724
Apr 24 15:06:37: %SYSLOG-0-EMERG: /netbsd: 2765           ism2                43164
Apr 24 15:06:37: %SYSLOG-0-EMERG: /netbsd: 2764           rcm                 84748
Apr 24 15:06:37: %SYSLOG-0-EMERG: /netbsd: 2763           csm                 17592
Apr 24 15:06:37: %SYSLOG-0-EMERG: /netbsd: 2762           pnsd                8136
Apr 24 15:06:37: %SYSLOG-0-EMERG: /netbsd: 2761           evtmd               7788
Apr 24 15:06:37: %SYSLOG-0-EMERG: /netbsd: 2759           exec_cli            14408
Apr 24 15:06:37: %SYSLOG-0-EMERG: /netbsd: 2758           pm                  11808
Apr 24 15:06:37: %SYSLOG-0-EMERG: /netbsd: 2757           syslogd             2300
Apr 24 15:06:37: %SYSLOG-0-EMERG: /netbsd: 2756           metad               33776
Apr 24 15:06:38: %SYSLOG-0-EMERG: /netbsd: 2755           inetd               1280
Apr 24 15:06:38: %SYSLOG-0-EMERG: /netbsd: 2751           mount_udrv          520
Apr 24 15:06:38: %SYSLOG-0-EMERG: /netbsd: 2750           loggd               17152
Apr 24 15:06:38: %SYSLOG-0-EMERG: /netbsd: 11             mount_mfs           832
Apr 24 15:06:38: %SYSLOG-0-EMERG: /netbsd: 9              mount_mfs           552
Apr 24 15:06:38: %SYSLOG-0-EMERG: /netbsd: 7              ptdstat_thread      33800
Apr 24 15:06:38: %SYSLOG-0-EMERG: /netbsd: 6              reboot_thread       33800
Apr 24 15:06:38: %SYSLOG-0-EMERG: /netbsd: 5              evnt_th             33800
Apr 24 15:06:38: %SYSLOG-0-EMERG: /netbsd: 4              ioflush             33800
Apr 24 15:06:38: %SYSLOG-0-EMERG: /netbsd: 3              reaper              33800
Apr 24 15:06:38: %SYSLOG-0-EMERG: /netbsd: 2              pagedaemon          33800
Apr 24 15:06:38: %SYSLOG-0-EMERG: /netbsd: 1              init                344
Apr 24 15:06:38: %SYSLOG-0-EMERG: /netbsd: 0              swapper             33800
Apr 24 15:06:41: %PM-6-INFO: Call pm_start_process RESTART from pm_scan_pending_process_jobs for rib
Apr 24 15:06:41: %PM-5-GEN: restarting <rib> now
Apr 24 15:06:41: %PUBSUB-6-INF: RIB [rbnl_cache_initialize]: initializing PUBSUB, pid(2851)
Apr 24 15:06:41: %BGP-3-ERR: IPC error in RIB route query
Apr 24 15:06:52: %L2TP-6-TUNNEL: Dropping SCCRQ from TP35F1: Wrong local address
Apr 24 15:06:53: %L2TP-6-TUNNEL: Dropping SCCRQ from TP35F1: Wrong local address
Apr 24 15:07:05: %L2TP-6-TUNNEL: TPCFCD: Couldn't create peer for incoming SCCRQ
Apr 24 15:07:07: %L2TP-6-TUNNEL: TPCFCD: Couldn't create peer for incoming SCCRQ
Apr 24 15:07:32: %L2TP-6-TUNNEL: Dropping SCCRQ from TP35F1: Wrong local address
Apr 24 15:07:34: %L2TP-6-TUNNEL: Dropping SCCRQ from TP35F1: Wrong local address
Apr 24 15:07:46: %L2TP-6-TUNNEL: TPCFCD: Couldn't create peer for incoming SCCRQ
Apr 24 15:07:48: %L2TP-6-TUNNEL: TPCFCD: Couldn't create peer for incoming SCCRQ
Apr 24 15:08:14: %L2TP-6-TUNNEL: Dropping SCCRQ from TP35F1: Wrong local address
Apr 24 15:08:15: %L2TP-6-TUNNEL: Dropping SCCRQ from TP35F1: Wrong local address
Apr 24 15:08:27: %L2TP-6-TUNNEL: TPCFCD: Couldn't create peer for incoming SCCRQ
Apr 24 15:08:29: %L2TP-6-TUNNEL: TPCFCD: Couldn't create peer for incoming SCCRQ
Apr 24 15:08:54: %L2TP-6-TUNNEL: Dropping SCCRQ from TP35F1: Wrong local address
Apr 24 15:08:56: %L2TP-6-TUNNEL: Dropping SCCRQ from TP35F1: Wrong local address
Apr 24 15:09:07: [0003]: [2/2:511:63:31/7/2/447]: %AAA-6-STAT_CNTR_RCV: AAA index: 1342177715; Statistics counter is not received. Hence using old or zero counters
Apr 24 15:09:07: [2/2:511:63:31/7/2/447]: %QOS-3-RCM_ERR: Sub processing failed: unbind policy [ZAPRET-IPOE]: error code 16: error message: qos use error
Apr 24 15:09:08: %L2TP-6-TUNNEL: TPCFCD: Couldn't create peer for incoming SCCRQ
Apr 24 15:09:09: %L2TP-6-TUNNEL: TPCFCD: Couldn't create peer for incoming SCCRQ
Apr 24 15:09:34: %L2TP-6-TUNNEL: Dropping SCCRQ from TP35F1: Wrong local address
Apr 24 15:09:36: %L2TP-6-TUNNEL: Dropping SCCRQ from TP35F1: Wrong local address
Apr 24 15:09:49: %L2TP-6-TUNNEL: TPCFCD: Couldn't create peer for incoming SCCRQ
Apr 24 15:09:50: %L2TP-6-TUNNEL: TPCFCD: Couldn't create peer for incoming SCCRQ
Apr 24 15:10:15: %L2TP-6-TUNNEL: Dropping SCCRQ from TP35F1: Wrong local address
Apr 24 15:10:17: %L2TP-6-TUNNEL: Dropping SCCRQ from TP35F1: Wrong local address
Apr 24 15:10:29: %L2TP-6-TUNNEL: TPCFCD: Couldn't create peer for incoming SCCRQ
Apr 24 15:10:31: %L2TP-6-TUNNEL: TPCFCD: Couldn't create peer for incoming SCCRQ
Apr 24 15:10:56: %L2TP-6-TUNNEL: Dropping SCCRQ from TP35F1: Wrong local address
Apr 24 15:10:58: %L2TP-6-TUNNEL: Dropping SCCRQ from TP35F1: Wrong local address
Apr 24 15:11:10: %L2TP-6-TUNNEL: TPCFCD: Couldn't create peer for incoming SCCRQ
Apr 24 15:11:12: %L2TP-6-TUNNEL: TPCFCD: Couldn't create peer for incoming SCCRQ
Apr 24 15:11:37: %L2TP-6-TUNNEL: Dropping SCCRQ from TP35F1: Wrong local address
Apr 24 15:11:39: %L2TP-6-TUNNEL: Dropping SCCRQ from TP35F1: Wrong local address
Apr 24 15:11:51: %L2TP-6-TUNNEL: TPCFCD: Couldn't create peer for incoming SCCR
Apr 24 15:11:53: %L2TP-6-TUNNEL: TPCFCD: Couldn't create peer for incoming SCCRQ
Apr 24 15:12:18: %L2TP-6-TUNNEL: Dropping SCCRQ from TP35F1: Wrong local address
Apr 24 15:12:20: %L2TP-6-TUNNEL: Dropping SCCRQ from TP35F1: Wrong local address
Apr 24 15:12:32: %L2TP-6-TUNNEL: TPCFCD: Couldn't create peer for incoming SCCRQ

В чём может быть проблема?

shicoy · April 25, 2015

Согласно предоставленной информации, проблема в обновлении биллинга.

Ivan Rostovikov · April 25, 2015

Насколько я понимаю биллинг управляет железкой по радиусу.

Изучайте логи радиуса , если надо снифьте радиус трафик и ищите причину - аномалию.

Скорее всего дело в банальном увеличении например аккаунтинг пакетов.

Eagle18 · December 22, 2016

Столкнулся с такой же проблемой после тог окак прописал в конфиге команду aaa accounting subscriber radius. Неужели такие эвенты напрягают SE?

Nik0n · December 26, 2016

SE100 (SEOS-12.1.1.11-Release), стоит с самого начало эксплуатации:

aaa accounting subscriber radius
aaa update subscriber 10

Биллинг UTM5 - ни чего не напрягается, все работает штатно :)

snark · December 29, 2016

Столкнулся с такой же проблемой после того как прописал в конфиге команду aaa accounting subscriber radius

Аккаунтинг не грузит SE от слова совсем, о чем нам и поведал наш уважаемый Nik0n.

Одновременный рост нагрузки на aaad и rib говорит о том, что на SE был совершен массовый набег сабскриберов, который aaad пытался разрулить общаясь с радиусом, а rib распихивал роуты тех, кого радиус пустил (рост нагрузки у ТС на bgp из той же оперы).

Sign In

Проблемв с se100 после обновления bgbilling

Recommended Posts

clinkclinkclink

shicoy

Ivan Rostovikov

Eagle18

Nik0n

snark

Join the conversation