[Gray swordsman]

Имеется AS1004 c ESP40. Используется как ISG для IPoE сессий с политиками. В лог переодически стали падать сообщения типа

 

%SYS-2-NOBLOCK: printf with blocking disabled. -Process= "Collection process", ipl= 0, pid= 295
-Traceback= 1#e5e7d1f65502267ddf1ee514a5f253e4  :5574997BD000+10741BE1 :5574997BD000+FCC159A :5574997BD000+FCB71DA :5574997BD000+FCA350B :5574997BD000+7399983 :5574997B
D000+7323369 :5574997BD000+7327716 :5574997BD000+73DD316 :5574997BD000+73DD20B
 

обратил внимание что при это количество используемой DRAM на QFP подходит к 90%. Отключил nbar и flow. Используемая DRAM упала до 83%.

Можно как-то понять под что используется память ? вроде как железка достаточно мощная или это ее предел в ресурсах и надо думать о ее замене ?

 

На текущий момент загрузка

#show platform resources  
**State Acronym: H - Healthy, W - Warning, C - Critical                                              
Resource                 Usage                 Max             Warning         Critical        State
----------------------------------------------------------------------------------------------------
RP0 (ok, active)                                                                               H     
Control Processor       11.48%                100%            90%             95%             H     
 DRAM                   2810MB(35%)           7881MB          90%             95%             H     
ESP0(ok, active)                                                                               W     
Control Processor       3.80%                 100%            90%             95%             H     
 DRAM                   979MB(12%)            7881MB          90%             95%             H     
QFP                                                                                           W     
 TCAM                   3284cells(0%)         524288cells     45%             55%             H     
 DRAM                   871562KB(83%)         1048576KB       80%             90%             W     
 IRAM                   78683KB(60%)          131072KB        80%             90%             H     
SIP0                                                                                           H     
Control Processor       1.51%                 100%            90%             95%             H     
 DRAM                   390MB(41%)            943MB           90%             95%             H    

 

#sh platform software status control-processor brief  
Load Average
Slot  Status  1-Min  5-Min 15-Min
 RP0 Healthy   0.12   0.16   0.16
ESP0 Healthy   0.01   0.02   0.05
SIP0 Healthy   0.00   0.02   0.05

Memory (kB)
Slot  Status    Total     Used (Pct)     Free (Pct) Committed (Pct)
 RP0 Healthy  8070864  2877680 (36%)  5193184 (64%)   5679480 (70%)
ESP0 Healthy  8070840  1001992 (12%)  7068848 (88%)   1044864 (13%)
SIP0 Healthy   965852   400340 (41%)   565512 (59%)    499740 (52%)

CPU Utilization
Slot  CPU   User System   Nice   Idle    IRQ   SIRQ IOwait
 RP0    0   6.19   1.69   0.00  92.10   0.00   0.00   0.00
        1   2.69   2.39   0.00  94.70   0.00   0.19   0.00
ESP0    0   0.59   0.39   0.00  99.00   0.00   0.00   0.00
        1   0.49   0.49   0.00  99.00   0.00   0.00   0.00
SIP0    0   1.40   0.40   0.00  98.19   0.00   0.00   0.00

 

#sh platform  
Chassis type: ASR1004              

Slot      Type                State                 Insert time (ago)  
--------- ------------------- --------------------- -----------------  
0         ASR1000-SIP40       ok                    1y25w          
0/0      SPA-1X10GE-L-V2     ok                    1y25w          
0/1      SPA-1X10GE-L-V2     ok                    1y25w          
R0        ASR1000-RP2         ok, active            1y25w          
F0        ASR1000-ESP40       ok, active            1y25w          
P0        ASR1004-PWR-DC      ok                    1y25w          
P1        ASR1004-PWR-DC      ok                    1y25w          

Slot      CPLD Version        Firmware Version                         
--------- ------------------- ---------------------------------------  
0         00200900            16.3(2r)                             
R0        14111801            16.9(5r)                             
F0        1003190E            16.3(2r)                            

 

#sh platform hardware qfp active datapath utilization
 CPP 0: Subdev 0            5 secs        1 min        5 min       60 min
Input:  Priority (pps)          543          824          803          747
                (bps)      2755104      5525104      4736784      4475584
   Non-Priority (pps)      1699274      1733221      1758936      1820705
                (bps)  11219882112  11543321152  11756876408  12238681736
          Total (pps)      1699817      1734045      1759739      1821452
                (bps)  11222637216  11548846256  11761613192  12243157320
Output: Priority (pps)          329          331          329          334
                (bps)       280704       283024       281512       293952
   Non-Priority (pps)      1681379      1713848      1738803      1798674
                (bps)  11132425176  11435062448  11639925304  12099637192
          Total (pps)      1681708      1714179      1739132      1799008
                (bps)  11132705880  11435345472  11640206816  12099931144
Processing: Load (pct)           26           27           27           28

#sh subscriber statistics  

Current Subscriber Statistics:
Number of sessions currently up: 4111
Number of sessions currently pending: 0
Number of sessions currently authenticated: 4110
Number of sessions currently unauthenticated: 1
Highest number of sessions ever up at one time: 4364
Mean up-time duration of sessions: 18:51:19
Total number of sessions up so far: 2784870
Mean call rate per minute: 3, per hour: 212
Number of calls in last one hour: 175
Number of sessions failed to come up: 1659

Current Flow Statistics:
Number of flows currently up: 282
Highest number of flows ever up at one time: 1016
Mean up-time duration of flows: 02:47:53
Number of flows failed to come up: 0
Total number of flows up so far: 1344339

IP/DHCP session type count:
IPv4 sessions = 4108
IPv6 sessions = 3

Feature Installation Count:
                                   Direction
Feature Name              None       Inbound    Outbound   
QoS Policy Map            0          3956       3956       
Absolute Timeout          4100       0          0          
Idle Timeout              0          0          4100       
Per-User ACL              0          0          4097       
L4 Redirect               0          145        145        
Policing                  0          137        137        

Switch Id Cleanup Statistics:
Number of sessions having invalid SMGR handle: 0
Number of sessions having invalid policy handle: 0
Number of sessions having invalid LTERM handle: 0
Number of sessions having invalid SIP handle: 0

SHDBs in use    : 4111     
SHDBs allocated : 3171710  
SHDBs freed     : 3167599  

SHDB handles associated with each client type

Client Name     Count
===========    =======
LTerm          4111     
AAA            4111     
CCM            4111     
L2TP CC        0        
SSS FM         4101     
IPSUB          4111     
IP_IF          0        
COA            0        
COA cluster    0        
ISG Classifier 4111     
CCM Group      4111     
PM             4111     
PM cluster     0       

 

[BOJIKA]

sh run | i nat

sh ver

Edited March 6 by BOJIKA

[Gray swordsman]

Cisco IOS XE Software, Version 16.03.09
Cisco IOS Software [Denali], ASR1000 Software (X86_64_LINUX_IOSD-UNIVERSALK9_NOLI-M), Version 16.3.9, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Mon 29-Jul-19 13:42 by mcpre


Cisco IOS-XE software, Copyright (c) 2005-2019 by cisco Systems, Inc.
All rights reserved.  Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0.  The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY.  You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.  For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.


ROM: IOS-XE ROMMON

ISG-1 uptime is 1 year, 25 weeks, 6 days, 20 hours, 58 minutes
Uptime for this control processor is 1 year, 25 weeks, 6 days, 21 hours, 0 minutes
System returned to ROM by reload at 10:21:34 MSK Fri Dec 18 2020
System restarted at 16:25:04 MSK Wed Sep 6 2023
System image file is "bootflash:/asr1000rpx86-universalk9_noli.16.03.09.SPA.bin"
Last reload reason: PowerOn



This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

License Type: RightToUse
License Level: advipservices
Next reload license Level: advipservices

cisco ASR1004 (RP2) processor (revision RP2) with 4128129K/6147K bytes of memory.
Processor board ID FOX1115GYPQ
2 Ten Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
8388608K bytes of physical memory.
1884159K bytes of eUSB flash at bootflash:.
78085207K bytes of SATA hard disk at harddisk:.
         


NAТ не используется

[BOJIKA]

sh platform hardware qfp active infrastructure exmem statistics user

тут посмотрите, какой процесс больше съел

[Gray swordsman]

#sh platform hardware qfp active infrastructure exmem statistics user  | i CVLA  
 17           69639072     69649408     CVLA                                       
 158          602419984    602520576    CVLA  
нижний на порядок больше всех остальных.

 

[BOJIKA]

https://bst.cisco.com/quickview/bug/CSCus59403
попробуйте обновиться...

 

например 16.12.8

[Gray swordsman]

да, спасибо, по вашей наводке наглилось и это тоже  CSCud60676, NBAR и netflow явно фигурирует в обоих случаях. а с апдейтом не вариант, насколько я помню наша последняя ветка с RTU лицензией. для начала попробую ребутнуть. если не вылечится буду думать.

 

[sirmax]

3 минуты назад, Gray swordsman сказал:

да, спасибо, по вашей наводке наглилось и это тоже  CSCud60676, NBAR и netflow явно фигурирует в обоих случаях. а с апдейтом не вариант, насколько я помню наша последняя ветка с RTU лицензией. для начала попробую ребутнуть. если не вылечится буду думать.

 

А подробнее про RTU? 
Я несколько раз такое сылшал но вот прям "без пруфов" как говорится

И слышал и обратное тоже 

[Gray swordsman]

тысячник ставили до ковида, как оно там точно было я уже не помню, но  если память не изменяет все старшие версии хотели лицензию.

сейчас имею

#sh license  
Index 1 Feature: adventerprise                   
       Period left: 8  weeks 3  days  
       Period Used: 53 minutes 4  seconds  
       License Type: EvalRightToUse
       License State: Active, Not in Use, EULA accepted
       License Count: Non-Counted
       License Priority: Low
Index 2 Feature: advipservices                   
       Period left: Life time
       License Type: RightToUse
       License State: Active, In Use
       License Count: Non-Counted
       License Priority: Low
Index 3 Feature: ipbase                         
<skip>

 

 

[BOJIKA]

там всё RTU. ничего не слетит...

 

bras#sh ver | i 16.12|uptime
Cisco IOS XE Software, Version 16.12.08
Cisco IOS Software [Gibraltar], ASR1000 Software (X86_64_LINUX_IOSD-UNIVERSALK9_NPE_NOLI-M), Version 16.12.8, RELEASE SOFTWARE (fc1)
bras uptime is 1 year, 24 weeks, 5 days, 15 hours, 59 minutes
System image file is "bootflash:/asr1000rpx86-universalk9_npe_noli.16.12.08.SPA.bin"
bras#sh sss statistics | i Number of sessions currently up
Number of sessions currently up: 36762
bras#sh license usage
License Authorization:
  Status: EVAL EXPIRED on Jul 23 01:04:14 2020 MSK

(ASR_1000_advipservicesk9_npe_noli):
  Description:
  Count: 1
  Version: 1.0
  Status: EVAL EXPIRED
  Export status: NOT RESTRICTED

(ASR_1000_BroadBand_2K):
  Description:
  Count: 19
  Version: 1.0
  Status: EVAL EXPIRED
  Export status: NOT RESTRICTED

 

[sirmax]

А 17-я ветка тоже?

[Gray swordsman]

о как. ок, поищу данный IOS. попробую поднять версию.

[fractal]

На asr всегда было rtu, у нас 1001hx с 17.12.04, все лицухи есть, включаю что хочу

[Gray swordsman]

Смог найти только asr1000rpx86-universalk9_noli.16.12.04.SPA.bin. Если у кого есть возможность поделитесь в личку asr1000rpx86-universalk9_npe_noli.16.12.08.SPA.bin.
 

[BOJIKA]

https://cios.dhitechnical.com/ASR_9XX_1XXX/ASR1006_1009-X/

логин пароль в html коде ссылки

[zhenya`]

А что у вас за ацлы на сессиях?