SCE2020 - SE Watchdog Module: An Error occurred - Активное оборудование Ethernet, IP, MPLS, SDN/NFV...

Sapiens

Posted July 1, 2013

Добрый день.

Перезагружается SCE2020 и выпадает в recovery mode. Перепрошивал по инструкции http://shop.nag.ru/article/pereustanovka-operatsionnoj-sistemy-i-vosstanovlenie-cisco-sce, устанавливал новый софт (pkg, pqi), сбрасывал конфиг - ничего не помогает.

Прописал watchdog software-reset disabled и no watchdog - перезагружаться перестала. Но при подключении оптических линков уходит в failure и останавливает траффик процессор. При этом растут ошибки errors и warnings. В логе пишет:

2013-07-01 05:53:37 | ERROR | CPU #000 | SE Watchdog Module: An Error occurred. Please report to Cisco's customer support

2013-07-01 05:53:37 | FATAL | CPU #000 | SE Self Sanity Checks Module: A Fatal Error occurred. Please report to Cisco's customer support

2013-07-01 05:53:37 | WARN | CPU #000 | The SE status changed to Failure due to fatal event

2013-07-01 05:53:37 | INFO | CPU #000 | trap:failure status

2013-07-01 05:54:18 | WARN | CPU #000 | SE Watchdog Module: A problem occurred. Please report to Cisco's customer support

SCE2000#

SCE2000#sh ver

System version: Version 3.8.5-classic Build 967

Build time: Dec 1 2012, 13:41:41 (Change-list 716021)

Software version is: Version 3.8.5-classic Build 967

Cryptography class: K9

Hardware information is:

rx : 0x0079

dp : 0x1d08

tx : 0x1712

ff : 0x0077

cls : 0x1e05

cpld : 0x0025

lic : 0x0202

lic-drv : 0x02020000

lic-prm : 0x02020000

rev : G001

Bootrom : 2.1.0

L2 cache : Samsung 0.5

lic type : 4GBE

optic mode : MM

Product S/N : CAT1202G0EW

Product ID : SCE2020-4XGBE-MM

Version ID : V03

Deviation :

Part number : 800-26601-07

Revision : A0

Software revision : G001

LineCard S/N : CAT1204G027

Power Supply type : AC

SML Application information is:

Application file: /tffs0/release_.sli

Application name: Engage SML Version 3.8.5 build 55 (Protocol Pack 33 build 23)

Using Lib - PL_V3.8.5PP_B42

Using Lib - Classifier_V3.8.5_B42

Application help: Entry point of Engage

Original source file: /auto/srbu-proj1-bgl/apps/users/gnbalasu/update/App/SML/Engage/V3.8.5/src/com/pcube/AppTemplate/Main/template_app_main.san

Compilation date: Fri, February 08, 2013 at 19:11:10

Compiler version: SANc v3.20 Build 14 built on: Tue 08/04/2009 06:58:22.;SME plugin v1.1

Capacity option used: 'EngageDefaultSCE2000'.

Logger status: Enabled

Platform: SCE2000 - 4xGBE

Management agent interface version: SCE Agent 3.8.5 Build 82

Software package file: /tffs0/images/SEOS.PKG

SCE2000 uptime is 1 hours, 17 minutes, 9 seconds

SCE2000#

SCE2000#sh run

#This is a general configuration file (running-config).

#Created on 08:20:37 UTC MON July 1 2013

#cli-type 1

#version 1

watchdog software-reset disabled

enable password level 5 0 "cisco"

enable password level 10 0 "cisco"

enable password level 15 0 "cisco"

no management-agent notifications notification-list 1417,1418,804,815,1404,1405,1406,1407,1408,400

no management-agent notifications notification-list 402,421,440,441,444,445,446,450,437,457

no management-agent notifications notification-list 3593,3594,3595,10040,4614,4615

snmp-server community "getsce" ro

snmp-server community "setsce" rw

interface LineCard 0

aggregative-global-controllers

aggregative-global-controller-dynamic mode suspend

aggregative-global-controller-dynamic mode active

connection-mode inline on-failure bypass

no watchdog

no silent

shutdown

service-bandwidth-prioritization-mode global

protocol-pack version 3.8.5PP33

flow-filter partition name "ignore_filter" first-rule 4 num-rules 32

flow-filter partition name "udpPortsToOpenBySw" first-rule 40 num-rules 21

attack-filter subscriber-notification ports 80

replace spare-memory code bytes 3145728

interface FastEthernet 0/0

ip address 192.168.55.11 255.255.255.0

speed 100

duplex full

interface FastEthernet 0/1

ip address 192.168.55.11 255.255.255.0

speed 100

duplex full

interface GigabitEthernet 0/1

interface GigabitEthernet 0/2

interface GigabitEthernet 0/3

interface GigabitEthernet 0/4

exit

ip default-gateway 192.168.55.1

line vty 0 4

exit

interface Mng 0/1

speed 100

duplex full

exit

interface Mng 0/2

speed 100

duplex full

exit

no subscriber LEG dhcp-lease-query

subscriber LEG dhcp-lease-query servers 127.0.0.1

logger device SCE-agent-Statistics-Log max-file-size 204800

management-agent property "com.pcube.management.framework.install.activation.operation" "Install"

management-agent property "com.pcube.management.framework.install.activated.package" "SCA BB"

management-agent property "com.pcube.management.framework.install.activated.version" "3.8.5 build 53"

management-agent property "com.pcube.management.framework.install.activation.date" "Mon Jul 01 03:21:02 UTC 2013"

SCE2000#

Подозрение на проблемы с флешем или железом. Наверняка в tech support info file можно нарыть что-то поподробнее но куда там смотреть непонятно.

tech support info file:

http://my-files.ru/oxiw.tech0701.zip

Share this post

Link to post

Share on other sites

murzik_one

Posted July 1, 2013

продуйте воздухом глаза у оптических модулей и поменяйте шнурки оптические. мне в свое время помогло...

при чем на мультимодовых шнурках заводиться вообще не хотела (хотя в обоих сторон MM модули), на одномодовых взлетела как стриж...

Share this post

Link to post

Share on other sites

Sapiens

Posted July 1, 2013

Продувал внутри - платы не раскручивал, просто снял корпус и почистил, продул.

А шнурки оптические поменять Вы имеете в виду оптические патчкорды или пигтейлы которые внутри sce?

Насчет одномодовых - сейчас попробую..

Share this post

Link to post

Share on other sites

murzik_one

Posted July 1, 2013

Продувал внутри - платы не раскручивал, просто снял корпус и почистил, продул.

А шнурки оптические поменять Вы имеете в виду оптические патчкорды или пигтейлы которые внутри sce?

Насчет одномодовых - сейчас попробую..

не. я так глубоко не лазил. хватило просто в оптические дырки на лицевой панели сжатым воздухом подуть =)

шнурки. да, имел ввиду оптические патчкорды. между SCE и оборудование куда оно включено.

Share this post

Link to post

Share on other sites

Sapiens

Posted July 1, 2013

В общем продул розетки на sce. Поменял sfp-ки, поменял патчкорды (правда на многомодовые, одномодовых с такими штекерами нет) Та же картина :(

Господа, кто знающий посмотрите все таки tech support info file может там будет понятно, где проблема.

Share this post

Link to post

Share on other sites

Butch3r

Posted July 1, 2013

Другую версию софта пробовали ставиь?

Share this post

Link to post

Share on other sites

Sapiens

Posted July 2, 2013

Да, другую версию тоже пробовал.

А кто-нибудь знает, можно ли вообще проверить флешку, ну или отформатировать ее (и можно ли будет тогда на нее снова залить весь необходимый софт)?

Share this post

Link to post

Share on other sites

max1976

Posted July 2, 2013 (edited)

Посмотрите debug файлы, которые находятся в каталоге cd /apps/data/scos/system/p3hidden/log/dbg . Поищите в этих файлах описание ошибки, из-за которой происходит ребут.

На своей SCE 8000 я нашел причину, по которой происходил ребут:

Interrupt Received from: NalaDPT[0].Tx[0].CTX_ISR.ECCSingleErrorInt - description: RLDRAM ECC single bit error interrupt   - count: 102
Interrupt Received from: NalaDPT[0].Tx[0].CTX_ISR.ECCMultiErrorInt - description: RLDRAM ECC multi bit error interrupt

Edited July 2, 2013 by max1976

Share this post

Link to post

Share on other sites

Sapiens

Posted July 3, 2013

У меня на sce2020 в папке /tffs0/app/ лежит только EN38553.SLI

Порылся в файле prevdbg.txt есть такое:

07/01/13 02:01:52 [000000244348:899:419] | 000 | 0000038130 | 0000000 | <WARNING> [0x0910:0x0022] RuC Interrupt Handler: Following Interrupt details. Device: 2, Info :

No additional info

07/01/13 02:01:52 [000000244348:899:434] | 000 | 0000038131 | 0000000 | <WARNING> [0x0910:0x0024] RuC Interrupt Handler: CLS Interrupt occured. Status: 0x2

07/01/13 02:01:52 [000000244348:899:437] | 000 | 0000038132 | 0000000 | <WARNING> [0x0910:0x0019] RuC Interrupt Handler: Classifier Interrupt - Free flow PPC #2 queue Empty

07/01/13 02:01:52 [000000244258:021:064] | 002 | 0000122119 | 0000000 | <<ERROR>> [0x0b00:0x0007] TestPacketManager: inject operation failed - port=1, TxRxStatus=3, injectCount=0

07/01/13 02:01:52 [000000244258:021:105] | 002 | 0000122120 | 0000000 | <<ERROR>> [0x0a10:0x0000] System Message: Dump message:

No current valid FC ID.

No current valid Host.

No current node.

No previous node.

No payload loaded

RUC Scheduler subtask:

RucMain[0].AllCPUsSubtasksTable[0].ExternalMessageHandler {GUID 64}

TraceLevel = 0

RucMain[0].AllCPUsSubtasksTable[0].ExternalMessageHandler Profiling: Last reset was 244220 seconds ago.

[measures: 142240640| rate: 582.43| avg: 1.06| min: 0.60| max: 3894572.50| fails: 0]

isEnabled: yes

Milliseconds to wait: 0

myRefillQuotaBeforeRun: no Quota: myInitialQuota = 100 microsec, myCurrentQuota = 99 microsec

myTargetTime = 3301985844 ticks, time to target = 171798128 microsec

myIsRunning = yes

и в файле umlog такое:

01-Jul 2013 07:03:40.509 GMT+00:00 [main] FATAL com.cisco.scasbb.agents.mbean.sce.ScasbbSubscriberOperationHandler - learnAppOperationIndices: Failed to get quota native funation indeces

com.pcube.management.common.WrapperException: com.pcube.management.framework.functionality.NativeMethodStatusFailException: Error Code = 8, Description: "Could not find app variable 'P_quotaGrantHandler' in '' table.", Detailed: "" (HND_getHandlerId:{name=P_quotaGrantHandler, task=0})

at com.pcube.management.api.common.ProxyHandlerApiDispatcher.invoke(Ljavax/management/ObjectName;Ljava/lang/String;[Ljava/lang/Object;[Ljava/lang/String;)Ljava/lang/Object; [bytecode offset 29]

at com.pcube.management.api.common.BaseApi.invoke(Ljavax/management/ObjectName;Ljava/lang/String;[Ljava/lang/Object;[Ljava/lang/String;)Ljava/lang/Object; [bytecode offset 9]

at com.pcube.management.api.se.PartyApi.getHandlerId(JLjava/lang/String;)Ljava/lang/Object; [bytecode offset 56]

at com.pcube.management.api.se.SEApiImpl.getHandlerId(JLjava/lang/String;)Ljava/lang/Object; [bytecode offset 6]

at com.cisco.scasbb.agents.mbean.sce.SubQuotaOperationHandler.learnAppOperationIndices()V [bytecode offset 35]

at com.cisco.scasbb.agents.mbean.sce.SubQuotaOperationHandler.<init>(Lcom/sun/jdmk/ProxyHandler;)V [bytecode offset 123]

at com.scms.sce.SCESubscriberApiMBean.postAllRegister()V [bytecode offset 49]

at java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; [native]

at com.pcube.management.framework.jdmk.ExtendedMBean.invokeMethod(Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object; [bytecode offset 3]

at com.pcube.management.framework.jdmk.ExtendedMBean.invoke(Ljava/lang/String;[Ljava/lang/Object;[Ljava/lang/String;)Ljava/lang/Object; [bytecode offset 230]

at com.sun.jdmk.MBeanServerImpl.invoke(Ljava/lang/Object;Ljava/lang/String;[Ljava/lang/Object;[Ljava/lang/String;)Ljava/lang/Object; [bytecode offset 99]

at com.sun.jdmk.MBeanServerImpl.invoke(Ljavax/management/ObjectName;Ljava/lang/String;[Ljava/lang/Object;[Ljava/lang/String;)Ljava/lang/Object; [bytecode offset 14]

at com.pcube.management.framework.jdmk.MBeanInstantiateHelper.postAllRegister()V [bytecode offset 62]

at com.pcube.management.framework.agent.NetworkEntityAgent.startupMBeans(Ljava/lang/String;)V [bytecode offset 379]

at com.pcube.management.framework.agent.NetworkEntityAgent.startupAgent()V [bytecode offset 63]

at com.pcube.management.framework.agent.NetworkEntityAgent.main([Ljava/lang/String;)V [bytecode offset 62]

at com.pcube.management.StartSEImpl.main([Ljava/lang/String;)V [bytecode offset 730]

at StartSE.main([Ljava/lang/String;)V [bytecode offset 1]

*** Wrapped Exception (wrapping level 1)

javax.management.MBeanException: com.pcube.management.framework.functionality.NativeMethodStatusFailException: Error Code = 8, Description: "Could not find app variable 'P_quotaGrantHandler' in '' table.", Detailed: "" (HND_getHandlerId:{name=P_quotaGrantHandler, task=0})

at com.pcube.management.framework.functionality.FunctionalityMBean.internalInvoke(Ljava/lang/String;[Ljava/lang/Object;[Ljava/lang/String;JLjava/lang/String;J)Ljava/lang/Object; [bytecode offset 73]

at com.pcube.management.framework.functionality.FunctionalityMBean.invoke(Ljava/lang/String;[Ljava/lang/Object;[Ljava/lang/String;)Ljava/lang/Object; [bytecode offset 65]

at com.sun.jdmk.MBeanServerImpl.invoke(Ljava/lang/Object;Ljava/lang/String;[Ljava/lang/Object;[Ljava/lang/String;)Ljava/lang/Object; [bytecode offset 99]

at com.sun.jdmk.MBeanServerImpl.invoke(Ljavax/management/ObjectName;Ljava/lang/String;[Ljava/lang/Object;[Ljava/lang/String;)Ljava/lang/Object; [bytecode offset 14]

at com.sun.jdmk.MBeanServerForwarder.invoke(Ljavax/management/ObjectName;Ljava/lang/String;[Ljava/lang/Object;[Ljava/lang/String;)Ljava/lang/Object; [bytecode offset 9]

at com.pcube.management.framework.agent.info.MBeanServerTraceInfo.invoke(Ljavax/management/ObjectName;Ljava/lang/String;[Ljava/lang/Object;[Ljava/lang/String;)Ljava/lang/Object; [bytecode offset 44]

at com.pcube.management.api.common.ProxyHandlerApiDispatcher.invoke(Ljavax/management/ObjectName;Ljava/lang/String;[Ljava/lang/Object;[Ljava/lang/String;)Ljava/lang/Object; [bytecode offset 9]