Jump to content

Cat 6506 повышеная загрузка цпу

Ivan Rostovikov
 Share

Recommended Posts

Ivan Rostovikov

Ivan Rostovikov

Posted
Posted

Помогите диагностировать проблему:

6506 используется в качестве ядра сети.

Собирает от NAS-ов по RIP примерно 3000 маршрутов.

Суммарный трафик по ним не превышает 1Гб.

Однако загрузка CPU :

 

CPU utilization for five seconds: 63%/61%; one minute: 63%; five minutes: 63%

Немогу понять где источник такой загрузки. Ведь по идее 3K маршрутов и 1Г трафика дла него - "как 2 пальца". Откуда 60 %% ???

 

 

System image file is "sup-bootdisk:s72033-adventerprisek9_wan-mz.122-33.SXI2.bin

cisco WS-C6506-E (R7000) processor (revision 1.2) with 458720K/65536K bytes of memory.

Processor board ID SAL1323RQZH

SR71000 CPU at 600Mhz, Implementation 0x504, Rev 1.2, 512KB L2 Cache

Last reset from s/w reset

52 Virtual Ethernet interfaces

50 Gigabit Ethernet interfaces

1917K bytes of non-volatile configuration memory.

8192K bytes of packet buffer memory.

 

#show module all

Mod Ports Card Type Model Serial No.

--- ----- -------------------------------------- ------------------ -----------

3 48 CEF720 48 port 1000mb SFP WS-X6748-SFP SAD083704PL

6 2 Supervisor Engine 720 (Active) WS-SUP720-3B SAL1130VGY0

 

Mod MAC addresses Hw Fw Sw Status

--- ---------------------------------- ------ ------------ ------------ -------

3 000b.fcf8.1790 to 000b.fcf8.17bf 1.1 12.2(18r)S1 12.2(33)SXI2 Ok

6 001a.2f3c.0d8c to 001a.2f3c.0d8f 5.4 8.4(2) 12.2(33)SXI2 Ok

 

Mod Sub-Module Model Serial Hw Status

---- --------------------------- ------------------ ----------- ------- -------

3 Centralized Forwarding Card WS-F6700-CFC SAL1327T7YX 4.1 Ok

6 Policy Feature Card 3 WS-F6K-PFC3B SAL1130VZ46 2.3 Ok

6 MSFC3 Daughterboard WS-SUP720 SAL1111JVQV 3.0 Ok

 

Mod Online Diag Status

---- -------------------

3 Pass

6 Pass

 

Ivan Rostovikov

Ivan Rostovikov

Posted
  • Author
Posted

В моем случае смысл есть. Но суть не в этом

Сам по себе RIP проблемы не делает. Проблему делает трафик. Если не направлять трафик по его маршрутам - все ок.

 

CPU utilization for five seconds: 61%/59%; one minute: 60%; five minutes: 62%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

273 1362060 4749231 286 0.95% 1.59% 1.59% 0 IP Input

494 1970020 13434428 146 0.55% 0.31% 0.32% 0 Port manager per

280 516580 167925 3076 0.31% 0.18% 0.20% 0 RIP Timers

24 1197300 11816633 101 0.23% 0.06% 0.06% 0 IPC Seat Manager

311 219216 96682 2267 0.15% 0.04% 0.04% 0 QOS Stats Gather

121 324 104 3115 0.07% 0.02% 0.01% 2 SSH Process

7 0 1 0 0.00% 0.00% 0.00% 0 PF Redun ICC Req

6 0 3 0 0.00% 0.00% 0.00% 0 IPC ISSU Dispatc

5 0 33 0 0.00% 0.00% 0.00% 0 Retransmission o

10 220 100 2200 0.00% 0.00% 0.00% 0 Pool Manager

8 0 1 0 0.00% 0.00% 0.00% 0 PF Redun ICC Req

9 1598872 97301 16432 0.00% 0.23% 0.28% 0 Check heaps

13 0 1 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT

11 0 2 0 0.00% 0.00% 0.00% 0 Timers

15 0 1 0 0.00% 0.00% 0.00% 0 Policy Manager

16 100 264 378 0.00% 0.00% 0.00% 0 Entity MIB API

17 20 112 178 0.00% 0.00% 0.00% 0 EEM ED Syslog

4 0 1 0 0.00% 0.00% 0.00% 0 Connection Mgr

19 28 8080 3 0.00% 0.00% 0.00% 0 IPC Dynamic Cach

12 162256 415941 390 0.00% 0.08% 0.07% 0 ARP Input

14 0 2 0 0.00% 0.00% 0.00% 0 AAA high-capacit

22 480 484202 0 0.00% 0.00% 0.00% 0 IPC Deferred Por

23 0 1 0 0.00% 0.00% 0.00% 0 IPC Process leve

18 0 1 0 0.00% 0.00% 0.00% 0 IFS Agent Manage

25 0 1 0 0.00% 0.00% 0.00% 0 IPC Session Serv

26 0 1 0 0.00% 0.00% 0.00% 0 IPC Stdby Update

27 0 2 0 0.00% 0.00% 0.00% 0 DDR Timers

28 0 2 0 0.00% 0.00% 0.00% 0 Dialer event

29 0 1 0 0.00% 0.00% 0.00% 0 ifIndex Receive

30 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun

31 0 1 0 0.00% 0.00% 0.00% 0 Crash writer

32 27528 175240 157 0.00% 0.01% 0.00% 0 EnvMon

20 0 1 0 0.00% 0.00% 0.00% 0 NTI Example Proc

34 32 48498 0 0.00% 0.00% 0.00% 0 Compute SRP rate

3 28 256 109 0.00% 0.00% 0.00% 0 SSH Event handle

36 108 1003 107 0.00% 0.00% 0.00% 0 rf proxy rp agen

21 4708 484196 9 0.00% 0.00% 0.00% 0 IPC Periodic Tim

38 4 3 1333 0.00% 0.00% 0.00% 0 client_entity_se

33 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version

40 0 1 0 0.00% 0.00% 0.00% 0 EM Action CNS

41 0 2 0 0.00% 0.00% 0.00% 0 Snmp ICC Process

42 4 61 65 0.00% 0.00% 0.00% 0 Cat6k SNMP

43 8 196 40 0.00% 0.00% 0.00% 0 Cat6k SNMP Trap

44 0 1 0 0.00% 0.00% 0.00% 0 ARP Snoop

45 716 484196 1 0.00% 0.00% 0.00% 0 Dynamic ARP Insp

46 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd

47 360 53422 6 0.00% 0.00% 0.00% 0 Net Background

48 0 3 0 0.00% 0.00% 0.00% 0 IDB Work

49 0 180 0 0.00% 0.00% 0.00% 0 Logger

50 804 484164 1 0.00% 0.00% 0.00% 0 TTY Background

51 26412 484476 54 0.00% 0.00% 0.00% 0 Per-Second Jobs

52 339940 8501 39988 0.00% 0.08% 0.06% 0 Per-minute Jobs

53 4 55 72 0.00% 0.00% 0.00% 0 Cat6k NTI ICC pr

54 44 249 176 0.00% 0.00% 0.00% 0 IF-MGR control p

55 0 174 0 0.00% 0.00% 0.00% 0 IF-MGR event pro

56 24 398 60 0.00% 0.00% 0.00% 0 Env action

57 0 2 0 0.00% 0.00% 0.00% 0 Flash ICC Server

58 4 809 4 0.00% 0.00% 0.00% 0 foundation

59 0 1 0 0.00% 0.00% 0.00% 0 VS Redirect EOBC

60 0 1 0 0.00% 0.00% 0.00% 0 VSIBC process

2 680 96983 7 0.00% 0.00% 0.00% 0 Load Meter

37 0 3 0 0.00% 0.00% 0.00% 0 rf proxy message

63 0 1 0 0.00% 0.00% 0.00% 0 EOBC Monitor Pro

64 1344 43958 30 0.00% 0.00% 0.00% 0 Net Input

65 4 7 571 0.00% 0.00% 0.00% 0 draco-oir-proces

 

 

#show ip cef summary

IPv4 CEF is enabled for distributed and running

VRF base:

3149 prefixes (3149/0 fwd/non-fwd)

Default network 0.0.0.0/0

Table id 0

Database epoch: 4 (3149 entries at this epoch)

 

Ivan Rostovikov

Ivan Rostovikov

Posted
  • Author
Posted

Конфиг простейший.

интерфейсы и вланы не привожу. больно много.

Что показывать в логах ?

 

 

mls netflow interface

mls qos

mls cef error action reset

!

spanning-tree mode pvst

spanning-tree loopguard default

spanning-tree extend system-id

no spanning-tree vlan 1-4000

diagnostic bootup level minimal

port-channel load-balance src-dst-mac

!

vlan internal allocation policy ascending

vlan access-log ratelimit 2000

interface Port-channel5

switchport

switchport trunk encapsulation dot1q

switchport mode trunk

!

interface GigabitEthernet3/1

switchport

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 14,200-205

switchport mode trunk

...

...

...

router rip

version 2

timers basic 30 60 60 120

redistribute connected

redistribute static

passive-interface default

network 123.123.123.0

no auto-summary

 

ip route 0.0.0.0 0.0.0.0 123.123.123.1

 

no ip http server

no ip http secure-server

!

access-list 111 remark

...

...

 

Собственно и все.

 

Ivan Rostovikov

Ivan Rostovikov

Posted
  • Author
Posted

Попробовал убрать RIP. Загнал все в 4 статичных маршрута. Обьем трафика остался прежний. Загрузка не изменилась. :-(

 

#show ip cef summary

IPv4 CEF is enabled for distributed and running

VRF base:

1203 prefixes (1203/0 fwd/non-fwd)

Default network 0.0.0.0/0

Table id 0

Database epoch: 4 (1203 entries at this epoch)

 

 

Ivan Rostovikov

Ivan Rostovikov

Posted
  • Author
Posted

Это я как раз понял. Но откуда им взяться ? Трафик то небольшой.!!!

 

 

#show platform hardware capacity

System Resources

PFC operating mode: PFC3B

Supervisor redundancy mode: administratively sso, operationally sso

Switching resources: Module Part number Series CEF mode

3 WS-X6748-SFP CEF720 CEF

6 WS-SUP720-3B supervisor CEF

 

Power Resources

Power supply redundancy mode: administratively redundant

operationally non-redundant (single power supply)

System power: 2771W, 0W (0%) inline, 918W (33%) total allocated

Powered devices: 0 total, 0 Class3, 0 Class2, 0 Class1, 0 Class0, 0 Cisco

 

Flash/NVRAM Resources

Usage: Module Device Bytes: Total Used %Used

3 dfc#3-bootflash: 15990784 0 0%

6 RP bootflash: 65536000 0 0%

6 SP sup-bootdisk: 512073728 267419648 52%

6 SP const_nvram: 129004 5176 4%

6 SP nvram: 1964024 25251 1%

 

CPU Resources

CPU utilization: Module 5 seconds 1 minute 5 minutes

3 32% / 0% 32% 32%

6 RP 43% / 42% 44% 45%

6 SP 10% / 1% 12% 12%

Processor memory: Module Bytes: Total Used %Used

3 206486208 40983700 20%

6 RP 360597552 94502984 26%

6 SP 348677840 110418820 32%

I/O memory: Module Bytes: Total Used %Used

6 RP 67108864 16487652 25%

6 SP 67108864 15705560 23%

 

EOBC Resources

Module Packets/sec Total packets Dropped packets

3 Rx: 67 94230102 0

Tx: 61 25762372 3

6 RP Rx: 65 44530591 0

Tx: 65 44394026 0

6 SP Rx: 6 20800805 0

Tx: 14 24115782 0

 

VLAN Resources

VLANs: 4094 total, 82 VTP, 0 extended, 13 internal, 3999 free

 

L2 Forwarding Resources

MAC Table usage: Module Collisions Total Used %Used

6 0 65536 3040 5%

 

VPN CAM usage: Total Used %Used

512 0 0%

L3 Forwarding Resources

FIB TCAM usage: Total Used %Used

72 bits (IPv4, MPLS, EoM) 196608 2915 1%

144 bits (IP mcast, IPv6) 32768 8 1%

 

detail: Protocol Used %Used

IPv4 2913 1%

MPLS 1 1%

EoM 1 1%

 

IPv6 1 1%

IPv4 mcast 4 1%

IPv6 mcast 3 1%

 

Adjacency usage: Total Used %Used

1048576 1078 1%

 

Forwarding engine load:

Module pps peak-pps peak-time

6 265973 840099 18:58:38 UTC Fri Nov 6 2009

 

Netflow Resources

TCAM utilization: Module Created Failed %Used

6 1 0 0%

ICAM utilization: Module Created Failed %Used

6 0 0 0%

 

Flowmasks: Mask# Type Features

IPv4: 0 reserved none

IPv4: 1 unused none

IPv4: 2 unused none

IPv4: 3 reserved none

 

IPv6: 0 reserved none

IPv6: 1 unused none

IPv6: 2 unused none

IPv6: 3 reserved none

 

CPU Rate Limiters Resources

Rate limiters: Total Used Reserved %Used

Layer 3 9 4 1 44%

Layer 2 5 3 3 60%

 

ACL/QoS TCAM Resources

Key: ACLent - ACL TCAM entries, ACLmsk - ACL TCAM masks, AND - ANDOR,

QoSent - QoS TCAM entries, QOSmsk - QoS TCAM masks, OR - ORAND,

Lbl-in - ingress label, Lbl-eg - egress label, LOUsrc - LOU source,

LOUdst - LOU destination, ADJ - ACL adjacency

 

Module ACLent ACLmsk QoSent QoSmsk Lbl-in Lbl-eg LOUsrc LOUdst AND OR ADJ

6 1% 3% 1% 1% 1% 1% 2% 2% 0% 0% 1%

 

L3 Multicast Resources

IPv4 replication mode: ingress

IPv6 replication mode: egress

Bi-directional PIM Designated Forwarder Table usage: 4 total, 0 (0%) used

Replication capability: Module IPv4 IPv6

3 egress egress

6 egress egress

MET table Entries: Module Total Used %Used

6 65516 6 1%

 

QoS Policer Resources

Aggregate policers: Module Total Used %Used

6 1024 1 1%

Microflow policer configurations: Module Total Used %Used

6 64 1 1%

 

Switch Fabric Resources

Bus utilization: current: 0%, peak was 3% at 23:06:03 MSK Wed Nov 11 2009

Fabric utilization: Ingress Egress

Module Chanl Speed rate peak rate peak

3 0 20G 0% 6% @13:56 11Nov09 1% 5% @22:31 11Nov09

3 1 20G 3% 8% @22:32 11Nov09 3% 9% @20:41 11Nov09

6 0 20G 2% 5% @00:31 12Nov09 2% 5% @01:33 12Nov09

Switching mode: Module Switching mode

3 acef

6 dcef

 

Interface Resources

Interface drops:

Module Total drops: Tx Rx Highest drop port: Tx Rx

no modules reported drops

 

Interface buffer sizes:

Module Bytes: Tx buffer Rx buffer

3 (asic-1) 1221120 173504

IBC Resources

Module Packets/sec Total packets Dropped packets

6 RP Rx: 76442 5594083367 0

Tx: 76431 5591601504 0

6 SP Rx: 64 39097190 0

Tx: 233 135237716 0

 

SPAN Resources

Source sessions: 16 maximum, 0 used

Type Max Used

Local 2(*) 0

Local-tx 14 0

RSPAN source 2(*) 0

ERSPAN source 2(*) 0

Capture 1(*) 0

Service module 1(*) 0

OAM loopback 1(*) 0

* - shared source sessions and the total can not exceed 2

Destination sessions: 64 maximum, 0 used

Type Max Used

RSPAN destination 64(*) 0

ERSPAN destination 23(*) 0

* - shared destination sessions and the total can not exceed 64

 

Multicast LTL Resources

Usage: 30656 Total, 841 Used

 

 

Ivan Rostovikov

Ivan Rostovikov

Posted
  • Author
Posted

А вот что сказал show profile terse:

 

System Total = 000083743

Interrupt Total = 000034286 (40 percent)

Sched Total = 000047586 (56 percent)

 

Interrupt [03] = 000033108 (39 percent)

Interrupt [04] = 000001195 (01 percent)

Interrupt [05] = 000000291 (00 percent)

 

Кто знает что такое Interrupt [03] ?

Ivan Rostovikov

Ivan Rostovikov

Posted
  • Author
Posted

>route-map-ы есть?

Нет.

>полисироутинг?

Нет.

>vrf-ы используются?

Нет.

 

Конфиг простейший. Устройство по сути используется как Vlan роутер.

Если кому интересно разобраться - яська 62393245.

UglyAdmin

UglyAdmin

Posted
Posted

Но откуда им взяться ? Трафик то небольшой.!!!

На 6500 CPU слабые, вряд ли пророутят хотя бы 100 мегабит через процессор. Через железо должно идти...

Igor Diakonov

Igor Diakonov

Posted
Posted

Посмотреть sh interfaces switching на предмет - на каком интерфейсе больше всего Fast или Process пакетов.

Потом показать конфиг интерфейса...ну, ещё может sh fm interface VlanXX :)

Ivan Rostovikov

Ivan Rostovikov

Posted
  • Author
Posted

interface Vlan1

ip address 197.2.228.30 255.255.255.224

ip route-cache same-interface

 

Vlan1

Throttle count 11

Drops RP 1458 SP 0

SPD Flushes Fast 1148 SSE 0

SPD Aggress Fast 0

SPD Priority Inputs 986580 Drops 0

 

Protocol Path Pkts In Chars In Pkts Out Chars Out

Other Process 6220 373200 0 0

Cache misses 0

Fast 0 0 0 0

Auton/SSE 0 0 0 0

IP Process 14219581 1734854986 13042892 1673739266

Cache misses 0

Fast 7730149444 5724100951480 7729842797 5723986968784

Auton/SSE 277167552 149527593369 8023564023 5940209479561

DEC MOP Process 0 0 625 80625

Cache misses 0

Fast 0 0 0 0

Auton/SSE 0 0 0 0

Spanning Tree Process 531182 58007706 0 0

Cache misses 0

Fast 0 0 0 0

Auton/SSE 0 0 0 0

ARP Process 238425 14305500 8291 928592

Cache misses 0

Fast 0 0 0 0

Auton/SSE 0 0 0 0

 

Interface: Vlan1 IP is enabled

hw_state[iNGRESS] = not reduced, hw_state[EGRESS] = not reduced

mcast = 0

priority = 0

flags = 0x0

parent[iNGRESS] = none

inbound label: 54

Feature IPV4 Default Result Feature:

-----------------------------------------------------------------------------

FM_FEATURE_IPV4_DEFAULT i/f: Vl1

=============================================================================

DPort - Destination Port SPort - Source Port Pro - Protocol

PT - Packet Type DPT - Dst. Packet Type SPT - Src. Packet Type

X - XTAG TOS - TOS Value Res - VMR Result

RFM - R-Recirc. Flag MRTNPC - M-Multicast Flag R - Reflexive flag

- F-Fragment flag - T-Tcp Control N - Non-cachable

- M-More Fragments - P-Mask Priority(H-High, L-Low)

Adj. - Adj. Index C - Capture Flag T - M(Mask)/V(Value)

FM - Flow Mask NULL - Null FM SAO - Source Only FM

DAO - Dest. Only FM SADA - Sour.& Dest. Only VSADA - Vlan SADA Only

ISADA - Intf. SADA FF - Full Flow VFF - Vlan Full Flow

IFF - Intf. FF F-VFF - Either FF or VFF IFF-FF - Either IFF or FF

A-VSD - Atleast VSADA A-FF - Atleast FF A-VFF - Atleast VFF

A-SON - Atleast SAO A-DON - Atleast DAO A-SD - Atleast SADA

SHORT - Shortest ISADA-L- ISADA Least FF-L - FF Least

IFF-L - IFF Least A-SFF - Any short than FF A-EFF - Any except FF

A-EVFF - Any except VFF SA-L - Source Least DA-L - Dest. Least

SADA-L - SADA Least FF-LESS- FF Less N-FF - Not FF

N-IFF - Not IFF A-LVFF - Any less than VFF FULL - Full Pkt Type

EUI - EUI 64 Pkt Type EMBD - Embedded Pkt Type ELNK - EUI Link Overlap

ESIT - EUI Site Overlap LINK - Link Pkt Type SITE - Site Pkt Type

SAO-CR - Source Only(CR) DAO-CR - Dest. Only(CR) FF-CR - Full flow(CR)

VFF-CR - Vlan Full flow(CR)ERR - Flowmask Error

+----+-+---------------+---------------+-----+-----+---+---+-+---+------+----+------+

|Indx|T| Dest Ip Addr | Source Ip Addr|DPort|SPort|Pro|RFM|X|ToS|MRTNPC|Adj.| FM |

+----+-+---------------+---------------+-----+-----+---+---+-+---+-----+----+------+

 

1 V 0.0.0.0 0.0.0.0 0 0 0 --- 0 0 ----L- ---- SHORT

M 0.0.0.0 0.0.0.0 0 0 0 000 0 0

TM_PERMIT_RESULT

 

Feature OTHER Default Result Feature:

-----------------------------------------------------------------------------

FM_FEATURE_OTHER_DEFAULT i/f: Vl1

=============================================================================

+----+-+--------------+--------------+----+----+

|Indx|T| Dest Node | Source Node |EtTy|EtCo|

+----+-+--------------+--------------+----+----+

 

1 V 0000.0000.0000 0000.0000.0000 0 0

M 0000.0000.0000 0000.0000.0000 0 0

TM_SOFT_BRIDGE_RESULT

 

 

Ivan Rostovikov

Ivan Rostovikov

Posted
  • Author
Posted

Проблема решена !

Помог вот этот раздел:

http://www.cisco.com/en/US/products/hw/swi...html#situations

 

Видимо свич генерил слишком много редиректов (сам до конца не понимаю почему). Однако прописал

no ip redirect
на всех vlan-ах

Теперь загрузка по прерываниям 1% :-)

 

 

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
 Share
Go to topic listing
×
×
  • Create New...
На сайте используются файлы cookie и сервисы аналитики для корректной работы форума и улучшения качества обслуживания. Продолжая использовать сайт, вы соглашаетесь с использованием файлов cookie и с Политикой конфиденциальности.