Jump to content
Калькуляторы

One or more, more specific prefixes could not be programmed into TCAM проблема с каталистом

Начались странные "тормоза" одного из коммутаторов

System image file is "flash:c3560-ipservicesk9-mz.122-44.SE.bin"
cisco WS-C3560G-24TS (PowerPC405) processor (revision D0) with 122880K/8184K bytes of memory.

 

В логе: (не часто)

%PLATFORM_UCAST-6-PREFIX:  One or more, more specific prefixes could not be programmed into TCAM and are being covered by a less specific prefix

 

CPU utilization for five seconds: 36%/27%; one minute: 38%; five minutes: 38%

 

sh sdm prefer
The current template is "desktop default" template.
The selected template optimizes the resources in
the switch to support this level of features for
8 routed interfaces and 1024 VLANs.

  number of unicast mac addresses:                  6K
  number of IPv4 IGMP groups + multicast routes:    1K
  number of IPv4 unicast routes:                    8K
    number of directly-connected IPv4 hosts:        6K
    number of indirect IPv4 routes:                 2K
  number of IPv4 policy based routing aces:         0
  number of IPv4/MAC qos aces:                      0.75K
  number of IPv4/MAC security aces:                 1K

 

 

sh platform tcam utilization
CAM Utilization for ASIC# 0                      Max            Used
                                             Masks/Values    Masks/values

Unicast mac addresses:                        784/6272        426/3323
IPv4 IGMP groups + multicast routes:          144/1152          7/28
IPv4 unicast directly-connected routes:       784/6272        426/3323
IPv4 unicast indirectly-connected routes:     272/2176        239/1758
IPv4 policy based routing aces:                 0/0             0/0
IPv4 qos aces:                                768/768         260/260
IPv4 security aces:                          1024/1024         85/85

 

 

Подскажите, как посмотреть, чего именно "слишком много" на нем, т.е. что именно с него убирать.

Маки?

Маршруты?

Файрволлы? Еще что то? Совершенно не ясно как дебагать (

 

На свитче полный комплект - ACL, много маков, много интефейсов, IGMP

Share this post


Link to post
Share on other sites

маршрутов много, скорее всего

лечится убиранием части маршрутов и ребутом только

Share this post


Link to post
Share on other sites
маршрутов много, скорее всего

лечится убиранием части маршрутов и ребутом только

я тоже думаю чт дело в маршрутах, НО как выяснить это точно? (понятно, что убрать маршруты, интиересует из какой строки sh что-то это следует)

Маршруты уберу, отпишу.

Share this post


Link to post
Share on other sites
маршрутов много, скорее всего

лечится убиранием части маршрутов и ребутом только

я тоже думаю чт дело в маршрутах, НО как выяснить это точно? (понятно, что убрать маршруты, интиересует из какой строки sh что-то это следует)

Маршруты уберу, отпишу.

sh ip ro summ

 

и sh sdm prefer

 

и сравнить значения количества маршрутов

Share this post


Link to post
Share on other sites
sh platform ip unicast counts

sh platform ip unicast failed route

#sh platform ip unicast counts
# of HL3U fibs 4482
# of HL3U adjs 1607
# of HL3U mpaths 0
# of HL3U covering-fibs 0
# of HL3U fibs with adj failures 0
Fibs of Prefix length 0, with TCAM fails: 0
Fibs of Prefix length 1, with TCAM fails: 0
<все одинаковые>
Fibs of Prefix length 33, with TCAM fails: 0

 

#sh platform ip unicast failed route
Total of 0 covering fib entries

 

 

 

sh ip ro summ

sh sdm prefer

и сравнить значения количества маршрутов

sh ip route vrf  VRF1 summary
connected       0           6           384         912
static          0           0           0           0
...
Total           76          397         29740       91276


sh ip route vrf  VRF2 summary
Total           14          583         37824       96864

 

Сумма 583 + 397 явно меньше чем number of indirect IPv4 routes: 2K

 

Share this post


Link to post
Share on other sites

тут не обязательно дело в indirect, наверняка может быть просто много directly-connected hosts, на них же тоже "как бы" маршруты.

В лог ругается круглосуточно или только в ЧНН?

Share this post


Link to post
Share on other sites

Valaskor

Не круглосуточно, но пока зависимость не выявлена.

я понимаю, что запись arp 192.168.0.1 aabb.ccdd.eeff это тоже "как бы" маршрут но их тоже меньше чем в sdm preffer

Share this post


Link to post
Share on other sites

И все же посмотрите в час пик:

sh mac address-table count | in Available

sh ip arp summary

 

Если там будет значение более 5000, скорее всего оно иногда бывает выше и упирается в ограничение.

 

Share this post


Link to post
Share on other sites

IPv4 unicast indirectly-connected routes: 272/2176 239/1758

Тут проблема.

 

У меня было подобное.

На 3750.

В профиле написано 2176 но реально - меньше.

Меняй профиль если позволяет ситуация.

Share this post


Link to post
Share on other sites

моё:

Cisco3#sh platform tcam utilization

CAM Utilization for ASIC# 0                      Max            Used
                                             Masks/Values    Masks/values

Unicast mac addresses:                        784/6272        542/4262
IPv4 IGMP groups + multicast routes:          144/1152          8/40
IPv4 unicast directly-connected routes:       784/6272        542/4262
IPv4 unicast indirectly-connected routes:     272/2176        103/750
IPv4 policy based routing aces:                 0/0             0/0
IPv4 qos aces:                                768/768         260/260
IPv4 security aces:                          1024/1024         91/91

Note: Allocation of TCAM entries per feature uses
a complex algorithm. The above information is meant
to provide an abstract view of the current TCAM utilization

 

Cisco4#sh platform tcam utilization

CAM Utilization for ASIC# 0                      Max            Used
                                             Masks/Values    Masks/values

Unicast mac addresses:                       6364/6364       7404/7404
IPv4 IGMP groups + multicast routes:         1120/1120         93/93
IPv4 unicast directly-connected routes:      6144/6144       4317/4317
IPv4 unicast indirectly-connected routes:    2048/2048        862/862
IPv4 policy based routing aces:               452/452          12/12
IPv4 qos aces:                                512/512           6/6
IPv4 security aces:                           964/964         236/236

Note: Allocation of TCAM entries per feature uses
a complex algorithm. The above information is meant
to provide an abstract view of the current TCAM utilization

Это утро. Вечером начинает сыпать такие же ошибки в лог. Сisco3. На Cisco4 - тиш да гладь.

Так что фигня ваш tcam utilization.

 

Share this post


Link to post
Share on other sites

Похожая ситуация но стэк из 6 catalyst 3750 просто виснет, с рутеров включенных в стэк перестают видется маки, хотя на консоль свитч доступен и в логах только (последняя запись перед ручным и вынужденным ребутом):

 

%PLATFORM_UCAST-4-PREFIX: One or more, more specific prefixes could not be programmed into TCAM and are being covered by a less specific prefix, and the packets may be software forwarded

 

 

sh sdm prefer
 The current template is "desktop vlan" template.
 The selected template optimizes the resources in
 the switch to support this level of features for
 8 routed interfaces and 1024 VLANs.

  number of unicast mac addresses:                  12K
  number of IPv4 IGMP groups + multicast routes:    1K
  number of IPv4 unicast routes:                    0
  number of IPv4 policy based routing aces:         0
  number of IPv4/MAC qos aces:                      0.5K
  number of IPv4/MAC security aces:                 1K

 

 

sh plat tcam utilization

CAM Utilization for ASIC# 0                      Max            Used
                                             Masks/Values    Masks/values

 Unicast mac addresses:                       1552/12416       595/4686
 IPv4 IGMP groups:                             152/1216         67/513
 IPv4 unicast routes:                            0/0             0/0
 IPv4 policy based routing aces:                 0/0             0/0
 IPv4 qos aces:                                768/768         260/260
 IPv4 security aces:                          1024/1024         39/39

Note: Allocation of TCAM entries per feature uses
a complex algorithm. The above information is meant
to provide an abstract view of the current TCAM utilization

 

show mac address-table count | i Available

Total Mac Address Space Available: 7504

 

sh ip ro sum
IP routing table name is Default-IP-Routing-Table(0)
IP routing table maximum-paths is 32
Route Source    Networks    Subnets     Overhead    Memory (bytes)
connected       0           1           64          152
static          0           0           0           0
ospf 1          1           7           512         1216
  Intra-area: 7 Inter-area: 1 External-1: 0 External-2: 0
  NSSA External-1: 0 NSSA External-2: 0
internal        2                                   2344
Total           3           8           576         3712
 

sh platform tcam usage

=============================================================================
                                  TCAM Table
 TCAM / SSRAM Table            TCAM            SSRAM
                                Start   Size X    Start   Size Y
=============================================================================
 Local Forwarding Table:            0   3540 1        0   3540   4
 Local Learning Table:              0   3540 1     D500   3540   2
 Secondary Forwarding Table:     3080    540 1    13F80    540   8
 QoS Table:                      35C0   1800 1    1B700   1800   4
 ACL Table:                      4DC0   2000 1    21700   2000   4
 IPV6 Secondary Forwarding Tabl  7E10     C0 2    1B3C0     60   8
 IPV6 Classification Table:      7ED0     80 2    29720     40   4
 IPV6 ACL Table:                 7F50     A0 2    29820     50   4
 Station Table:                     0      0 0    29980   3540   4
 MAC Address Table:                 0      0 0    36E80      2   8
 Multicast Expansion Table:         0      0 0    36F00    422   8
 VLAN List Table:                   0      0 0    39080     40  10
 Equal Cost Route Table:            0      0 0    39080      0  20

 X - Number of 144-bit TCAM entries per descriptor
 Y - Number of bytes per descriptor
=============================================================================
SDM template realignment scheme-1 in effect
 

sh platform tcam errors

TCAM Memory Consistency Checker Errors
--------------------------------------
TCAM Space          Values      Masks     Fixups    Retries   Failures
Unassigned               0          0          0          0          0
HFTM                     0          0          0          0          0
HQATM                    0          0          0          0          0
 

 

sh switch
Switch/Stack Mac Address : fcfb.fb3f.a480
                                           H/W   Current
Switch#  Role   Mac Address     Priority Version  State
----------------------------------------------------------
 1       Member 001f.6c5b.0a00     3      0       Ready
 2       Member 203a.079b.c500     4      0       Ready
*3       Master fcfb.fb3f.a480     5      0       Ready
 4       Member 0000.0000.0000     0      0       Provisioned
 5       Member 0000.0000.0000     0      0       Provisioned
 6       Member 001c.b052.8e80     1      0       Ready
 

 

В ЛОГАХ более ничего, отправляются удаленно (настроен rsyslog)!

 

Все это лечится только РЕБУТОМ! Кто может помочь?

 

Share this post


Link to post
Share on other sites

+ ospf bdr

 

show processes cpu sorted 5min | exclude 0.00
CPU utilization for five seconds: 16%/3%; one minute: 18%; five minutes: 18%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
 163     3114111   6255333        497  0.95%  3.28%  3.33%   0 HL2MCM
 293       70977      8897       7977  0.31%  0.27%  0.56%   1 Virtual Exec
 144      448198     63062       7107  0.47%  0.53%  0.49%   0 HL3U bkgrd proce
 259      184184   1114817        165  0.31%  0.36%  0.33%   0 HRPC hl2mcm igmp
 199      274180    191122       1434  1.91%  0.37%  0.30%   0 CDP Protocol
 206      184255   1235115        149  0.15%  0.24%  0.22%   0 IP Input
  87      212597      8388      25345  1.11%  0.20%  0.20%   0 HULC Tcam Memory
 321       91284     10836       8424  1.75%  0.23%  0.12%   0 SNMP ENGINE
  80       76019    260898        291  0.15%  0.10%  0.08%   0 HRPC hlfm reques
  60       66202    262638        252  0.31%  0.07%  0.05%   0 EEM ED ND
 152       42779     16824       2542  0.15%  0.05%  0.04%   0 HQM Stack Proces
  67       42073   4383868          9  0.15%  0.05%  0.03%   0 Fifo Error Detec
 105       34991    164644        212  0.15%  0.05%  0.03%   0 HRPC pm-counters
 

 

show processes memory sorted
Processor Pool Total:   71072628 Used:   37773920 Free:   33298708
      I/O Pool Total:   12582912 Used:    8713572 Free:    3869340
Driver te Pool Total:    1048576 Used:         40 Free:    1048536

 PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process
   0   0   52158652   10939488   36872920          0          0 *Init*
  80   0   22310952   19732212    1198620     802008          0 HRPC hlfm reques
  66   0    1537668     428560     710344      88020          0 Stack Mgr Notifi
   0   0   87083512   90114304     537636   10378808    2020040 *Dead*
 250   0   25136160   20087980     354520      20304          0 IGMPSN
 163   0     412632     152736     294564          0          0 HL2MCM
 164   0    1166964     906724     294460          0          0 HL2MCM
 259   0  121483020   96939736     261716    1121796          0 HRPC hl2mcm igmp
 189   0     343116     618448     250464       5076          0 VMATM Callback
 285   0     232900          0     242952      99792          0 EEM ED Syslog
 309   0     229364        340     232248          0          0 CEF: IPv4 proces
   9   0     233452       1412     226660          0          0 ARP Input
 199   0  332683184   48171680     198188       5076          0 CDP Protocol
  14   0     261188     173260     176124          0      17160 Entity MIB API
 291   0     166576          0     173628          0          0 EEM Server
 258   0     146168          0     156220          0          0 IP RIB Update
 248   0     251580    1481800     139304          0          0 IGMPSN L2MCM
 253   0     239828    1073664     138092          0          0 MLDSN L2MCM
 226   0     179488      36756     124272      15228          0 XDR mcast
 

 
 

sh platform tcam utilization

CAM Utilization for ASIC# 0                      Max            Used
                                             Masks/Values    Masks/values

 Unicast mac addresses:                       1552/12416       597/4702
 IPv4 IGMP groups:                             152/1216         68/518
 IPv4 unicast routes:                            0/0             0/0
 IPv4 policy based routing aces:                 0/0             0/0
 IPv4 qos aces:                                768/768         260/260
 IPv4 security aces:                          1024/1024         39/39

Note: Allocation of TCAM entries per feature uses
a complex algorithm. The above information is meant
to provide an abstract view of the current TCAM utilization
 

 
 
 
sw-lob-03#sh mac address-table count | i Avail
Total Mac Address Space Available: 7471
 
 
sw-lob-03#sh mac address-table | count 1/0/1
Number of lines which match regexp = 95
sw-lob-03#sh mac address-table | count 1/0/2
Number of lines which match regexp = 7
sw-lob-03#sh mac address-table | count 1/0/3
Number of lines which match regexp = 36
sw-lob-03#sh mac address-table | count 1/0/4
Number of lines which match regexp = 275
sw-lob-03#sh mac address-table | count 1/0/5
Number of lines which match regexp = 177
sw-lob-03#sh mac address-table | count 1/0/6
Number of lines which match regexp = 87
sw-lob-03#sh mac address-table | count 1/0/7
Number of lines which match regexp = 0
sw-lob-03#sh mac address-table | count 1/0/8
Number of lines which match regexp = 0
sw-lob-03#sh mac address-table | count 1/0/9
Number of lines which match regexp = 18
sw-lob-03#sh mac address-table | count 1/0/10
Number of lines which match regexp = 54
sw-lob-03#sh mac address-table | count 1/0/11
Number of lines which match regexp = 11
sw-lob-03#sh mac address-table | count 1/0/12
Number of lines which match regexp = 16
 
sw-lob-03#sh mac address-table | count 2/0/1
Number of lines which match regexp = 119
sw-lob-03#sh mac address-table | count 2/0/2
Number of lines which match regexp = 0
sw-lob-03#sh mac address-table | count 2/0/3
Number of lines which match regexp = 20
sw-lob-03#sh mac address-table | count 2/0/4
Number of lines which match regexp = 9
sw-lob-03#sh mac address-table | count 2/0/5
Number of lines which match regexp = 81
sw-lob-03#sh mac address-table | count 2/0/6
Number of lines which match regexp = 105
sw-lob-03#sh mac address-table | count 2/0/7
Number of lines which match regexp = 0
sw-lob-03#sh mac address-table | count 2/0/8
Number of lines which match regexp = 11
sw-lob-03#sh mac address-table | count 2/0/9
Number of lines which match regexp = 47
sw-lob-03#sh mac address-table | count 2/0/10
Number of lines which match regexp = 53
sw-lob-03#sh mac address-table | count 2/0/12
Number of lines which match regexp = 23
 
 
sw-lob-03#sh mac address-table | count 3/0/1
Number of lines which match regexp = 6
sw-lob-03#sh mac address-table | count 3/0/2
Number of lines which match regexp = 25
sw-lob-03#sh mac address-table | count 3/0/3
Number of lines which match regexp = 51
sw-lob-03#sh mac address-table | count 3/0/4
Number of lines which match regexp = 0
sw-lob-03#sh mac address-table | count 3/0/5
Number of lines which match regexp = 0
sw-lob-03#sh mac address-table | count 3/0/6
Number of lines which match regexp = 1
sw-lob-03#sh mac address-table | count 3/0/7
Number of lines which match regexp = 1
sw-lob-03#sh mac address-table | count 3/0/8
Number of lines which match regexp = 0
sw-lob-03#sh mac address-table | count 3/0/9
Number of lines which match regexp = 0
sw-lob-03#sh mac address-table | count 3/0/10
Number of lines which match regexp = 0
sw-lob-03#sh mac address-table | count 3/0/11
Number of lines which match regexp = 1
sw-lob-03#sh mac address-table | count 3/0/12
Number of lines which match regexp = 1
sw-lob-03#sh mac address-table | count 3/0/13
Number of lines which match regexp = 1
sw-lob-03#sh mac address-table | count 3/0/14
Number of lines which match regexp = 0
sw-lob-03#sh mac address-table | count 3/0/15
Number of lines which match regexp = 0
sw-lob-03#sh mac address-table | count 3/0/16
Number of lines which match regexp = 1
sw-lob-03#sh mac address-table | count 3/0/17
Number of lines which match regexp = 0
sw-lob-03#sh mac address-table | count 3/0/18
Number of lines which match regexp = 0
sw-lob-03#sh mac address-table | count 3/0/19
Number of lines which match regexp = 1
sw-lob-03#sh mac address-table | count 3/0/20
Number of lines which match regexp = 1
sw-lob-03#sh mac address-table | count 3/0/21
Number of lines which match regexp = 2
sw-lob-03#sh mac address-table | count 3/0/22
Number of lines which match regexp = 2
sw-lob-03#sh mac address-table | count 3/0/23
Number of lines which match regexp = 0
sw-lob-03#sh mac address-table | count 3/0/24
Number of lines which match regexp = 2
 
sw-lob-03#sh mac address-table interface gi 6/0/1 | count 6/0/1
Number of lines which match regexp = 1
sw-lob-03#sh mac address-table | count 6/0/1
Number of lines which match regexp = 2738
sw-lob-03#sh mac address-table | count 6/0/2
Number of lines which match regexp = 133
sw-lob-03#sh mac address-table | count 6/0/3
Number of lines which match regexp = 0
sw-lob-03#sh mac address-table | count 6/0/4
Number of lines which match regexp = 0
sw-lob-03#sh mac address-table | count 6/0/5
Number of lines which match regexp = 0
sw-lob-03#sh mac address-table | count 6/0/6
Number of lines which match regexp = 0
sw-lob-03#sh mac address-table | count 6/0/7
Number of lines which match regexp = 0
sw-lob-03#sh mac address-table | count 6/0/8
Number of lines which match regexp = 13
sw-lob-03#sh mac address-table | count 6/0/9
Number of lines which match regexp = 2
sw-lob-03#sh mac address-table | count 6/0/10
Number of lines which match regexp = 1
sw-lob-03#sh mac address-table | count 6/0/11
Number of lines which match regexp = 3
sw-lob-03#sh mac address-table | count 6/0/12
Number of lines which match regexp = 1
sw-lob-03#sh mac address-table | count 6/0/13
Number of lines which match regexp = 18
sw-lob-03#sh mac address-table | count 6/0/14
Number of lines which match regexp = 6
sw-lob-03#sh mac address-table | count 6/0/15
Number of lines which match regexp = 1
sw-lob-03#sh mac address-table | count 6/0/16
Number of lines which match regexp = 1
 
sw-lob-03#sh mac address-table interface te 6/0/1 | count 6/0/1
Number of lines which match regexp = 2717

Share this post


Link to post
Share on other sites

number of IPv4 unicast routes:                    0

у вас шаблон sdm выбран под л2 без ip routing.

оно и не может префиксы из оспф заинсталить.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this