Jump to content

Гадит карта в сеть?

kostas
 Share

Recommended Posts

kostas

kostas

Posted
Posted

Приветствую всех.

 

Столкнулся с интересной и пакостной проблемой - как будто хост в сеть гадит и кладет весь L2 сегмент.

Опишу по порядку, чтобы сложилось четкое представление о конфигурации и самой проблеме.

 

Есть несколько десяткой хостов одной сети. Все хосты ранее были подключены к N3K, работали более года без проблем и впоследствии были перевезены и подключены к N7K тем же набором кабелей.

 

1. Коммутатор Cisco Nexus 7K + несколько FEX 2232

2. Серверы с картами X520DA2

3. DAC кабели 3 метра, разные - Huawei, Dlink, какие-то еще - сиска определяет их по разному, но все видит, линки в других вланах стабильные

4. Хосты подключены одним портом, туннели никакие на хостах не строятся

 

Есть влан (540) в котором происходит это проблема. В это влане порядка 40 хостов. Хосты расположены на разных FEX. Трафик в сегменте не большой.

 

Начиналось все так: В определенный момент времени начинали падать и сразу же подниматься линки у нескольких хостов, по очереди. В логах сиски выглядело это так: 

Sep 30 14:25:02 ds5 : 2022 Sep 30 14:25:01 MSK: %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet101/1/18 is down (Link failure)
Sep 30 14:25:04 ds5 : 2022 Sep 30 14:25:03 MSK: %ETHPORT-5-SPEED: Interface Ethernet103/1/4, operational speed changed to 1 Gbps
Sep 30 14:25:04 ds5 : 2022 Sep 30 14:25:03 MSK: %ETHPORT-5-IF_DUPLEX: Interface Ethernet103/1/4, operational duplex mode changed to Full
Sep 30 14:25:04 ds5 : 2022 Sep 30 14:25:03 MSK: %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet103/1/4, operational Receive Flow Control state changed to off
Sep 30 14:25:04 ds5 : 2022 Sep 30 14:25:03 MSK: %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet103/1/4, operational Transmit Flow Control state changed to on
Sep 30 14:25:04 ds5 : 2022 Sep 30 14:25:03 MSK: %ETHPORT-5-IF_UP: Interface Ethernet103/1/4 is up in mode access
Sep 30 14:25:04 ds5 : 2022 Sep 30 14:25:03 MSK: %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet101/1/18, operational Transmit Flow Control state changed to on
Sep 30 14:25:04 ds5 : 2022 Sep 30 14:25:03 MSK: %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet101/1/18, operational Receive Flow Control state changed to off
Sep 30 14:25:04 ds5 : 2022 Sep 30 14:25:03 MSK: %ETHPORT-5-SPEED: Interface Ethernet101/1/18, operational speed changed to 10 Gbps
Sep 30 14:25:04 ds5 : 2022 Sep 30 14:25:03 MSK: %ETHPORT-5-IF_DUPLEX: Interface Ethernet101/1/18, operational duplex mode changed to Full
Sep 30 14:25:04 ds5 : 2022 Sep 30 14:25:03 MSK: %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet101/1/18, operational Receive Flow Control state changed to off
Sep 30 14:25:04 ds5 : 2022 Sep 30 14:25:03 MSK: %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet101/1/18, operational Transmit Flow Control state changed to on
Sep 30 14:25:05 ds5 : 2022 Sep 30 14:25:04 MSK: %ETHPORT-5-IF_UP: Interface Ethernet101/1/18 is up in mode access
Sep 30 14:25:31 ds5 : 2022 Sep 30 14:25:30 MSK: %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet103/1/4 is down (Link failure)
Sep 30 14:25:32 ds5 : 2022 Sep 30 14:25:31 MSK: %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet101/1/28 is down (Link failure)
Sep 30 14:25:32 ds5 : 2022 Sep 30 14:25:31 MSK: %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet103/1/4, operational Transmit Flow Control state changed to on
Sep 30 14:25:32 ds5 : 2022 Sep 30 14:25:31 MSK: %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet103/1/4, operational Receive Flow Control state changed to off
Sep 30 14:25:34 ds5 : 2022 Sep 30 14:25:33 MSK: %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet101/1/28, operational Transmit Flow Control state changed to on
Sep 30 14:25:34 ds5 : 2022 Sep 30 14:25:33 MSK: %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet101/1/28, operational Receive Flow Control state changed to off
Sep 30 14:25:34 ds5 : 2022 Sep 30 14:25:33 MSK: %ETHPORT-5-SPEED: Interface Ethernet101/1/28, operational speed changed to 10 Gbps
Sep 30 14:25:34 ds5 : 2022 Sep 30 14:25:33 MSK: %ETHPORT-5-IF_DUPLEX: Interface Ethernet101/1/28, operational duplex mode changed to Full
Sep 30 14:25:34 ds5 : 2022 Sep 30 14:25:33 MSK: %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet101/1/28, operational Receive Flow Control state changed to off
Sep 30 14:25:34 ds5 : 2022 Sep 30 14:25:33 MSK: %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet101/1/28, operational Transmit Flow Control state changed to on
Sep 30 14:25:34 ds5 : 2022 Sep 30 14:25:33 MSK: %ETHPORT-5-IF_UP: Interface Ethernet101/1/28 is up in mode access

Методом тыка мы нашел, что именно с одного хоста начиналась вся эта свистопляска. Погасил порт этого хоста на коммутаторе и настала тишина. Больше месяца все работало без проблем. На хосте была переустановлена ОС (win), DAC заменен на новый, другой марки. Хост стоял без работы какое-то время и все было тихо, как на днях снова все моргнуло. Было принято решение о замене сетевой карты, но не успели. Сегодня ночью по заббиксу пропала часть хостов и больше не появилось. Как выяснилось, стали не доступны все хосты этой сети, кроме одного который был подключен на внутреннюю 1GE карту, в отличии от всех остальных. Более того, часть интерфейсов флапала, а часть нет, но трафик по ним не шел. MAC адресов в влане не было, кроме той одной доступной на 1GE.

Пошел выключать все интерфейсы по одному, пока не дошел одного, выключение которого привело сразу же к восстановлению работы всех хостов. Оставил его выключенным и до сих пор все работает.

 

Вопрос - что это может быть? Напомню, что DAC менял, ОС менял. Проблемные хосты работали с месяц без каких либо проблем.

 

 

jffulcrum

jffulcrum

Posted
Posted

Обычная проблема в наших бизнес-центрах - когда забыли на новом коммутаторе доступа наглухо выключить и запретить LLDP (а также ND, BFD, LACP, STP), и какой-нибудь шибко продвинутый роутер или сервер арендатора начинал с соседом в лице нашего коммутатора что-нибудь строить...

kostas

kostas

Posted
  • Author
Posted

Тут нет коммутаторов, к портам подключены только серверы одни портом. В логах сиски все перестроения обычно оседают. Но самое главное, пострадал один L2 сегмент, но не полностью, а только хосты подключенные на 10GE, 1GE на штатном интерфейсе сервера в это время работал без проблем.

[anp/hsw]

[anp/hsw]

Posted
Posted

flow control-то отключите, вон сколько жалоб на него.

Это первое, что надо сделать в операторской сети. От него все беды.

Он может и не быть причиной ваших проблем, но добавлять масла в огонь.

Gray swordsman

Gray swordsman

Posted
Posted

Первое падение линка до проблемного хоста это следствие или причина ? если причина, то скорее всего запускается какое-то перестроение stp, если следствие то как мне кажется это какая-то l2 петля. В логе flow-control как мне кажется к данной проблеме не относится от слова совсем.

roma33rus

roma33rus

Posted
Posted
4 часа назад, Gray swordsman сказал:

Первое падение линка до проблемного хоста это следствие или причина ? если причина, то скорее всего запускается какое-то перестроение stp, если следствие то как мне кажется это какая-то l2 петля. В логе flow-control как мне кажется к данной проблеме не относится от слова совсем.


stp не выключает линки, а блокирует прохождение трафика.

frol13

frol13

Posted
Posted

Я может не совсем попаду в кон, но похожая проблема у нас была во внутренней сети в офисе. У нас оказалась банальщина - не было надежного контакта где-то в разъемах на линке. В результате многочисленных "отваливаний"- восстановлений падала вся таблица интерфейсов из-за многочисленных ее перестроений с последующим выходом ее за пределы лимита памяти, ей отведенной. 

kostas

kostas

Posted
  • Author
Posted

Хосты подключены в разные FEX расширители, тч проблема с линейкой маловероятна, имхо.

Петли и STP тут неоткуда взяться - сервер оконечное устройство, без каких либо коммутаторов с единственным интерфейсом. И как уже упоминали STP Линк не кладет.

flow control попробую отключить, но сдается, что не с этим связано, тк проблема только в одном влане и только на 10GE интерфейсах. Все остальные вланы и 1GE линки живут как ни бывало.

Не знаю, падение линка - это следствие или причина. Оба конца сигнализируют о потери несущей.

 

Вот лог порта, который кладет все порты 10GE в одном влане 

>>>>FSM: <Ethernet106/1/3> has 191 logged transitions<<<<<

1) Event:ESQ_REQ length:38, at 175705 usecs after Sun Dec 11 13:12:06 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_PIXM_LOCAL(153), Opc:MTS_OPC_PIXM_SET_MULT_CBL_VLAN_PORT_STATE(69641)

2) Event:ESQ_REQ length:38, at 176537 usecs after Sun Dec 11 13:12:06 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Sending MTS_OPC_LC_PORT_CLIENT_CFG and/or PARAM_CHANGE opcode

3) Event:ESQ_REQ length:38, at 176543 usecs after Sun Dec 11 13:12:06 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ELTM(192), Opc:MTS_OPC_ELTM_INTF_FEATURE_REQ_RESP(72748)

4) Event:ESQ_REQ length:38, at 176547 usecs after Sun Dec 11 13:12:06 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_VLAN_MGR(167), Opc:MTS_OPC_VLAN_MGR_GET_PORT_TRUNKING_MEMBERSHIP(23562)

5) Event:ESQ_REQ length:38, at 176549 usecs after Sun Dec 11 13:12:06 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_DIAGCLIENT(366), Opc:MTS_OPC_GOLD_PP_TEST_RESUME(59417)

6) Event:ESQ_REQ length:38, at 176604 usecs after Sun Dec 11 13:12:06 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_VLAN_MGR(167), Opc:MTS_OPC_VLAN_MGR_GET_PORT_MEMBERSHIP_2(23617)

7) Event:ESQ_RSP length:38, at 178036 usecs after Sun Dec 11 13:12:06 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_VLAN_MGR(167), Opc:MTS_OPC_VLAN_MGR_GET_PORT_MEMBERSHIP_2(23617)

8) Event:ESQ_REQ length:38, at 178044 usecs after Sun Dec 11 13:12:06 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ELTM(192), Opc:MTS_OPC_ELTM_INTF_FEATURE_REQ(72713)

9) Event:ESQ_REQ length:38, at 178046 usecs after Sun Dec 11 13:12:06 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:SUCCESS path done, skip-to-end, feed EV_CFG_DONE

10) FSM:<Ethernet106/1/3> Transition at 178163 usecs after Sun Dec 11 13:12:06 2022
    Previous state: [ETH_PORT_FSM_ST_WAIT_APPLY_CONFIG]
    Triggered event: [ETH_PORT_FSM_EV_CFG_DONE]
    Next state: [ETH_PORT_FSM_ST_INIT_EVAL]

11) FSM:<Ethernet106/1/3> Transition at 178226 usecs after Sun Dec 11 13:12:06 2022
    Previous state: [ETH_PORT_FSM_ST_INIT_EVAL]
    Triggered event: [ETH_PORT_FSM_EV_IE_ADMIN_DOWN]
    Next state: [ETH_PORT_FSM_ST_DOWN]

12) FSM:<Ethernet106/1/3> Transition at 997753 usecs after Mon Dec 12 15:55:57 2022
    Previous state: [ETH_PORT_FSM_ST_DOWN]
    Triggered event: [ETH_PORT_FSM_EV_ADMIN_CFG_EVAL]
    Next state: [FSM_ST_NO_CHANGE]

13) FSM:<Ethernet106/1/3> Transition at 997814 usecs after Mon Dec 12 15:55:57 2022
    Previous state: [ETH_PORT_FSM_ST_DOWN]
    Triggered event: [ETH_PORT_FSM_EV_CFG_FLAP]
    Next state: [FSM_ST_NO_CHANGE]

14) FSM:<Ethernet106/1/3> Transition at 997821 usecs after Mon Dec 12 15:55:57 2022
    Previous state: [ETH_PORT_FSM_ST_DOWN]
    Triggered event: [ETH_PORT_FSM_EV_PHY_DOWN_DONE]
    Next state: [ETH_PORT_FSM_ST_WAIT_APPLY_CONFIG]

15) Event:ESQ_START length:38, at 997837 usecs after Mon Dec 12 15:55:57 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq Type:SERIAL

16) Event:ESQ_REQ length:38, at 997958 usecs after Mon Dec 12 15:55:57 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:determine steps to skip

17) Event:ESQ_REQ length:38, at 998003 usecs after Mon Dec 12 15:55:57 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_REGISTRY(0), Opc:MTS_OPC_SYNC_MESSAGE(0)

18) Event:ESQ_REQ length:38, at 998006 usecs after Mon Dec 12 15:55:57 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_DIAGCLIENT(366), Opc:MTS_OPC_GOLD_PP_TEST_STOP(59414)

19) Event:ESQ_REQ length:38, at 998007 usecs after Mon Dec 12 15:55:57 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_PORT_CLIENT(181), Opc:MTS_OPC_LC_PORT_FCOT_SPROM_READ(8194)

20) Event:ESQ_REQ length:38, at 998009 usecs after Mon Dec 12 15:55:57 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_PIXM_LOCAL(153), Opc:MTS_OPC_PIXM_SET_MULT_CBL_VLAN_PORT_STATE(69641)

21) Event:ESQ_REQ length:38, at 999668 usecs after Mon Dec 12 15:55:57 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Sending MTS_OPC_LC_PORT_CLIENT_CFG and/or PARAM_CHANGE opcode

22) Event:ESQ_REQ length:38, at 999697 usecs after Mon Dec 12 15:55:57 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ELTM(192), Opc:MTS_OPC_ELTM_INTF_FEATURE_REQ_RESP(72748)

23) Event:ESQ_REQ length:38, at 999701 usecs after Mon Dec 12 15:55:57 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_VLAN_MGR(167), Opc:MTS_OPC_VLAN_MGR_GET_PORT_TRUNKING_MEMBERSHIP(23562)

24) Event:ESQ_REQ length:38, at 999704 usecs after Mon Dec 12 15:55:57 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_DIAGCLIENT(366), Opc:MTS_OPC_GOLD_PP_TEST_RESUME(59417)

25) Event:ESQ_REQ length:38, at 999785 usecs after Mon Dec 12 15:55:57 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_VLAN_MGR(167), Opc:MTS_OPC_VLAN_MGR_GET_PORT_MEMBERSHIP_2(23617)

26) Event:ESQ_RSP length:38, at 1061 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_VLAN_MGR(167), Opc:MTS_OPC_VLAN_MGR_GET_PORT_MEMBERSHIP_2(23617)

27) Event:ESQ_REQ length:38, at 1069 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ELTM(192), Opc:MTS_OPC_ELTM_INTF_FEATURE_REQ(72713)

28) Event:ESQ_REQ length:38, at 1071 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:SUCCESS path done, skip-to-end, feed EV_CFG_DONE

29) FSM:<Ethernet106/1/3> Transition at 1269 usecs after Mon Dec 12 15:55:58 2022
    Previous state: [ETH_PORT_FSM_ST_WAIT_APPLY_CONFIG]
    Triggered event: [ETH_PORT_FSM_EV_CFG_DONE]
    Next state: [ETH_PORT_FSM_ST_INIT_EVAL]

30) FSM:<Ethernet106/1/3> Transition at 1548 usecs after Mon Dec 12 15:55:58 2022
    Previous state: [ETH_PORT_FSM_ST_INIT_EVAL]
    Triggered event: [ETH_PORT_FSM_EV_IE_PORT_INIT]
    Next state: [ETH_PORT_FSM_ST_SPAN_EVAL]

31) FSM:<Ethernet106/1/3> Transition at 1551 usecs after Mon Dec 12 15:55:58 2022
    Previous state: [ETH_PORT_FSM_ST_SPAN_EVAL]
    Triggered event: [ETH_PORT_FSM_EV_NON_SPAN_DEST]
    Next state: [ETH_PORT_FSM_ST_WAIT_PRE_CFG]

32) Event:ESQ_START length:38, at 1575 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq Type:SERIAL

33) Event:ESQ_REQ length:38, at 1605 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:determine steps to skip

34) Event:ESQ_REQ length:38, at 1624 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Release pacer credits

35) Event:ESQ_REQ length:38, at 1628 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:check for issu credits

36) Event:ESQ_REQ length:38, at 1649 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:check_1 if port is eventually going to be down

37) Event:ESQ_REQ length:38, at 1653 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_OHMS(1011), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)

38) Event:ESQ_REQ length:38, at 1671 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_SATMGR(619), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
    RRtoken:0xFEF90D6B

39) Event:ESQ_RSP length:38, at 2445 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_SATMGR(619), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
    RRtoken:0xFEF90D6B

40) Event:ESQ_REQ length:38, at 2477 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_SIM(1167), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)

41) Event:ESQ_REQ length:38, at 2726 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_FCOE(473), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)

42) Event:ESQ_REQ length:38, at 2746 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ETH_SPAN(174), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
    RRtoken:0xFEF90D69

43) Event:ESQ_RSP length:38, at 3094 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_ETH_SPAN(174), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
    RRtoken:0xFEF90D69

44) Event:ESQ_REQ length:38, at 3122 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_CTS(400), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
    RRtoken:0xFEF90D68

45) Event:ESQ_REQ length:38, at 3242 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_MKA(1509), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
    RRtoken:0xFEF90D67

46) Event:ESQ_REQ length:38, at 3372 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_DOT1X(208), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)

47) Event:ESQ_REQ length:38, at 3389 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ELTM(192), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
    RRtoken:0xFEF90D65

48) Event:ESQ_RSP length:38, at 303531 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_ELTM(192), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
    RRtoken:0xFEF90D65

49) Event:ESQ_REQ length:38, at 303578 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_L2FM(221), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)

50) Event:ESQ_REQ length:38, at 303610 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ETH_PORT_SEC(191), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
    RRtoken:0xFEF90D64

51) Event:ESQ_REQ length:38, at 303713 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_QOS_MANAGER(59), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)

52) Event:ESQ_REQ length:38, at 303731 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_RPM_CTRL(348), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
    RRtoken:0xFEF90D63

53) Event:ESQ_RSP length:38, at 605136 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_RPM_CTRL(348), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
    RRtoken:0xFEF90D63

54) Event:ESQ_REQ length:38, at 605168 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ASSOC_MGR(714), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
    RRtoken:0xFEF90D62

55) Event:ESQ_REQ length:38, at 605294 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:check_2 if port is eventually going to be down

56) Event:ESQ_REQ length:38, at 605337 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Pacer Check

57) Event:ESQ_REQ length:38, at 605363 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:PORT_CMDS Programming

58) Event:ESQ_REQ length:38, at 605385 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:check_3 if port is eventually going to be down

59) Event:ESQ_REQ length:38, at 605402 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_FWM(602), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
    RRtoken:0xFEF90D61

60) Event:ESQ_RSP length:38, at 817213 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_FWM(602), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
    RRtoken:0xFEF90D61

61) Event:ESQ_REQ length:38, at 817314 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_PORT_CLIENT(181), Opc:MTS_OPC_ETHPM_LCP_SET_PORT_CFG(61443)
    RRtoken:0xFEF90D60

62) Event:ESQ_RSP length:38, at 845833 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_PORT_CLIENT(181), Opc:MTS_OPC_ETHPM_LCP_SET_PORT_CFG(61443)
    RRtoken:0xFEF90D60

63) Event:ESQ_REQ length:38, at 845861 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Pacer Reset

64) FSM:<Ethernet106/1/3> Transition at 845927 usecs after Mon Dec 12 15:55:58 2022
    Previous state: [ETH_PORT_FSM_ST_WAIT_PRE_CFG]
    Triggered event: [ETH_PORT_FSM_EV_PRE_CFG_DONE]
    Next state: [ETH_PORT_FSM_ST_LINK_INIT]

65) FSM:<Ethernet106/1/3> Transition at 109257 usecs after Mon Dec 12 15:55:59 2022
    Previous state: [ETH_PORT_FSM_ST_LINK_INIT]
    Triggered event: [ETH_PORT_FSM_EV_LINK_UP]
    Next state: [ETH_PORT_FSM_ST_WAIT_BRINGUP]

66) Event:ESQ_START length:38, at 109367 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq Type:SERIAL

67) Event:ESQ_REQ length:38, at 109409 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:determine steps to skip

68) Event:ESQ_REQ length:38, at 109413 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:check for issu

69) Event:ESQ_REQ length:38, at 109414 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:check for pacer credits

70) Event:ESQ_REQ length:38, at 109476 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_PIXM_LOCAL(153), Opc:MTS_OPC_PIXM_SET_MULT_CBL_VLAN_PORT_STATE(69641)

71) Event:ESQ_RSP length:38, at 111602 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_PIXM_LOCAL(153), Opc:MTS_OPC_PIXM_SET_MULT_CBL_VLAN_PORT_STATE(69641)

72) Event:ESQ_REQ length:38, at 111629 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ASSOC_MGR(714), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)
    RRtoken:0xFEF90D5F

73) Event:ESQ_REQ length:38, at 112720 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_REGISTRY(0), Opc:MTS_OPC_IM_PHY_LINK_STATE_CHANGE(62470)

74) Event:ESQ_REQ length:38, at 112771 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_LLDP(456), Opc:MTS_OPC_DCX_CONFIG_FEATURE(96271)

75) Event:ESQ_REQ length:38, at 112796 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_CTS(400), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)
    RRtoken:0xFEF90D5E

76) Event:ESQ_REQ length:38, at 124672 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_MKA(1509), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)
    RRtoken:0xFEF90D5D

77) Event:ESQ_REQ length:38, at 124769 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_DOT1X(208), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)

78) Event:ESQ_REQ length:38, at 124820 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_FCOE(473), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)

79) Event:ESQ_REQ length:38, at 124824 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_PORT_CLIENT(181), Opc:MTS_OPC_ETHPM_LCP_SET_PORT_CFG(61443)

80) Event:ESQ_REQ length:38, at 124826 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_VLAN_MGR(167), Opc:MTS_OPC_VLAN_MGR_GET_PORT_TRUNKING_MEMBERSHIP(23562)

81) Event:ESQ_REQ length:38, at 124866 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_VLAN_MGR(167), Opc:MTS_OPC_VLAN_MGR_GET_PORT_MEMBERSHIP(23560)

82) Event:ESQ_RSP length:38, at 413650 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_VLAN_MGR(167), Opc:MTS_OPC_VLAN_MGR_GET_PORT_MEMBERSHIP(23560)

83) Event:ESQ_REQ length:38, at 413676 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ELTM(192), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)
    RRtoken:0xFEF90D59

84) Event:ESQ_RSP length:38, at 414045 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_ELTM(192), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)
    RRtoken:0xFEF90D59

85) Event:ESQ_REQ length:38, at 414119 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ETH_PORT_CHANNEL_MGR(378), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)
    RRtoken:0x145F60F7

86) Event:ESQ_RSP length:38, at 414275 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_ETH_PORT_CHANNEL_MGR(378), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)
    RRtoken:0x145F60F7

87) Event:ESQ_REQ length:38, at 414339 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_PVLAN(180), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)

88) Event:ESQ_REQ length:38, at 414405 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_IPQOS_MGR(377), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)
    RRtoken:0xFEF90D57

89) Event:ESQ_RSP length:38, at 415298 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_IPQOS_MGR(377), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)
    RRtoken:0xFEF90D57

90) Event:ESQ_REQ length:38, at 415319 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_PORT_CLIENT(181), Opc:MTS_OPC_LC_PORT_CLIENT_CONFIG(8186)

91) Event:ESQ_REQ length:38, at 415333 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ELTM(192), Opc:MTS_OPC_ETHPM_PORT_BRINGUP_LACP_INDIVIDUAL(61523)

92) FSM:<Ethernet106/1/3> Transition at 415389 usecs after Mon Dec 12 15:55:59 2022
    Previous state: [ETH_PORT_FSM_ST_WAIT_BRINGUP]
    Triggered event: [ETH_PORT_FSM_EV_BRINGUP_DONE]
    Next state: [ETH_PORT_FSM_ST_WAIT_PHASE1_CTRL_UP]

93) Event:ESQ_START length:38, at 415418 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq Type:SERIAL

94) Event:ESQ_REQ length:38, at 415446 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Port up determine steps to skip

95) Event:ESQ_REQ length:38, at 415471 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:check_1 if port is eventually going to be down

96) Event:ESQ_REQ length:38, at 415494 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:PORT_CMDS Programming

97) Event:ESQ_REQ length:38, at 415513 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Sending state up notif to shared vdc

98) Event:ESQ_REQ length:38, at 415521 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_LIM(1237), Opc:MTS_OPC_ETHPM_LIM_REQ_RSP(61504)

99) FSM:<Ethernet106/1/3> Transition at 415548 usecs after Mon Dec 12 15:55:59 2022
    Previous state: [ETH_PORT_FSM_ST_WAIT_PHASE1_CTRL_UP]
    Triggered event: [ETH_PORT_FSM_EV_START_PRE_LOGICAL_BRINGUP]
    Next state: [ETH_PORT_FSM_ST_WAIT_PRE_LOGICAL_UP]

100) Event:ESQ_START length:38, at 415566 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq Type:SERIAL

101) Event:ESQ_REQ length:38, at 415610 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:determine steps to skip

102) Event:ESQ_REQ length:38, at 415616 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Waiting to bring Peer Link up

103) FSM:<Ethernet106/1/3> Transition at 415640 usecs after Mon Dec 12 15:55:59 2022
    Previous state: [ETH_PORT_FSM_ST_WAIT_PRE_LOGICAL_UP]
    Triggered event: [ETH_PORT_FSM_EV_PRE_LOGICAL_DONE]
    Next state: [FSM_ST_NO_CHANGE]

104) FSM:<Ethernet106/1/3> Transition at 416077 usecs after Mon Dec 12 15:55:59 2022
    Previous state: [ETH_PORT_FSM_ST_WAIT_PRE_LOGICAL_UP]
    Triggered event: [ETH_PORT_FSM_EV_START_LOGICAL_UP]
    Next state: [ETH_PORT_FSM_ST_WAIT_LOGICAL_UP]

105) FSM:<Ethernet106/1/3> Transition at 416081 usecs after Mon Dec 12 15:55:59 2022
    Previous state: [ETH_PORT_FSM_ST_WAIT_LOGICAL_UP]
    Triggered event: [ETH_PORT_FSM_EV_ACCESS_UP]
    Next state: [ETH_PORT_FSM_ST_WAIT_PHASE2_CTRL_UP]

106) Event:ESQ_START length:38, at 416097 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq Type:SERIAL

107) Event:ESQ_REQ length:38, at 416107 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Phase2 Port up determine steps to skip

108) FSM:<Ethernet106/1/3> Transition at 416131 usecs after Mon Dec 12 15:55:59 2022
    Previous state: [ETH_PORT_FSM_ST_WAIT_PHASE2_CTRL_UP]
    Triggered event: [ETH_PORT_FSM_EV_ACCESS_UP]
    Next state: [ETH_PORT_FSM_ST_L2_UP]

109) FSM:<Ethernet106/1/3> Transition at 247300 usecs after Mon Dec 12 15:57:42 2022
    Previous state: [ETH_PORT_FSM_ST_L2_UP]
    Triggered event: [ETH_PORT_FSM_EV_ADMIN_CFG_EVAL]
    Next state: [FSM_ST_NO_CHANGE]

110) FSM:<Ethernet106/1/3> Transition at 247341 usecs after Mon Dec 12 15:57:42 2022
    Previous state: [ETH_PORT_FSM_ST_L2_UP]
    Triggered event: [ETH_PORT_FSM_EV_CFG_FLAP]
    Next state: [FSM_ST_NO_CHANGE]

111) Event:ESQ_START length:38, at 247365 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq Type:SERIAL

112) Event:ESQ_REQ length:38, at 247390 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Graceful shut steps to skip

113) Event:ESQ_REQ length:38, at 247422 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq: create graceful_down req payload

114) Event:ESQ_REQ length:38, at 248432 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_REGISTRY(0), Opc:MTS_OPC_IM_L3_IF_GRACEFUL_DOWN(62520)
    RRtoken:0x145FD764

115) Event:ESQ_RSP length:38, at 248535 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_REGISTRY(0), Opc:MTS_OPC_IM_L3_IF_GRACEFUL_DOWN(62520)
    RRtoken:0x145FD764

116) Event:ESQ_REQ length:38, at 248570 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_PORT_CLIENT(181), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEFEEEB9

117) Event:ESQ_REQ length:38, at 248604 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Satellite port down pacer request

118) Event:ESQ_REQ length:38, at 249553 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:send state change

119) FSM:<Ethernet106/1/3> Transition at 249639 usecs after Mon Dec 12 15:57:42 2022
    Previous state: [ETH_PORT_FSM_ST_L2_UP]
    Triggered event: [ETH_PORT_FSM_EV_PHASE1_CTRL_DOWN]
    Next state: [FSM_ST_NO_CHANGE]

120) Event:ESQ_START length:38, at 249686 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq Type:SERIAL

121) Event:ESQ_REQ length:38, at 249712 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Phase1 port down determine steps to skip

122) Event:ESQ_REQ length:38, at 249718 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_LIM(1237), Opc:MTS_OPC_ETHPM_LIM_REQ_RSP(61504)

123) FSM:<Ethernet106/1/3> Transition at 249756 usecs after Mon Dec 12 15:57:42 2022
    Previous state: [ETH_PORT_FSM_ST_L2_UP]
    Triggered event: [ETH_PORT_FSM_EV_LOGICAL_DOWN]
    Next state: [ETH_PORT_FSM_ST_WAIT_LOGICAL_DOWN]

124) Event:ESQ_START length:38, at 250300 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq Type:SERIAL

125) Event:ESQ_REQ length:38, at 250331 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_STP(171), Opc:MTS_OPC_ETHPM_PORT_LOGICAL_CLEANUP(61447)
    RRtoken:0xFEF90D2C

126) Event:ESQ_RSP length:38, at 260949 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_STP(171), Opc:MTS_OPC_ETHPM_PORT_LOGICAL_CLEANUP(61447)
    RRtoken:0xFEF90D2C

127) Event:ESQ_REQ length:38, at 260982 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_RPM_CTRL(348), Opc:MTS_OPC_ETHPM_PORT_LOGICAL_CLEANUP(61447)
    RRtoken:0xFEF90D2B

128) Event:ESQ_RSP length:38, at 262488 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_RPM_CTRL(348), Opc:MTS_OPC_ETHPM_PORT_LOGICAL_CLEANUP(61447)
    RRtoken:0xFEF90D2B

129) Event:ESQ_REQ length:38, at 262563 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_PVLAN(180), Opc:MTS_OPC_ETHPM_PORT_LOGICAL_CLEANUP(61447)

130) Event:ESQ_REQ length:38, at 262580 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_L2FM(221), Opc:MTS_OPC_ETHPM_PORT_LOGICAL_CLEANUP(61447)
    RRtoken:0xFEF90D29

131) Event:ESQ_RSP length:38, at 263888 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_L2FM(221), Opc:MTS_OPC_ETHPM_PORT_LOGICAL_CLEANUP(61447)
    RRtoken:0xFEF90D29

132) Event:ESQ_REQ length:38, at 263919 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ETH_PORT_SEC(191), Opc:MTS_OPC_ETHPM_PORT_LOGICAL_CLEANUP(61447)
    RRtoken:0xFEF90D28

133) Event:ESQ_REQ length:38, at 264035 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_MCECM(450), Opc:MTS_OPC_ETHPM_PORT_LOGICAL_CLEANUP(61447)

134) Event:ESQ_REQ length:38, at 264055 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_FIPSM(477), Opc:MTS_OPC_ETHPM_PORT_LOGICAL_CLEANUP(61447)
    RRtoken:0xFEF90D27

135) Event:ESQ_REQ length:38, at 264119 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ASSOC_MGR(714), Opc:MTS_OPC_FLUSH_SYNC(7)
    RRtoken:0xFEF90D26

136) FSM:<Ethernet106/1/3> Transition at 551571 usecs after Mon Dec 12 15:57:42 2022
    Previous state: [ETH_PORT_FSM_ST_WAIT_LOGICAL_DOWN]
    Triggered event: [ETH_PORT_FSM_EV_LOGICAL_DOWN_DONE]
    Next state: [ETH_PORT_FSM_ST_WAIT_PHASE2_CTRL_DOWN]

137) Event:ESQ_START length:38, at 551584 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq Type:SERIAL

138) Event:ESQ_REQ length:38, at 551602 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Phase2 Port down determine steps to skip

139) Event:ESQ_REQ length:38, at 551605 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Wait for LIM response

140) FSM:<Ethernet106/1/3> Transition at 551629 usecs after Mon Dec 12 15:57:42 2022
    Previous state: [ETH_PORT_FSM_ST_WAIT_PHASE2_CTRL_DOWN]
    Triggered event: [ETH_PORT_FSM_EV_LOGICAL_DOWN_DONE]
    Next state: [ETH_PORT_FSM_ST_WAIT_PROTOCOL_DOWN]

141) Event:ESQ_START length:38, at 551637 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq Type:SERIAL

142) Event:ESQ_REQ length:38, at 551673 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:determine steps to skip

143) Event:ESQ_REQ length:38, at 551698 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ETH_PORT_CHANNEL_MGR(378), Opc:MTS_OPC_ETHPM_PORT_CLEANUP(61444)
    RRtoken:0xFEF90D25

144) Event:ESQ_RSP length:38, at 551939 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_ETH_PORT_CHANNEL_MGR(378), Opc:MTS_OPC_ETHPM_PORT_CLEANUP(61444)
    RRtoken:0xFEF90D25

145) Event:ESQ_REQ length:38, at 551944 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_OSPF(320), Opc:MTS_OPC_ETHPM_PORT_CLEANUP(61444)

146) Event:ESQ_REQ length:38, at 552006 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_PVLAN(180), Opc:MTS_OPC_ETHPM_PORT_CLEANUP(61444)

147) Event:ESQ_REQ length:38, at 552069 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_STP(171), Opc:MTS_OPC_ETHPM_PORT_CLEANUP(61444)
    RRtoken:0x145FD7C3

148) Event:ESQ_RSP length:38, at 552192 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_STP(171), Opc:MTS_OPC_ETHPM_PORT_CLEANUP(61444)
    RRtoken:0x145FD7C3

149) FSM:<Ethernet106/1/3> Transition at 552256 usecs after Mon Dec 12 15:57:42 2022
    Previous state: [ETH_PORT_FSM_ST_WAIT_PROTOCOL_DOWN]
    Triggered event: [ETH_PORT_FSM_EV_PROTO_DOWN_DONE]
    Next state: [ETH_PORT_FSM_ST_PROTOCOL_DOWN]

150) FSM:<Ethernet106/1/3> Transition at 552260 usecs after Mon Dec 12 15:57:42 2022
    Previous state: [ETH_PORT_FSM_ST_PROTOCOL_DOWN]
    Triggered event: [ETH_PORT_FSM_EV_PHY_DOWN]
    Next state: [ETH_PORT_FSM_ST_WAIT_PHYSICAL_DOWN]

151) Event:ESQ_START length:38, at 552287 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq Type:SERIAL

152) Event:ESQ_REQ length:38, at 552313 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:determine steps to skip

153) Event:ESQ_REQ length:38, at 552336 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Send state change for shared intf if not sent in log down

154) Event:ESQ_REQ length:38, at 553519 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_REGISTRY(0), Opc:MTS_OPC_IM_PHY_LINK_STATE_CHANGE(62470)

155) Event:ESQ_REQ length:38, at 553578 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_PORT_CLIENT(181), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEF90D23

156) Event:ESQ_RSP length:38, at 575093 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_PORT_CLIENT(181), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEF90D23

157) Event:ESQ_REQ length:38, at 575100 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Wait for debounce logic

158) Event:ESQ_RSP length:38, at 675296 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Wait for debounce logic

159) Event:ESQ_REQ length:38, at 675336 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_SIM(1167), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)

160) Event:ESQ_REQ length:38, at 675416 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_FCOE(473), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)

161) Event:ESQ_REQ length:38, at 675469 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_DOT1X(208), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)

162) Event:ESQ_REQ length:38, at 675487 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_CTS(400), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEF90D20

163) Event:ESQ_REQ length:38, at 675557 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_MKA(1509), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEF90D1F

164) Event:ESQ_REQ length:38, at 675618 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_RPM_CTRL(348), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEF90D1E

165) Event:ESQ_RSP length:38, at 854162 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_RPM_CTRL(348), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEF90D1E

166) Event:ESQ_REQ length:38, at 854194 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ETH_PORT_SEC(191), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEF90D1D

167) Event:ESQ_REQ length:38, at 854318 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_L2FM(221), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEFE3F26

168) Event:ESQ_REQ length:38, at 854346 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ELTM(192), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEF90D1C

169) Event:ESQ_RSP length:38, at 156978 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_ELTM(192), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEF90D1C

170) Event:ESQ_REQ length:38, at 157019 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ETH_SPAN(174), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEF90D1B

171) Event:ESQ_RSP length:38, at 157366 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_ETH_SPAN(174), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEF90D1B

172) Event:ESQ_REQ length:38, at 157392 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_SATMGR(619), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEF90D1A

173) Event:ESQ_RSP length:38, at 157975 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_SATMGR(619), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEF90D1A

174) Event:ESQ_REQ length:38, at 157998 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ASSOC_MGR(714), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEF90D19

175) FSM:<Ethernet106/1/3> Transition at 158109 usecs after Mon Dec 12 15:57:43 2022
    Previous state: [ETH_PORT_FSM_ST_WAIT_PHYSICAL_DOWN]
    Triggered event: [ETH_PORT_FSM_EV_PHY_DOWN_DONE]
    Next state: [ETH_PORT_FSM_ST_WAIT_APPLY_CONFIG]

176) Event:ESQ_START length:38, at 158134 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq Type:SERIAL

177) Event:ESQ_REQ length:38, at 158257 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:determine steps to skip

178) Event:ESQ_REQ length:38, at 158303 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_REGISTRY(0), Opc:MTS_OPC_SYNC_MESSAGE(0)

179) Event:ESQ_REQ length:38, at 158306 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_DIAGCLIENT(366), Opc:MTS_OPC_GOLD_PP_TEST_STOP(59414)

180) Event:ESQ_REQ length:38, at 158307 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_PORT_CLIENT(181), Opc:MTS_OPC_LC_PORT_FCOT_SPROM_READ(8194)

181) Event:ESQ_REQ length:38, at 158309 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_PIXM_LOCAL(153), Opc:MTS_OPC_PIXM_SET_MULT_CBL_VLAN_PORT_STATE(69641)

182) Event:ESQ_REQ length:38, at 159468 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Sending MTS_OPC_LC_PORT_CLIENT_CFG and/or PARAM_CHANGE opcode

183) Event:ESQ_REQ length:38, at 159475 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ELTM(192), Opc:MTS_OPC_ELTM_INTF_FEATURE_REQ_RESP(72748)

184) Event:ESQ_REQ length:38, at 159479 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_VLAN_MGR(167), Opc:MTS_OPC_VLAN_MGR_GET_PORT_TRUNKING_MEMBERSHIP(23562)

185) Event:ESQ_REQ length:38, at 159481 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_DIAGCLIENT(366), Opc:MTS_OPC_GOLD_PP_TEST_RESUME(59417)

186) Event:ESQ_REQ length:38, at 159539 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_VLAN_MGR(167), Opc:MTS_OPC_VLAN_MGR_GET_PORT_MEMBERSHIP_2(23617)

187) Event:ESQ_RSP length:38, at 162016 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_VLAN_MGR(167), Opc:MTS_OPC_VLAN_MGR_GET_PORT_MEMBERSHIP_2(23617)

188) Event:ESQ_REQ length:38, at 162024 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ELTM(192), Opc:MTS_OPC_ELTM_INTF_FEATURE_REQ(72713)

189) Event:ESQ_REQ length:38, at 162027 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:SUCCESS path done, skip-to-end, feed EV_CFG_DONE

190) FSM:<Ethernet106/1/3> Transition at 162167 usecs after Mon Dec 12 15:57:43 2022
    Previous state: [ETH_PORT_FSM_ST_WAIT_APPLY_CONFIG]
    Triggered event: [ETH_PORT_FSM_EV_CFG_DONE]
    Next state: [ETH_PORT_FSM_ST_INIT_EVAL]

191) FSM:<Ethernet106/1/3> Transition at 162233 usecs after Mon Dec 12 15:57:43 2022
    Previous state: [ETH_PORT_FSM_ST_INIT_EVAL]
    Triggered event: [ETH_PORT_FSM_EV_IE_ADMIN_DOWN]
    Next state: [ETH_PORT_FSM_ST_DOWN]

    Curr state: [ETH_PORT_FSM_ST_DOWN]

 В данном случае хосты подключены DAC кабелями. Вряд ли тут что-то с контактом может быть.

 

Мне вот интересно, что может такое происходить, что это влияет только на физику 10GE интерфейсов находящихся в разных FEX? Это какой-то полтергейст.

frol13

frol13

Posted
Posted
5 часов назад, kostas сказал:

тк проблема только в одном влане и только на 10GE интерфейсах.

Я бы из общих соображений кабельное хозяйство не исключал. 10GE должны быть к физике чувствительны.

  • 1 month later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
 Share
Go to topic listing
×
×
  • Create New...
На сайте используются файлы cookie и сервисы аналитики для корректной работы форума и улучшения качества обслуживания. Продолжая использовать сайт, вы соглашаетесь с использованием файлов cookie и с Политикой конфиденциальности.