Jump to content
Калькуляторы

Гадит карта в сеть?

Reply to this topic

kostas   

kostas
Posted · Report post

Приветствую всех.

 

Столкнулся с интересной и пакостной проблемой - как будто хост в сеть гадит и кладет весь L2 сегмент.

Опишу по порядку, чтобы сложилось четкое представление о конфигурации и самой проблеме.

 

Есть несколько десяткой хостов одной сети. Все хосты ранее были подключены к N3K, работали более года без проблем и впоследствии были перевезены и подключены к N7K тем же набором кабелей.

 

1. Коммутатор Cisco Nexus 7K + несколько FEX 2232

2. Серверы с картами X520DA2

3. DAC кабели 3 метра, разные - Huawei, Dlink, какие-то еще - сиска определяет их по разному, но все видит, линки в других вланах стабильные

4. Хосты подключены одним портом, туннели никакие на хостах не строятся

 

Есть влан (540) в котором происходит это проблема. В это влане порядка 40 хостов. Хосты расположены на разных FEX. Трафик в сегменте не большой.

 

Начиналось все так: В определенный момент времени начинали падать и сразу же подниматься линки у нескольких хостов, по очереди. В логах сиски выглядело это так: 

Sep 30 14:25:02 ds5 : 2022 Sep 30 14:25:01 MSK: %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet101/1/18 is down (Link failure)
Sep 30 14:25:04 ds5 : 2022 Sep 30 14:25:03 MSK: %ETHPORT-5-SPEED: Interface Ethernet103/1/4, operational speed changed to 1 Gbps
Sep 30 14:25:04 ds5 : 2022 Sep 30 14:25:03 MSK: %ETHPORT-5-IF_DUPLEX: Interface Ethernet103/1/4, operational duplex mode changed to Full
Sep 30 14:25:04 ds5 : 2022 Sep 30 14:25:03 MSK: %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet103/1/4, operational Receive Flow Control state changed to off
Sep 30 14:25:04 ds5 : 2022 Sep 30 14:25:03 MSK: %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet103/1/4, operational Transmit Flow Control state changed to on
Sep 30 14:25:04 ds5 : 2022 Sep 30 14:25:03 MSK: %ETHPORT-5-IF_UP: Interface Ethernet103/1/4 is up in mode access
Sep 30 14:25:04 ds5 : 2022 Sep 30 14:25:03 MSK: %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet101/1/18, operational Transmit Flow Control state changed to on
Sep 30 14:25:04 ds5 : 2022 Sep 30 14:25:03 MSK: %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet101/1/18, operational Receive Flow Control state changed to off
Sep 30 14:25:04 ds5 : 2022 Sep 30 14:25:03 MSK: %ETHPORT-5-SPEED: Interface Ethernet101/1/18, operational speed changed to 10 Gbps
Sep 30 14:25:04 ds5 : 2022 Sep 30 14:25:03 MSK: %ETHPORT-5-IF_DUPLEX: Interface Ethernet101/1/18, operational duplex mode changed to Full
Sep 30 14:25:04 ds5 : 2022 Sep 30 14:25:03 MSK: %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet101/1/18, operational Receive Flow Control state changed to off
Sep 30 14:25:04 ds5 : 2022 Sep 30 14:25:03 MSK: %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet101/1/18, operational Transmit Flow Control state changed to on
Sep 30 14:25:05 ds5 : 2022 Sep 30 14:25:04 MSK: %ETHPORT-5-IF_UP: Interface Ethernet101/1/18 is up in mode access
Sep 30 14:25:31 ds5 : 2022 Sep 30 14:25:30 MSK: %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet103/1/4 is down (Link failure)
Sep 30 14:25:32 ds5 : 2022 Sep 30 14:25:31 MSK: %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet101/1/28 is down (Link failure)
Sep 30 14:25:32 ds5 : 2022 Sep 30 14:25:31 MSK: %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet103/1/4, operational Transmit Flow Control state changed to on
Sep 30 14:25:32 ds5 : 2022 Sep 30 14:25:31 MSK: %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet103/1/4, operational Receive Flow Control state changed to off
Sep 30 14:25:34 ds5 : 2022 Sep 30 14:25:33 MSK: %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet101/1/28, operational Transmit Flow Control state changed to on
Sep 30 14:25:34 ds5 : 2022 Sep 30 14:25:33 MSK: %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet101/1/28, operational Receive Flow Control state changed to off
Sep 30 14:25:34 ds5 : 2022 Sep 30 14:25:33 MSK: %ETHPORT-5-SPEED: Interface Ethernet101/1/28, operational speed changed to 10 Gbps
Sep 30 14:25:34 ds5 : 2022 Sep 30 14:25:33 MSK: %ETHPORT-5-IF_DUPLEX: Interface Ethernet101/1/28, operational duplex mode changed to Full
Sep 30 14:25:34 ds5 : 2022 Sep 30 14:25:33 MSK: %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet101/1/28, operational Receive Flow Control state changed to off
Sep 30 14:25:34 ds5 : 2022 Sep 30 14:25:33 MSK: %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet101/1/28, operational Transmit Flow Control state changed to on
Sep 30 14:25:34 ds5 : 2022 Sep 30 14:25:33 MSK: %ETHPORT-5-IF_UP: Interface Ethernet101/1/28 is up in mode access

Методом тыка мы нашел, что именно с одного хоста начиналась вся эта свистопляска. Погасил порт этого хоста на коммутаторе и настала тишина. Больше месяца все работало без проблем. На хосте была переустановлена ОС (win), DAC заменен на новый, другой марки. Хост стоял без работы какое-то время и все было тихо, как на днях снова все моргнуло. Было принято решение о замене сетевой карты, но не успели. Сегодня ночью по заббиксу пропала часть хостов и больше не появилось. Как выяснилось, стали не доступны все хосты этой сети, кроме одного который был подключен на внутреннюю 1GE карту, в отличии от всех остальных. Более того, часть интерфейсов флапала, а часть нет, но трафик по ним не шел. MAC адресов в влане не было, кроме той одной доступной на 1GE.

Пошел выключать все интерфейсы по одному, пока не дошел одного, выключение которого привело сразу же к восстановлению работы всех хостов. Оставил его выключенным и до сих пор все работает.

 

Вопрос - что это может быть? Напомню, что DAC менял, ОС менял. Проблемные хосты работали с месяц без каких либо проблем.

 

 

Share this post

Link to post
Share on other sites

jffulcrum   

jffulcrum
Posted · Report post

Обычная проблема в наших бизнес-центрах - когда забыли на новом коммутаторе доступа наглухо выключить и запретить LLDP (а также ND, BFD, LACP, STP), и какой-нибудь шибко продвинутый роутер или сервер арендатора начинал с соседом в лице нашего коммутатора что-нибудь строить...

Share this post

Link to post
Share on other sites

kostas   

kostas
Posted · Report post

Тут нет коммутаторов, к портам подключены только серверы одни портом. В логах сиски все перестроения обычно оседают. Но самое главное, пострадал один L2 сегмент, но не полностью, а только хосты подключенные на 10GE, 1GE на штатном интерфейсе сервера в это время работал без проблем.

Share this post

Link to post
Share on other sites

[anp/hsw]   

[anp/hsw]
Posted · Report post

flow control-то отключите, вон сколько жалоб на него.

Это первое, что надо сделать в операторской сети. От него все беды.

Он может и не быть причиной ваших проблем, но добавлять масла в огонь.

Share this post

Link to post
Share on other sites

Gray swordsman   

Gray swordsman
Posted · Report post

Первое падение линка до проблемного хоста это следствие или причина ? если причина, то скорее всего запускается какое-то перестроение stp, если следствие то как мне кажется это какая-то l2 петля. В логе flow-control как мне кажется к данной проблеме не относится от слова совсем.

Share this post

Link to post
Share on other sites

roma33rus   

roma33rus
Posted · Report post
4 часа назад, Gray swordsman сказал:

Первое падение линка до проблемного хоста это следствие или причина ? если причина, то скорее всего запускается какое-то перестроение stp, если следствие то как мне кажется это какая-то l2 петля. В логе flow-control как мне кажется к данной проблеме не относится от слова совсем.


stp не выключает линки, а блокирует прохождение трафика.

Share this post

Link to post
Share on other sites

frol13   

frol13
Posted · Report post

Я может не совсем попаду в кон, но похожая проблема у нас была во внутренней сети в офисе. У нас оказалась банальщина - не было надежного контакта где-то в разъемах на линке. В результате многочисленных "отваливаний"- восстановлений падала вся таблица интерфейсов из-за многочисленных ее перестроений с последующим выходом ее за пределы лимита памяти, ей отведенной. 

Share this post

Link to post
Share on other sites

kostas   

kostas
Posted · Report post

Хосты подключены в разные FEX расширители, тч проблема с линейкой маловероятна, имхо.

Петли и STP тут неоткуда взяться - сервер оконечное устройство, без каких либо коммутаторов с единственным интерфейсом. И как уже упоминали STP Линк не кладет.

flow control попробую отключить, но сдается, что не с этим связано, тк проблема только в одном влане и только на 10GE интерфейсах. Все остальные вланы и 1GE линки живут как ни бывало.

Не знаю, падение линка - это следствие или причина. Оба конца сигнализируют о потери несущей.

 

Вот лог порта, который кладет все порты 10GE в одном влане 

>>>>FSM: <Ethernet106/1/3> has 191 logged transitions<<<<<

1) Event:ESQ_REQ length:38, at 175705 usecs after Sun Dec 11 13:12:06 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_PIXM_LOCAL(153), Opc:MTS_OPC_PIXM_SET_MULT_CBL_VLAN_PORT_STATE(69641)

2) Event:ESQ_REQ length:38, at 176537 usecs after Sun Dec 11 13:12:06 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Sending MTS_OPC_LC_PORT_CLIENT_CFG and/or PARAM_CHANGE opcode

3) Event:ESQ_REQ length:38, at 176543 usecs after Sun Dec 11 13:12:06 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ELTM(192), Opc:MTS_OPC_ELTM_INTF_FEATURE_REQ_RESP(72748)

4) Event:ESQ_REQ length:38, at 176547 usecs after Sun Dec 11 13:12:06 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_VLAN_MGR(167), Opc:MTS_OPC_VLAN_MGR_GET_PORT_TRUNKING_MEMBERSHIP(23562)

5) Event:ESQ_REQ length:38, at 176549 usecs after Sun Dec 11 13:12:06 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_DIAGCLIENT(366), Opc:MTS_OPC_GOLD_PP_TEST_RESUME(59417)

6) Event:ESQ_REQ length:38, at 176604 usecs after Sun Dec 11 13:12:06 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_VLAN_MGR(167), Opc:MTS_OPC_VLAN_MGR_GET_PORT_MEMBERSHIP_2(23617)

7) Event:ESQ_RSP length:38, at 178036 usecs after Sun Dec 11 13:12:06 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_VLAN_MGR(167), Opc:MTS_OPC_VLAN_MGR_GET_PORT_MEMBERSHIP_2(23617)

8) Event:ESQ_REQ length:38, at 178044 usecs after Sun Dec 11 13:12:06 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ELTM(192), Opc:MTS_OPC_ELTM_INTF_FEATURE_REQ(72713)

9) Event:ESQ_REQ length:38, at 178046 usecs after Sun Dec 11 13:12:06 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:SUCCESS path done, skip-to-end, feed EV_CFG_DONE

10) FSM:<Ethernet106/1/3> Transition at 178163 usecs after Sun Dec 11 13:12:06 2022
    Previous state: [ETH_PORT_FSM_ST_WAIT_APPLY_CONFIG]
    Triggered event: [ETH_PORT_FSM_EV_CFG_DONE]
    Next state: [ETH_PORT_FSM_ST_INIT_EVAL]

11) FSM:<Ethernet106/1/3> Transition at 178226 usecs after Sun Dec 11 13:12:06 2022
    Previous state: [ETH_PORT_FSM_ST_INIT_EVAL]
    Triggered event: [ETH_PORT_FSM_EV_IE_ADMIN_DOWN]
    Next state: [ETH_PORT_FSM_ST_DOWN]

12) FSM:<Ethernet106/1/3> Transition at 997753 usecs after Mon Dec 12 15:55:57 2022
    Previous state: [ETH_PORT_FSM_ST_DOWN]
    Triggered event: [ETH_PORT_FSM_EV_ADMIN_CFG_EVAL]
    Next state: [FSM_ST_NO_CHANGE]

13) FSM:<Ethernet106/1/3> Transition at 997814 usecs after Mon Dec 12 15:55:57 2022
    Previous state: [ETH_PORT_FSM_ST_DOWN]
    Triggered event: [ETH_PORT_FSM_EV_CFG_FLAP]
    Next state: [FSM_ST_NO_CHANGE]

14) FSM:<Ethernet106/1/3> Transition at 997821 usecs after Mon Dec 12 15:55:57 2022
    Previous state: [ETH_PORT_FSM_ST_DOWN]
    Triggered event: [ETH_PORT_FSM_EV_PHY_DOWN_DONE]
    Next state: [ETH_PORT_FSM_ST_WAIT_APPLY_CONFIG]

15) Event:ESQ_START length:38, at 997837 usecs after Mon Dec 12 15:55:57 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq Type:SERIAL

16) Event:ESQ_REQ length:38, at 997958 usecs after Mon Dec 12 15:55:57 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:determine steps to skip

17) Event:ESQ_REQ length:38, at 998003 usecs after Mon Dec 12 15:55:57 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_REGISTRY(0), Opc:MTS_OPC_SYNC_MESSAGE(0)

18) Event:ESQ_REQ length:38, at 998006 usecs after Mon Dec 12 15:55:57 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_DIAGCLIENT(366), Opc:MTS_OPC_GOLD_PP_TEST_STOP(59414)

19) Event:ESQ_REQ length:38, at 998007 usecs after Mon Dec 12 15:55:57 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_PORT_CLIENT(181), Opc:MTS_OPC_LC_PORT_FCOT_SPROM_READ(8194)

20) Event:ESQ_REQ length:38, at 998009 usecs after Mon Dec 12 15:55:57 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_PIXM_LOCAL(153), Opc:MTS_OPC_PIXM_SET_MULT_CBL_VLAN_PORT_STATE(69641)

21) Event:ESQ_REQ length:38, at 999668 usecs after Mon Dec 12 15:55:57 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Sending MTS_OPC_LC_PORT_CLIENT_CFG and/or PARAM_CHANGE opcode

22) Event:ESQ_REQ length:38, at 999697 usecs after Mon Dec 12 15:55:57 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ELTM(192), Opc:MTS_OPC_ELTM_INTF_FEATURE_REQ_RESP(72748)

23) Event:ESQ_REQ length:38, at 999701 usecs after Mon Dec 12 15:55:57 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_VLAN_MGR(167), Opc:MTS_OPC_VLAN_MGR_GET_PORT_TRUNKING_MEMBERSHIP(23562)

24) Event:ESQ_REQ length:38, at 999704 usecs after Mon Dec 12 15:55:57 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_DIAGCLIENT(366), Opc:MTS_OPC_GOLD_PP_TEST_RESUME(59417)

25) Event:ESQ_REQ length:38, at 999785 usecs after Mon Dec 12 15:55:57 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_VLAN_MGR(167), Opc:MTS_OPC_VLAN_MGR_GET_PORT_MEMBERSHIP_2(23617)

26) Event:ESQ_RSP length:38, at 1061 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_VLAN_MGR(167), Opc:MTS_OPC_VLAN_MGR_GET_PORT_MEMBERSHIP_2(23617)

27) Event:ESQ_REQ length:38, at 1069 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ELTM(192), Opc:MTS_OPC_ELTM_INTF_FEATURE_REQ(72713)

28) Event:ESQ_REQ length:38, at 1071 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:SUCCESS path done, skip-to-end, feed EV_CFG_DONE

29) FSM:<Ethernet106/1/3> Transition at 1269 usecs after Mon Dec 12 15:55:58 2022
    Previous state: [ETH_PORT_FSM_ST_WAIT_APPLY_CONFIG]
    Triggered event: [ETH_PORT_FSM_EV_CFG_DONE]
    Next state: [ETH_PORT_FSM_ST_INIT_EVAL]

30) FSM:<Ethernet106/1/3> Transition at 1548 usecs after Mon Dec 12 15:55:58 2022
    Previous state: [ETH_PORT_FSM_ST_INIT_EVAL]
    Triggered event: [ETH_PORT_FSM_EV_IE_PORT_INIT]
    Next state: [ETH_PORT_FSM_ST_SPAN_EVAL]

31) FSM:<Ethernet106/1/3> Transition at 1551 usecs after Mon Dec 12 15:55:58 2022
    Previous state: [ETH_PORT_FSM_ST_SPAN_EVAL]
    Triggered event: [ETH_PORT_FSM_EV_NON_SPAN_DEST]
    Next state: [ETH_PORT_FSM_ST_WAIT_PRE_CFG]

32) Event:ESQ_START length:38, at 1575 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq Type:SERIAL

33) Event:ESQ_REQ length:38, at 1605 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:determine steps to skip

34) Event:ESQ_REQ length:38, at 1624 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Release pacer credits

35) Event:ESQ_REQ length:38, at 1628 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:check for issu credits

36) Event:ESQ_REQ length:38, at 1649 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:check_1 if port is eventually going to be down

37) Event:ESQ_REQ length:38, at 1653 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_OHMS(1011), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)

38) Event:ESQ_REQ length:38, at 1671 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_SATMGR(619), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
    RRtoken:0xFEF90D6B

39) Event:ESQ_RSP length:38, at 2445 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_SATMGR(619), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
    RRtoken:0xFEF90D6B

40) Event:ESQ_REQ length:38, at 2477 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_SIM(1167), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)

41) Event:ESQ_REQ length:38, at 2726 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_FCOE(473), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)

42) Event:ESQ_REQ length:38, at 2746 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ETH_SPAN(174), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
    RRtoken:0xFEF90D69

43) Event:ESQ_RSP length:38, at 3094 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_ETH_SPAN(174), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
    RRtoken:0xFEF90D69

44) Event:ESQ_REQ length:38, at 3122 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_CTS(400), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
    RRtoken:0xFEF90D68

45) Event:ESQ_REQ length:38, at 3242 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_MKA(1509), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
    RRtoken:0xFEF90D67

46) Event:ESQ_REQ length:38, at 3372 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_DOT1X(208), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)

47) Event:ESQ_REQ length:38, at 3389 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ELTM(192), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
    RRtoken:0xFEF90D65

48) Event:ESQ_RSP length:38, at 303531 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_ELTM(192), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
    RRtoken:0xFEF90D65

49) Event:ESQ_REQ length:38, at 303578 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_L2FM(221), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)

50) Event:ESQ_REQ length:38, at 303610 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ETH_PORT_SEC(191), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
    RRtoken:0xFEF90D64

51) Event:ESQ_REQ length:38, at 303713 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_QOS_MANAGER(59), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)

52) Event:ESQ_REQ length:38, at 303731 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_RPM_CTRL(348), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
    RRtoken:0xFEF90D63

53) Event:ESQ_RSP length:38, at 605136 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_RPM_CTRL(348), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
    RRtoken:0xFEF90D63

54) Event:ESQ_REQ length:38, at 605168 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ASSOC_MGR(714), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
    RRtoken:0xFEF90D62

55) Event:ESQ_REQ length:38, at 605294 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:check_2 if port is eventually going to be down

56) Event:ESQ_REQ length:38, at 605337 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Pacer Check

57) Event:ESQ_REQ length:38, at 605363 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:PORT_CMDS Programming

58) Event:ESQ_REQ length:38, at 605385 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:check_3 if port is eventually going to be down

59) Event:ESQ_REQ length:38, at 605402 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_FWM(602), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
    RRtoken:0xFEF90D61

60) Event:ESQ_RSP length:38, at 817213 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_FWM(602), Opc:MTS_OPC_ETHPM_PORT_PRE_CFG(61441)
    RRtoken:0xFEF90D61

61) Event:ESQ_REQ length:38, at 817314 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_PORT_CLIENT(181), Opc:MTS_OPC_ETHPM_LCP_SET_PORT_CFG(61443)
    RRtoken:0xFEF90D60

62) Event:ESQ_RSP length:38, at 845833 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_PORT_CLIENT(181), Opc:MTS_OPC_ETHPM_LCP_SET_PORT_CFG(61443)
    RRtoken:0xFEF90D60

63) Event:ESQ_REQ length:38, at 845861 usecs after Mon Dec 12 15:55:58 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Pacer Reset

64) FSM:<Ethernet106/1/3> Transition at 845927 usecs after Mon Dec 12 15:55:58 2022
    Previous state: [ETH_PORT_FSM_ST_WAIT_PRE_CFG]
    Triggered event: [ETH_PORT_FSM_EV_PRE_CFG_DONE]
    Next state: [ETH_PORT_FSM_ST_LINK_INIT]

65) FSM:<Ethernet106/1/3> Transition at 109257 usecs after Mon Dec 12 15:55:59 2022
    Previous state: [ETH_PORT_FSM_ST_LINK_INIT]
    Triggered event: [ETH_PORT_FSM_EV_LINK_UP]
    Next state: [ETH_PORT_FSM_ST_WAIT_BRINGUP]

66) Event:ESQ_START length:38, at 109367 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq Type:SERIAL

67) Event:ESQ_REQ length:38, at 109409 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:determine steps to skip

68) Event:ESQ_REQ length:38, at 109413 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:check for issu

69) Event:ESQ_REQ length:38, at 109414 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:check for pacer credits

70) Event:ESQ_REQ length:38, at 109476 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_PIXM_LOCAL(153), Opc:MTS_OPC_PIXM_SET_MULT_CBL_VLAN_PORT_STATE(69641)

71) Event:ESQ_RSP length:38, at 111602 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_PIXM_LOCAL(153), Opc:MTS_OPC_PIXM_SET_MULT_CBL_VLAN_PORT_STATE(69641)

72) Event:ESQ_REQ length:38, at 111629 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ASSOC_MGR(714), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)
    RRtoken:0xFEF90D5F

73) Event:ESQ_REQ length:38, at 112720 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_REGISTRY(0), Opc:MTS_OPC_IM_PHY_LINK_STATE_CHANGE(62470)

74) Event:ESQ_REQ length:38, at 112771 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_LLDP(456), Opc:MTS_OPC_DCX_CONFIG_FEATURE(96271)

75) Event:ESQ_REQ length:38, at 112796 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_CTS(400), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)
    RRtoken:0xFEF90D5E

76) Event:ESQ_REQ length:38, at 124672 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_MKA(1509), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)
    RRtoken:0xFEF90D5D

77) Event:ESQ_REQ length:38, at 124769 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_DOT1X(208), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)

78) Event:ESQ_REQ length:38, at 124820 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_FCOE(473), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)

79) Event:ESQ_REQ length:38, at 124824 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_PORT_CLIENT(181), Opc:MTS_OPC_ETHPM_LCP_SET_PORT_CFG(61443)

80) Event:ESQ_REQ length:38, at 124826 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_VLAN_MGR(167), Opc:MTS_OPC_VLAN_MGR_GET_PORT_TRUNKING_MEMBERSHIP(23562)

81) Event:ESQ_REQ length:38, at 124866 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_VLAN_MGR(167), Opc:MTS_OPC_VLAN_MGR_GET_PORT_MEMBERSHIP(23560)

82) Event:ESQ_RSP length:38, at 413650 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_VLAN_MGR(167), Opc:MTS_OPC_VLAN_MGR_GET_PORT_MEMBERSHIP(23560)

83) Event:ESQ_REQ length:38, at 413676 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ELTM(192), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)
    RRtoken:0xFEF90D59

84) Event:ESQ_RSP length:38, at 414045 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_ELTM(192), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)
    RRtoken:0xFEF90D59

85) Event:ESQ_REQ length:38, at 414119 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ETH_PORT_CHANNEL_MGR(378), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)
    RRtoken:0x145F60F7

86) Event:ESQ_RSP length:38, at 414275 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_ETH_PORT_CHANNEL_MGR(378), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)
    RRtoken:0x145F60F7

87) Event:ESQ_REQ length:38, at 414339 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_PVLAN(180), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)

88) Event:ESQ_REQ length:38, at 414405 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_IPQOS_MGR(377), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)
    RRtoken:0xFEF90D57

89) Event:ESQ_RSP length:38, at 415298 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_IPQOS_MGR(377), Opc:MTS_OPC_ETHPM_PORT_BRINGUP(61442)
    RRtoken:0xFEF90D57

90) Event:ESQ_REQ length:38, at 415319 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_PORT_CLIENT(181), Opc:MTS_OPC_LC_PORT_CLIENT_CONFIG(8186)

91) Event:ESQ_REQ length:38, at 415333 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ELTM(192), Opc:MTS_OPC_ETHPM_PORT_BRINGUP_LACP_INDIVIDUAL(61523)

92) FSM:<Ethernet106/1/3> Transition at 415389 usecs after Mon Dec 12 15:55:59 2022
    Previous state: [ETH_PORT_FSM_ST_WAIT_BRINGUP]
    Triggered event: [ETH_PORT_FSM_EV_BRINGUP_DONE]
    Next state: [ETH_PORT_FSM_ST_WAIT_PHASE1_CTRL_UP]

93) Event:ESQ_START length:38, at 415418 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq Type:SERIAL

94) Event:ESQ_REQ length:38, at 415446 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Port up determine steps to skip

95) Event:ESQ_REQ length:38, at 415471 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:check_1 if port is eventually going to be down

96) Event:ESQ_REQ length:38, at 415494 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:PORT_CMDS Programming

97) Event:ESQ_REQ length:38, at 415513 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Sending state up notif to shared vdc

98) Event:ESQ_REQ length:38, at 415521 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_LIM(1237), Opc:MTS_OPC_ETHPM_LIM_REQ_RSP(61504)

99) FSM:<Ethernet106/1/3> Transition at 415548 usecs after Mon Dec 12 15:55:59 2022
    Previous state: [ETH_PORT_FSM_ST_WAIT_PHASE1_CTRL_UP]
    Triggered event: [ETH_PORT_FSM_EV_START_PRE_LOGICAL_BRINGUP]
    Next state: [ETH_PORT_FSM_ST_WAIT_PRE_LOGICAL_UP]

100) Event:ESQ_START length:38, at 415566 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq Type:SERIAL

101) Event:ESQ_REQ length:38, at 415610 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:determine steps to skip

102) Event:ESQ_REQ length:38, at 415616 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Waiting to bring Peer Link up

103) FSM:<Ethernet106/1/3> Transition at 415640 usecs after Mon Dec 12 15:55:59 2022
    Previous state: [ETH_PORT_FSM_ST_WAIT_PRE_LOGICAL_UP]
    Triggered event: [ETH_PORT_FSM_EV_PRE_LOGICAL_DONE]
    Next state: [FSM_ST_NO_CHANGE]

104) FSM:<Ethernet106/1/3> Transition at 416077 usecs after Mon Dec 12 15:55:59 2022
    Previous state: [ETH_PORT_FSM_ST_WAIT_PRE_LOGICAL_UP]
    Triggered event: [ETH_PORT_FSM_EV_START_LOGICAL_UP]
    Next state: [ETH_PORT_FSM_ST_WAIT_LOGICAL_UP]

105) FSM:<Ethernet106/1/3> Transition at 416081 usecs after Mon Dec 12 15:55:59 2022
    Previous state: [ETH_PORT_FSM_ST_WAIT_LOGICAL_UP]
    Triggered event: [ETH_PORT_FSM_EV_ACCESS_UP]
    Next state: [ETH_PORT_FSM_ST_WAIT_PHASE2_CTRL_UP]

106) Event:ESQ_START length:38, at 416097 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq Type:SERIAL

107) Event:ESQ_REQ length:38, at 416107 usecs after Mon Dec 12 15:55:59 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Phase2 Port up determine steps to skip

108) FSM:<Ethernet106/1/3> Transition at 416131 usecs after Mon Dec 12 15:55:59 2022
    Previous state: [ETH_PORT_FSM_ST_WAIT_PHASE2_CTRL_UP]
    Triggered event: [ETH_PORT_FSM_EV_ACCESS_UP]
    Next state: [ETH_PORT_FSM_ST_L2_UP]

109) FSM:<Ethernet106/1/3> Transition at 247300 usecs after Mon Dec 12 15:57:42 2022
    Previous state: [ETH_PORT_FSM_ST_L2_UP]
    Triggered event: [ETH_PORT_FSM_EV_ADMIN_CFG_EVAL]
    Next state: [FSM_ST_NO_CHANGE]

110) FSM:<Ethernet106/1/3> Transition at 247341 usecs after Mon Dec 12 15:57:42 2022
    Previous state: [ETH_PORT_FSM_ST_L2_UP]
    Triggered event: [ETH_PORT_FSM_EV_CFG_FLAP]
    Next state: [FSM_ST_NO_CHANGE]

111) Event:ESQ_START length:38, at 247365 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq Type:SERIAL

112) Event:ESQ_REQ length:38, at 247390 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Graceful shut steps to skip

113) Event:ESQ_REQ length:38, at 247422 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq: create graceful_down req payload

114) Event:ESQ_REQ length:38, at 248432 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_REGISTRY(0), Opc:MTS_OPC_IM_L3_IF_GRACEFUL_DOWN(62520)
    RRtoken:0x145FD764

115) Event:ESQ_RSP length:38, at 248535 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_REGISTRY(0), Opc:MTS_OPC_IM_L3_IF_GRACEFUL_DOWN(62520)
    RRtoken:0x145FD764

116) Event:ESQ_REQ length:38, at 248570 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_PORT_CLIENT(181), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEFEEEB9

117) Event:ESQ_REQ length:38, at 248604 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Satellite port down pacer request

118) Event:ESQ_REQ length:38, at 249553 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:send state change

119) FSM:<Ethernet106/1/3> Transition at 249639 usecs after Mon Dec 12 15:57:42 2022
    Previous state: [ETH_PORT_FSM_ST_L2_UP]
    Triggered event: [ETH_PORT_FSM_EV_PHASE1_CTRL_DOWN]
    Next state: [FSM_ST_NO_CHANGE]

120) Event:ESQ_START length:38, at 249686 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq Type:SERIAL

121) Event:ESQ_REQ length:38, at 249712 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Phase1 port down determine steps to skip

122) Event:ESQ_REQ length:38, at 249718 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_LIM(1237), Opc:MTS_OPC_ETHPM_LIM_REQ_RSP(61504)

123) FSM:<Ethernet106/1/3> Transition at 249756 usecs after Mon Dec 12 15:57:42 2022
    Previous state: [ETH_PORT_FSM_ST_L2_UP]
    Triggered event: [ETH_PORT_FSM_EV_LOGICAL_DOWN]
    Next state: [ETH_PORT_FSM_ST_WAIT_LOGICAL_DOWN]

124) Event:ESQ_START length:38, at 250300 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq Type:SERIAL

125) Event:ESQ_REQ length:38, at 250331 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_STP(171), Opc:MTS_OPC_ETHPM_PORT_LOGICAL_CLEANUP(61447)
    RRtoken:0xFEF90D2C

126) Event:ESQ_RSP length:38, at 260949 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_STP(171), Opc:MTS_OPC_ETHPM_PORT_LOGICAL_CLEANUP(61447)
    RRtoken:0xFEF90D2C

127) Event:ESQ_REQ length:38, at 260982 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_RPM_CTRL(348), Opc:MTS_OPC_ETHPM_PORT_LOGICAL_CLEANUP(61447)
    RRtoken:0xFEF90D2B

128) Event:ESQ_RSP length:38, at 262488 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_RPM_CTRL(348), Opc:MTS_OPC_ETHPM_PORT_LOGICAL_CLEANUP(61447)
    RRtoken:0xFEF90D2B

129) Event:ESQ_REQ length:38, at 262563 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_PVLAN(180), Opc:MTS_OPC_ETHPM_PORT_LOGICAL_CLEANUP(61447)

130) Event:ESQ_REQ length:38, at 262580 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_L2FM(221), Opc:MTS_OPC_ETHPM_PORT_LOGICAL_CLEANUP(61447)
    RRtoken:0xFEF90D29

131) Event:ESQ_RSP length:38, at 263888 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_L2FM(221), Opc:MTS_OPC_ETHPM_PORT_LOGICAL_CLEANUP(61447)
    RRtoken:0xFEF90D29

132) Event:ESQ_REQ length:38, at 263919 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ETH_PORT_SEC(191), Opc:MTS_OPC_ETHPM_PORT_LOGICAL_CLEANUP(61447)
    RRtoken:0xFEF90D28

133) Event:ESQ_REQ length:38, at 264035 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_MCECM(450), Opc:MTS_OPC_ETHPM_PORT_LOGICAL_CLEANUP(61447)

134) Event:ESQ_REQ length:38, at 264055 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_FIPSM(477), Opc:MTS_OPC_ETHPM_PORT_LOGICAL_CLEANUP(61447)
    RRtoken:0xFEF90D27

135) Event:ESQ_REQ length:38, at 264119 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ASSOC_MGR(714), Opc:MTS_OPC_FLUSH_SYNC(7)
    RRtoken:0xFEF90D26

136) FSM:<Ethernet106/1/3> Transition at 551571 usecs after Mon Dec 12 15:57:42 2022
    Previous state: [ETH_PORT_FSM_ST_WAIT_LOGICAL_DOWN]
    Triggered event: [ETH_PORT_FSM_EV_LOGICAL_DOWN_DONE]
    Next state: [ETH_PORT_FSM_ST_WAIT_PHASE2_CTRL_DOWN]

137) Event:ESQ_START length:38, at 551584 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq Type:SERIAL

138) Event:ESQ_REQ length:38, at 551602 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Phase2 Port down determine steps to skip

139) Event:ESQ_REQ length:38, at 551605 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Wait for LIM response

140) FSM:<Ethernet106/1/3> Transition at 551629 usecs after Mon Dec 12 15:57:42 2022
    Previous state: [ETH_PORT_FSM_ST_WAIT_PHASE2_CTRL_DOWN]
    Triggered event: [ETH_PORT_FSM_EV_LOGICAL_DOWN_DONE]
    Next state: [ETH_PORT_FSM_ST_WAIT_PROTOCOL_DOWN]

141) Event:ESQ_START length:38, at 551637 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq Type:SERIAL

142) Event:ESQ_REQ length:38, at 551673 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:determine steps to skip

143) Event:ESQ_REQ length:38, at 551698 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ETH_PORT_CHANNEL_MGR(378), Opc:MTS_OPC_ETHPM_PORT_CLEANUP(61444)
    RRtoken:0xFEF90D25

144) Event:ESQ_RSP length:38, at 551939 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_ETH_PORT_CHANNEL_MGR(378), Opc:MTS_OPC_ETHPM_PORT_CLEANUP(61444)
    RRtoken:0xFEF90D25

145) Event:ESQ_REQ length:38, at 551944 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_OSPF(320), Opc:MTS_OPC_ETHPM_PORT_CLEANUP(61444)

146) Event:ESQ_REQ length:38, at 552006 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_PVLAN(180), Opc:MTS_OPC_ETHPM_PORT_CLEANUP(61444)

147) Event:ESQ_REQ length:38, at 552069 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_STP(171), Opc:MTS_OPC_ETHPM_PORT_CLEANUP(61444)
    RRtoken:0x145FD7C3

148) Event:ESQ_RSP length:38, at 552192 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_STP(171), Opc:MTS_OPC_ETHPM_PORT_CLEANUP(61444)
    RRtoken:0x145FD7C3

149) FSM:<Ethernet106/1/3> Transition at 552256 usecs after Mon Dec 12 15:57:42 2022
    Previous state: [ETH_PORT_FSM_ST_WAIT_PROTOCOL_DOWN]
    Triggered event: [ETH_PORT_FSM_EV_PROTO_DOWN_DONE]
    Next state: [ETH_PORT_FSM_ST_PROTOCOL_DOWN]

150) FSM:<Ethernet106/1/3> Transition at 552260 usecs after Mon Dec 12 15:57:42 2022
    Previous state: [ETH_PORT_FSM_ST_PROTOCOL_DOWN]
    Triggered event: [ETH_PORT_FSM_EV_PHY_DOWN]
    Next state: [ETH_PORT_FSM_ST_WAIT_PHYSICAL_DOWN]

151) Event:ESQ_START length:38, at 552287 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq Type:SERIAL

152) Event:ESQ_REQ length:38, at 552313 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:determine steps to skip

153) Event:ESQ_REQ length:38, at 552336 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Send state change for shared intf if not sent in log down

154) Event:ESQ_REQ length:38, at 553519 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_REGISTRY(0), Opc:MTS_OPC_IM_PHY_LINK_STATE_CHANGE(62470)

155) Event:ESQ_REQ length:38, at 553578 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_PORT_CLIENT(181), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEF90D23

156) Event:ESQ_RSP length:38, at 575093 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_PORT_CLIENT(181), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEF90D23

157) Event:ESQ_REQ length:38, at 575100 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Wait for debounce logic

158) Event:ESQ_RSP length:38, at 675296 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Wait for debounce logic

159) Event:ESQ_REQ length:38, at 675336 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_SIM(1167), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)

160) Event:ESQ_REQ length:38, at 675416 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_FCOE(473), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)

161) Event:ESQ_REQ length:38, at 675469 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_DOT1X(208), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)

162) Event:ESQ_REQ length:38, at 675487 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_CTS(400), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEF90D20

163) Event:ESQ_REQ length:38, at 675557 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_MKA(1509), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEF90D1F

164) Event:ESQ_REQ length:38, at 675618 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_RPM_CTRL(348), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEF90D1E

165) Event:ESQ_RSP length:38, at 854162 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_RPM_CTRL(348), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEF90D1E

166) Event:ESQ_REQ length:38, at 854194 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ETH_PORT_SEC(191), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEF90D1D

167) Event:ESQ_REQ length:38, at 854318 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_L2FM(221), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEFE3F26

168) Event:ESQ_REQ length:38, at 854346 usecs after Mon Dec 12 15:57:42 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ELTM(192), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEF90D1C

169) Event:ESQ_RSP length:38, at 156978 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_ELTM(192), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEF90D1C

170) Event:ESQ_REQ length:38, at 157019 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ETH_SPAN(174), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEF90D1B

171) Event:ESQ_RSP length:38, at 157366 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_ETH_SPAN(174), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEF90D1B

172) Event:ESQ_REQ length:38, at 157392 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_SATMGR(619), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEF90D1A

173) Event:ESQ_RSP length:38, at 157975 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_SATMGR(619), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEF90D1A

174) Event:ESQ_REQ length:38, at 157998 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ASSOC_MGR(714), Opc:MTS_OPC_ETHPM_PORT_PHY_CLEANUP(61446)
    RRtoken:0xFEF90D19

175) FSM:<Ethernet106/1/3> Transition at 158109 usecs after Mon Dec 12 15:57:43 2022
    Previous state: [ETH_PORT_FSM_ST_WAIT_PHYSICAL_DOWN]
    Triggered event: [ETH_PORT_FSM_EV_PHY_DOWN_DONE]
    Next state: [ETH_PORT_FSM_ST_WAIT_APPLY_CONFIG]

176) Event:ESQ_START length:38, at 158134 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq Type:SERIAL

177) Event:ESQ_REQ length:38, at 158257 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:determine steps to skip

178) Event:ESQ_REQ length:38, at 158303 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_REGISTRY(0), Opc:MTS_OPC_SYNC_MESSAGE(0)

179) Event:ESQ_REQ length:38, at 158306 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_DIAGCLIENT(366), Opc:MTS_OPC_GOLD_PP_TEST_STOP(59414)

180) Event:ESQ_REQ length:38, at 158307 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_PORT_CLIENT(181), Opc:MTS_OPC_LC_PORT_FCOT_SPROM_READ(8194)

181) Event:ESQ_REQ length:38, at 158309 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_PIXM_LOCAL(153), Opc:MTS_OPC_PIXM_SET_MULT_CBL_VLAN_PORT_STATE(69641)

182) Event:ESQ_REQ length:38, at 159468 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:Sending MTS_OPC_LC_PORT_CLIENT_CFG and/or PARAM_CHANGE opcode

183) Event:ESQ_REQ length:38, at 159475 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ELTM(192), Opc:MTS_OPC_ELTM_INTF_FEATURE_REQ_RESP(72748)

184) Event:ESQ_REQ length:38, at 159479 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_VLAN_MGR(167), Opc:MTS_OPC_VLAN_MGR_GET_PORT_TRUNKING_MEMBERSHIP(23562)

185) Event:ESQ_REQ length:38, at 159481 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_DIAGCLIENT(366), Opc:MTS_OPC_GOLD_PP_TEST_RESUME(59417)

186) Event:ESQ_REQ length:38, at 159539 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_VLAN_MGR(167), Opc:MTS_OPC_VLAN_MGR_GET_PORT_MEMBERSHIP_2(23617)

187) Event:ESQ_RSP length:38, at 162016 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_RX] Src:MTS_SAP_VLAN_MGR(167), Opc:MTS_OPC_VLAN_MGR_GET_PORT_MEMBERSHIP_2(23617)

188) Event:ESQ_REQ length:38, at 162024 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    [E_MTS_TX] Dst:MTS_SAP_ELTM(192), Opc:MTS_OPC_ELTM_INTF_FEATURE_REQ(72713)

189) Event:ESQ_REQ length:38, at 162027 usecs after Mon Dec 12 15:57:43 2022
    Instance:526975104, Seq Id:0x1, Ret:SUCCESS
    Seq:SUCCESS path done, skip-to-end, feed EV_CFG_DONE

190) FSM:<Ethernet106/1/3> Transition at 162167 usecs after Mon Dec 12 15:57:43 2022
    Previous state: [ETH_PORT_FSM_ST_WAIT_APPLY_CONFIG]
    Triggered event: [ETH_PORT_FSM_EV_CFG_DONE]
    Next state: [ETH_PORT_FSM_ST_INIT_EVAL]

191) FSM:<Ethernet106/1/3> Transition at 162233 usecs after Mon Dec 12 15:57:43 2022
    Previous state: [ETH_PORT_FSM_ST_INIT_EVAL]
    Triggered event: [ETH_PORT_FSM_EV_IE_ADMIN_DOWN]
    Next state: [ETH_PORT_FSM_ST_DOWN]

    Curr state: [ETH_PORT_FSM_ST_DOWN]

 В данном случае хосты подключены DAC кабелями. Вряд ли тут что-то с контактом может быть.

 

Мне вот интересно, что может такое происходить, что это влияет только на физику 10GE интерфейсов находящихся в разных FEX? Это какой-то полтергейст.

Share this post

Link to post
Share on other sites

frol13   

frol13
Posted · Report post
5 часов назад, kostas сказал:

тк проблема только в одном влане и только на 10GE интерфейсах.

Я бы из общих соображений кабельное хозяйство не исключал. 10GE должны быть к физике чувствительны.

Share this post

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
Followers 0
Go to topic listing Активное оборудование Ethernet, IP, MPLS, SDN/NFV...