Jump to content
Калькуляторы

Пограничный шлюз, high interrupts rate

Прошу совета c интерпретацией количества прерываний на пограничном шлюзе.

 

Что имеем:

2 * Intel Xeon E5649, 2.53 ГГц, по 4ГБ Reg DDR3 на процессор

10GbE к аплинку через Intel 82599ES
4 * 1GbE в свою сеть через Intel 82576 (lacp)

 

linux 3.16, стандартное ядро deb 8

На бордере работает bird, сейчас full view от одного аплинка

абоненты на белых адресах

ipt_ratelimit, ipt_netflow

прерывания сетевых раскиданы по ядрам

HyperThreading выключен, governor  = performance

В sysctl оптимизации таймаутов и буферов памяти

Ring buffer на картах, обслуживающих трафик, выставлен в 4096, txqueulen 10000 

Трафик в ЧНН до 3 Гбит в одну сторону, 500-800 мбит в другую

 

есть совсем немножко нат, собственный офис, трафика <10 мбит в пике

 

В вечерние часы вижу 300-350к прерываний в системе при 7-9к context switches
И тут мне непонятно, это адекватная ситуация или число прерываний должно быть сильно меньше?

 

пакетная нагрузка в 7 вечера, в ЧНН на 10-15% выше

  bwm-ng v0.6 (probing every 0.500s), press 'h' for help
  input: /proc/net/dev type: rate
  /         iface                   Rx                   Tx                Total
  ==============================================================================
            eth10:       244011.90 P/s        130565.47 P/s        374577.37 P/s
             eth3:         1672.62 P/s          4367.06 P/s          6039.68 P/s
             eth2:            0.00 P/s             0.00 P/s             0.00 P/s
            bond0:       134738.09 P/s        238146.82 P/s        372884.91 P/s
             eth0:         2474.21 P/s          2908.73 P/s          5382.94 P/s
  ------------------------------------------------------------------------------
            total:       382896.81 P/s        375988.08 P/s        758884.90 P/s

 

perf top

   6,26%  [kernel]                               [k] ipt_do_table
   4,14%  [kernel]                               [k] _raw_spin_lock
   3,39%  [kernel]                               [k] irq_entries_start
   3,18%  [kernel]                               [k] do_csum
   3,02%  [kernel]                               [k] igb_poll
   2,79%  [kernel]                               [k] hash_net4_test
   2,47%  [kernel]                               [k] __nf_conntrack_find_get
   2,42%  [kernel]                               [k] fib_table_lookup
   2,24%  [kernel]                               [k] netflow_target
   2,08%  [kernel]                               [k] ixgbe_clean_rx_irq
   1,83%  [kernel]                               [k] ixgbe_poll
   1,75%  [kernel]                               [k] fib_get_table
   1,68%  [kernel]                               [k] igb_xmit_frame_ring
   1,42%  [kernel]                               [k] __build_skb
   1,40%  [kernel]                               [k] tcp_packet
   1,37%  [kernel]                               [k] nf_iterate
   1,36%  [kernel]                               [k] ip_set_test
   1,36%  [kernel]                               [k] skb_release_head_state
   1,27%  [kernel]                               [k] consume_skb
   1,20%  [kernel]                               [k] ratelimit_mt
   1,02%  [kernel]                               [k] __netif_receive_skb_core
   0,99%  [kernel]                               [k] ip_forward
   0,94%  [kernel]                               [k] ixgbe_xmit_frame_ring
   0,89%  [kernel]                               [k] check_leaf.isra.6
   0,87%  [kernel]                               [k] put_page
   0,86%  [kernel]                               [k] __x86_indirect_thunk_rax
   0,86%  [kernel]                               [k] ip_route_input_noref
   0,85%  [kernel]                               [k] ip_finish_output
   0,79%  [kernel]                               [k] build_skb
   0,72%  [kernel]                               [k] ip_rcv
   0,72%  [kernel]                               [k] nf_conntrack_in
   0,66%  [kernel]                               [k] __tick_nohz_idle_enter
   0,65%  [kernel]                               [k] fib_rules_lookup
   0,64%  [kernel]                               [k] _raw_read_lock
   0,64%  [kernel]                               [k] tcp_mt
   0,61%  [kernel]                               [k] kmem_cache_free
   0,60%  [kernel]                               [k] swiotlb_map_page
   0,59%  bird                                   [.] 0x000000000000b8fc
   0,58%  [kernel]                               [k] udp_mt
   0,57%  [kernel]                               [k] skb_release_data
   0,57%  [kernel]                               [k] kmem_cache_alloc
   0,54%  [kernel]                               [k] __dev_queue_xmit
   0,54%  [kernel]                               [k] __napi_complete
   0,53%  [kernel]                               [k] cpu_startup_entry
   0,53%  [kernel]                               [k] dql_completed
   0,51%  [kernel]                               [k] memcmp
   0,49%  [kernel]                               [k] native_read_tsc
   0,48%  [kernel]                               [k] igb_msix_ring
   0,46%  [kernel]                               [k] put_compound_page
   0,45%  [kernel]                               [k] iptable_mangle_hook



 

graph_interrupts_context_switches.png

Share this post


Link to post
Share on other sites

По perf top у вас все в норме, ни softirq, ни irqrestore в топе нет - значит система нагрузку прожевывает. IPTables свое отъедает, но приемлемо, spin_lock скорее всего тоже благодаря IPTables, но тоже не что-то ненормальное. Вот  do_csum - это интересно, сделайте-ка ethtool -S

Share this post


Link to post
Share on other sites
~# ethtool -S eth10 | grep -Evw 0
NIC statistics:
     rx_packets: 299957210773
     tx_packets: 167333742925
     rx_bytes: 367767869594335
     tx_bytes: 61494923738649
     rx_pkts_nic: 299957210061
     tx_pkts_nic: 167333741970
     rx_bytes_nic: 368967704454037
     tx_bytes_nic: 62327648896571
     lsc_int: 8
     broadcast: 23
     alloc_rx_page: 6154267
     fdir_match: 170665935563
     fdir_miss: 133364612674
     fdir_overflow: 410863
     tx_queue_0_packets: 28891980592
     tx_queue_0_bytes: 10685970850857
     tx_queue_1_packets: 27498041580
     tx_queue_1_bytes: 10162852915621
     tx_queue_2_packets: 27335983842
     tx_queue_2_bytes: 9960629662333
     tx_queue_3_packets: 28773211579
     tx_queue_3_bytes: 10520131379125
     tx_queue_4_packets: 27532420741
     tx_queue_4_bytes: 10161551175178
     tx_queue_5_packets: 27302104601
     tx_queue_5_bytes: 10003787761728
     rx_queue_0_packets: 51614362275
     rx_queue_0_bytes: 63364226109995
     rx_queue_1_packets: 49841962754
     rx_queue_1_bytes: 60995919894981
     rx_queue_2_packets: 49452395733
     rx_queue_2_bytes: 60479458941723
     rx_queue_3_packets: 50499798834
     rx_queue_3_bytes: 62333800123113
     rx_queue_4_packets: 49399170119
     rx_queue_4_bytes: 60535799041291
     rx_queue_5_packets: 49149521065
     rx_queue_5_bytes: 60058665491400

 

сетевые карты бондинга

~# ethtool -S eth6 | grep -Evw 0
NIC statistics:
     rx_packets: 47803703178
     tx_packets: 74807651019
     rx_bytes: 18924327887430
     tx_bytes: 91042111370196
     rx_broadcast: 201145
     tx_broadcast: 2095305
     rx_multicast: 63189
     tx_multicast: 63185
     multicast: 63189
     rx_crc_errors: 3
     rx_long_byte_count: 18924327887430
     rx_errors: 6
     tx_queue_0_packets: 12813339926
     tx_queue_0_bytes: 15485455469721
     tx_queue_1_packets: 12538971967
     tx_queue_1_bytes: 15196538664629
     tx_queue_2_packets: 12314760385
     tx_queue_2_bytes: 14925834940462
     tx_queue_3_packets: 12623790857
     tx_queue_3_bytes: 15468208980900
     tx_queue_4_packets: 12426793957
     tx_queue_4_bytes: 14960451187944
     tx_queue_5_packets: 12089993929
     tx_queue_5_bytes: 14690676241250
     rx_queue_0_packets: 8318443509
     rx_queue_0_bytes: 3244662445535
     rx_queue_1_packets: 7970165224
     rx_queue_1_bytes: 3264264848995
     rx_queue_2_packets: 7721128658
     rx_queue_2_bytes: 3033209849060
     rx_queue_3_packets: 8189127803
     rx_queue_3_bytes: 3131136547054
     rx_queue_4_packets: 7827302061
     rx_queue_4_bytes: 3054928067458
     rx_queue_5_packets: 7777535925
     rx_queue_5_bytes: 3004911316754
~# ethtool -S eth7 | grep -Evw 0
NIC statistics:
     rx_packets: 46869460541
     tx_packets: 73512292638
     rx_bytes: 17871713424629
     tx_bytes: 90661823178438
     rx_broadcast: 158190
     rx_multicast: 63190
     tx_multicast: 63186
     multicast: 63190
     rx_long_byte_count: 17871713424629
     tx_queue_0_packets: 12720376392
     tx_queue_0_bytes: 15669302646244
     tx_queue_1_packets: 12151107797
     tx_queue_1_bytes: 14909928240831
     tx_queue_2_packets: 12091186720
     tx_queue_2_bytes: 14847065453808
     tx_queue_3_packets: 12373762457
     tx_queue_3_bytes: 15326811694952
     tx_queue_4_packets: 12019573995
     tx_queue_4_bytes: 14809961912454
     tx_queue_5_packets: 12156337489
     tx_queue_5_bytes: 14793234150663
     rx_queue_0_packets: 8100246491
     rx_queue_0_bytes: 3120520109449
     rx_queue_1_packets: 7634454147
     rx_queue_1_bytes: 2889039982727
     rx_queue_2_packets: 7665176634
     rx_queue_2_bytes: 2856902457812
     rx_queue_3_packets: 7966193963
     rx_queue_3_bytes: 2937793647323
     rx_queue_4_packets: 7730087193
     rx_queue_4_bytes: 2907836055427
     rx_queue_5_packets: 7773356285
     rx_queue_5_bytes: 2972144206556
~# ethtool -S eth8 | grep -Evw 0
NIC statistics:
     rx_packets: 38418272175
     tx_packets: 73885253309
     rx_bytes: 14307097920076
     tx_bytes: 90667519760581
     rx_broadcast: 203518
     rx_multicast: 63190
     tx_multicast: 63185
     multicast: 63190
     rx_long_byte_count: 14307097920076
     tx_queue_0_packets: 12700502764
     tx_queue_0_bytes: 15538754826056
     tx_queue_1_packets: 12346182169
     tx_queue_1_bytes: 15035142899487
     tx_queue_2_packets: 12242994843
     tx_queue_2_bytes: 14828543653037
     tx_queue_3_packets: 12412076812
     tx_queue_3_bytes: 15289255041694
     tx_queue_4_packets: 12150924185
     tx_queue_4_bytes: 14884172161155
     tx_queue_5_packets: 12032572538
     tx_queue_5_bytes: 14783274895753
     rx_queue_0_packets: 6638242079
     rx_queue_0_bytes: 2466196814545
     rx_queue_1_packets: 6342521859
     rx_queue_1_bytes: 2375307212658
     rx_queue_2_packets: 6285574419
     rx_queue_2_bytes: 2296719119223
     rx_queue_3_packets: 6640808823
     rx_queue_3_bytes: 2449964960131
     rx_queue_4_packets: 6261530101
     rx_queue_4_bytes: 2273089070702
     rx_queue_5_packets: 6249594896
     rx_queue_5_bytes: 2292147654303
~# ethtool -S eth9 | grep -Evw 0
NIC statistics:
     rx_packets: 37878524889
     tx_packets: 73960236716
     rx_bytes: 13850014826513
     tx_bytes: 91015598892924
     rx_broadcast: 156212
     rx_multicast: 63190
     tx_multicast: 63185
     multicast: 63190
     rx_crc_errors: 9
     rx_long_byte_count: 13850014826513
     rx_errors: 17
     tx_queue_0_packets: 12768985871
     tx_queue_0_bytes: 15712006733389
     tx_queue_1_packets: 12183230243
     tx_queue_1_bytes: 14901258256197
     tx_queue_2_packets: 12192467007
     tx_queue_2_bytes: 14950712331623
     tx_queue_3_packets: 12432981271
     tx_queue_3_bytes: 15307822231330
     tx_queue_4_packets: 12089993510
     tx_queue_4_bytes: 14918445418368
     tx_queue_5_packets: 12292578818
     tx_queue_5_bytes: 14917782501297
     rx_queue_0_packets: 6501247023
     rx_queue_0_bytes: 2376872769386
     rx_queue_1_packets: 6198612038
     rx_queue_1_bytes: 2190368274272
     rx_queue_2_packets: 6247532633
     rx_queue_2_bytes: 2231574347257
     rx_queue_3_packets: 6566195981
     rx_queue_3_bytes: 2438590474001
     rx_queue_4_packets: 6220091816
     rx_queue_4_bytes: 2272657781605
     rx_queue_5_packets: 6144845398
     rx_queue_5_bytes: 2188437080436

 

Share this post


Link to post
Share on other sites
~# ethtool -k eth10
Features for eth10:
rx-checksumming: off
tx-checksumming: off
        tx-checksum-ipv4: off
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: off
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
        tx-tcp-segmentation: off
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp6-segmentation: off
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: on [fixed]

 

~# ethtool -k eth6
Features for eth6:
rx-checksumming: off
tx-checksumming: on
        tx-checksum-ipv4: on
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: on
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: on
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
        tx-tcp-segmentation: off
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp6-segmentation: off
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]

~# ethtool -k eth7
Features for eth7:
rx-checksumming: off
tx-checksumming: on
        tx-checksum-ipv4: on
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: on
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: on
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
        tx-tcp-segmentation: off
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp6-segmentation: off
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]

~# ethtool -k eth8
Features for eth8:
rx-checksumming: off
tx-checksumming: on
        tx-checksum-ipv4: on
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: on
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: on
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
        tx-tcp-segmentation: off
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp6-segmentation: off
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]

~# ethtool -k eth9
Features for eth9:
rx-checksumming: off
tx-checksumming: on
        tx-checksum-ipv4: on
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: on
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: on
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
        tx-tcp-segmentation: off
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp6-segmentation: off
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]

 

Share this post


Link to post
Share on other sites
5 часов назад, Mallorn сказал:

rx-checksumming: off

С умыслом отключено или забыли включить? Просто 3% на 12 ядрах на расчет контрольных сумм - это заметные затраты.

Share this post


Link to post
Share on other sites
В 26.12.2018 в 21:05, Mallorn сказал:

4 * 1GbE в свою сеть через Intel 82576 (lacp)

300к прервываний на 4 гигабита не так и много. Правда переключений контекста на примерно таком же трафике у меня на порядок меньше при равном количестве прерываний. Я бы предложил подумать на будущее отказаться от бондинга, оставив только десятки. Нагрузка на ЦП должна будет уменьшиться ощутимо. Ну и применение двухголовых систем на роутинге все же не всегда полезно, игры с NUMA не бесплатны.

Share this post


Link to post
Share on other sites
20 hours ago, jffulcrum said:

С умыслом отключено или забыли включить? Просто 3% на 12 ядрах на расчет контрольных сумм - это заметные затраты.

Скорее по глупости выключил, в /etc/rc.local было прописано

ethtool -K eth10 rx off tx off

и так для каждой карты

 

Теперь perf top выглядит так:

   6,43%  [kernel]                  [k] ipt_do_table
   4,39%  [kernel]                  [k] _raw_spin_lock
   4,20%  [kernel]                  [k] irq_entries_start
   3,21%  [kernel]                  [k] igb_poll
   2,84%  [kernel]                  [k] hash_net4_test
   2,39%  [kernel]                  [k] fib_table_lookup
   2,17%  [kernel]                  [k] netflow_target
   2,13%  [kernel]                  [k] __nf_conntrack_find_get
   2,12%  [kernel]                  [k] ixgbe_clean_rx_irq
   2,00%  [kernel]                  [k] ixgbe_poll
   1,78%  [kernel]                  [k] fib_get_table
   1,75%  [kernel]                  [k] igb_xmit_frame_ring
   1,47%  [kernel]                  [k] __build_skb
   1,46%  [kernel]                  [k] skb_release_head_state
   1,42%  [kernel]                  [k] nf_iterate
   1,41%  [kernel]                  [k] ip_set_test
   1,34%  [kernel]                  [k] tcp_packet
   1,27%  [kernel]                  [k] consume_skb
   1,17%  [kernel]                  [k] ratelimit_mt
   1,03%  [kernel]                  [k] __netif_receive_skb_core
   1,02%  [kernel]                  [k] ip_forward
   0,98%  [kernel]                  [k] ixgbe_xmit_frame_ring
   0,91%  [kernel]                  [k] put_page
   0,91%  [kernel]                  [k] __x86_indirect_thunk_rax
   0,89%  [kernel]                  [k] ip_route_input_noref
   0,89%  [kernel]                  [k] ip_finish_output
   0,80%  [kernel]                  [k] check_leaf.isra.6
   0,78%  [kernel]                  [k] build_skb
   0,77%  [kernel]                  [k] __tick_nohz_idle_enter
   0,75%  [kernel]                  [k] ip_rcv
   0,72%  [kernel]                  [k] nf_conntrack_in
   0,69%  [kernel]                  [k] fib_rules_lookup
   0,67%  [kernel]                  [k] _raw_read_lock
   0,65%  [kernel]                  [k] udp_mt
   0,62%  [kernel]                  [k] tcp_mt
   0,61%  [kernel]                  [k] kmem_cache_free
   0,60%  [kernel]                  [k] __napi_complete
   0,59%  [kernel]                  [k] kmem_cache_alloc
   0,59%  [kernel]                  [k] cpu_startup_entry
   0,59%  [kernel]                  [k] swiotlb_map_page
   0,57%  [kernel]                  [k] dql_completed
   0,56%  [kernel]                  [k] native_read_tsc
   0,56%  [kernel]                  [k] __dev_queue_xmit
   0,55%  bird                      [.] 0x000000000000b8fc
   0,53%  [kernel]                  [k] skb_release_data
   0,52%  [kernel]                  [k] igb_msix_ring

 

34 minutes ago, taf_321 said:

300к прервываний на 4 гигабита не так и много. Правда переключений контекста на примерно таком же трафике у меня на порядок меньше при равном количестве прерываний. Я бы предложил подумать на будущее отказаться от бондинга, оставив только десятки. Нагрузка на ЦП должна будет уменьшиться ощутимо. Ну и применение двухголовых систем на роутинге все же не всегда полезно, игры с NUMA не бесплатны.

В планах убрать бондинг, жду Cisco C4948E и вторую карту Intel X520 в пограничный.

По поводу context switches - у меня еще и tc там работал раньше, в ЧНН их количество поднималось до 50к. Уход на ipt_ratelimit позволил снизить эту цифру до 10-12к.В 

 

В целом, вы хотите сказать, что такое количество прерываний с учетом бондинга и двухпроцессорной системы вполне логичное?

Share this post


Link to post
Share on other sites
30 минут назад, Mallorn сказал:

В целом, вы хотите сказать, что такое количество прерываний с учетом бондинга и двухпроцессорной системы вполне логичное?

В целом да. Хотя цифра для context switches все равно какая-то завышенная. Посмотрел у себя на брасах, при примерно таком же трафике + NAT для клиентов + accel-ppp с PPPoE + шейпер на hfsc+iptables+ipset и все равно получается цифра в районе 2-3к

 

А, да, на роутерах и брасах ядро загружено с параметрами "spectre_v2=off nopti"

Share this post


Link to post
Share on other sites
В 27.12.2018 в 00:05, Mallorn сказал:

сейчас full view от одного аплинка

Зачем? Ради прикола FV?

 

Share this post


Link to post
Share on other sites
On 12/28/2018 at 11:12 AM, taf_321 said:

Ну и применение двухголовых систем на роутинге все же не всегда полезно, игры с NUMA не бесплатны.

т.е. однопортовые лучше ставить?

 

Share this post


Link to post
Share on other sites

Все же подразумевалась _однопроцессорная_ система. Количество ядер на процессор уже можно накручивать в рамках бюджета. Но тоже желательно смотреть, чтобы они равномерно распределялись между очередями сетевой карты.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this