1.. SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2
3=================================================
4Mellanox ConnectX(R) mlx5 core VPI Network Driver
5=================================================
6
7Copyright (c) 2019, Mellanox Technologies LTD.
8
9Contents
10========
11
12- `Enabling the driver and kconfig options`_
13- `Devlink info`_
14- `Devlink parameters`_
15- `Bridge offload`_
16- `mlx5 subfunction`_
17- `mlx5 function attributes`_
18- `Devlink health reporters`_
19- `mlx5 tracepoints`_
20
21Enabling the driver and kconfig options
22=======================================
23
24| mlx5 core is modular and most of the major mlx5 core driver features can be selected (compiled in/out)
25| at build time via kernel Kconfig flags.
26| Basic features, ethernet net device rx/tx offloads and XDP, are available with the most basic flags
27| CONFIG_MLX5_CORE=y/m and CONFIG_MLX5_CORE_EN=y.
28| For the list of advanced features please see below.
29
30**CONFIG_MLX5_CORE=(y/m/n)** (module mlx5_core.ko)
31
32|    The driver can be enabled by choosing CONFIG_MLX5_CORE=y/m in kernel config.
33|    This will provide mlx5 core driver for mlx5 ulps to interface with (mlx5e, mlx5_ib).
34
35
36**CONFIG_MLX5_CORE_EN=(y/n)**
37
38|    Choosing this option will allow basic ethernet netdevice support with all of the standard rx/tx offloads.
39|    mlx5e is the mlx5 ulp driver which provides netdevice kernel interface, when chosen, mlx5e will be
40|    built-in into mlx5_core.ko.
41
42
43**CONFIG_MLX5_EN_ARFS=(y/n)**
44
45|     Enables Hardware-accelerated receive flow steering (arfs) support, and ntuple filtering.
46|     https://community.mellanox.com/s/article/howto-configure-arfs-on-connectx-4
47
48
49**CONFIG_MLX5_EN_RXNFC=(y/n)**
50
51|    Enables ethtool receive network flow classification, which allows user defined
52|    flow rules to direct traffic into arbitrary rx queue via ethtool set/get_rxnfc API.
53
54
55**CONFIG_MLX5_CORE_EN_DCB=(y/n)**:
56
57|    Enables `Data Center Bridging (DCB) Support <https://community.mellanox.com/s/article/howto-auto-config-pfc-and-ets-on-connectx-4-via-lldp-dcbx>`_.
58
59
60**CONFIG_MLX5_MPFS=(y/n)**
61
62|    Ethernet Multi-Physical Function Switch (MPFS) support in ConnectX NIC.
63|    MPFs is required for when `Multi-Host <http://www.mellanox.com/page/multihost>`_ configuration is enabled to allow passing
64|    user configured unicast MAC addresses to the requesting PF.
65
66
67**CONFIG_MLX5_ESWITCH=(y/n)**
68
69|    Ethernet SRIOV E-Switch support in ConnectX NIC. E-Switch provides internal SRIOV packet steering
70|    and switching for the enabled VFs and PF in two available modes:
71|           1) `Legacy SRIOV mode (L2 mac vlan steering based) <https://community.mellanox.com/s/article/howto-configure-sr-iov-for-connectx-4-connectx-5-with-kvm--ethernet-x>`_.
72|           2) `Switchdev mode (eswitch offloads) <https://www.mellanox.com/related-docs/prod_software/ASAP2_Hardware_Offloading_for_vSwitches_User_Manual_v4.4.pdf>`_.
73
74
75**CONFIG_MLX5_CORE_IPOIB=(y/n)**
76
77|    IPoIB offloads & acceleration support.
78|    Requires CONFIG_MLX5_CORE_EN to provide an accelerated interface for the rdma
79|    IPoIB ulp netdevice.
80
81
82**CONFIG_MLX5_FPGA=(y/n)**
83
84|    Build support for the Innova family of network cards by Mellanox Technologies.
85|    Innova network cards are comprised of a ConnectX chip and an FPGA chip on one board.
86|    If you select this option, the mlx5_core driver will include the Innova FPGA core and allow
87|    building sandbox-specific client drivers.
88
89
90**CONFIG_MLX5_EN_IPSEC=(y/n)**
91
92|    Enables `IPSec XFRM cryptography-offload accelaration <http://www.mellanox.com/related-docs/prod_software/Mellanox_Innova_IPsec_Ethernet_Adapter_Card_User_Manual.pdf>`_.
93
94**CONFIG_MLX5_EN_TLS=(y/n)**
95
96|   TLS cryptography-offload accelaration.
97
98
99**CONFIG_MLX5_INFINIBAND=(y/n/m)** (module mlx5_ib.ko)
100
101|   Provides low-level InfiniBand/RDMA and `RoCE <https://community.mellanox.com/s/article/recommended-network-configuration-examples-for-roce-deployment>`_ support.
102
103**CONFIG_MLX5_SF=(y/n)**
104
105|   Build support for subfunction.
106|   Subfunctons are more light weight than PCI SRIOV VFs. Choosing this option
107|   will enable support for creating subfunction devices.
108
109**External options** ( Choose if the corresponding mlx5 feature is required )
110
111- CONFIG_PTP_1588_CLOCK: When chosen, mlx5 ptp support will be enabled
112- CONFIG_VXLAN: When chosen, mlx5 vxlan support will be enabled.
113- CONFIG_MLXFW: When chosen, mlx5 firmware flashing support will be enabled (via devlink and ethtool).
114
115Devlink info
116============
117
118The devlink info reports the running and stored firmware versions on device.
119It also prints the device PSID which represents the HCA board type ID.
120
121User command example::
122
123   $ devlink dev info pci/0000:00:06.0
124      pci/0000:00:06.0:
125      driver mlx5_core
126      versions:
127         fixed:
128            fw.psid MT_0000000009
129         running:
130            fw.version 16.26.0100
131         stored:
132            fw.version 16.26.0100
133
134Devlink parameters
135==================
136
137flow_steering_mode: Device flow steering mode
138---------------------------------------------
139The flow steering mode parameter controls the flow steering mode of the driver.
140Two modes are supported:
1411. 'dmfs' - Device managed flow steering.
1422. 'smfs  - Software/Driver managed flow steering.
143
144In DMFS mode, the HW steering entities are created and managed through the
145Firmware.
146In SMFS mode, the HW steering entities are created and managed though by
147the driver directly into Hardware without firmware intervention.
148
149SMFS mode is faster and provides better rule inserstion rate compared to default DMFS mode.
150
151User command examples:
152
153- Set SMFS flow steering mode::
154
155    $ devlink dev param set pci/0000:06:00.0 name flow_steering_mode value "smfs" cmode runtime
156
157- Read device flow steering mode::
158
159    $ devlink dev param show pci/0000:06:00.0 name flow_steering_mode
160      pci/0000:06:00.0:
161      name flow_steering_mode type driver-specific
162      values:
163         cmode runtime value smfs
164
165enable_roce: RoCE enablement state
166----------------------------------
167RoCE enablement state controls driver support for RoCE traffic.
168When RoCE is disabled, there is no gid table, only raw ethernet QPs are supported and traffic on the well known UDP RoCE port is handled as raw ethernet traffic.
169
170To change RoCE enablement state a user must change the driverinit cmode value and run devlink reload.
171
172User command examples:
173
174- Disable RoCE::
175
176    $ devlink dev param set pci/0000:06:00.0 name enable_roce value false cmode driverinit
177    $ devlink dev reload pci/0000:06:00.0
178
179- Read RoCE enablement state::
180
181    $ devlink dev param show pci/0000:06:00.0 name enable_roce
182      pci/0000:06:00.0:
183      name enable_roce type generic
184      values:
185         cmode driverinit value true
186
187esw_port_metadata: Eswitch port metadata state
188----------------------------------------------
189When applicable, disabling Eswitch metadata can increase packet rate
190up to 20% depending on the use case and packet sizes.
191
192Eswitch port metadata state controls whether to internally tag packets with
193metadata. Metadata tagging must be enabled for multi-port RoCE, failover
194between representors and stacked devices.
195By default metadata is enabled on the supported devices in E-switch.
196Metadata is applicable only for E-switch in switchdev mode and
197users may disable it when NONE of the below use cases will be in use:
1981. HCA is in Dual/multi-port RoCE mode.
1992. VF/SF representor bonding (Usually used for Live migration)
2003. Stacked devices
201
202When metadata is disabled, the above use cases will fail to initialize if
203users try to enable them.
204
205- Show eswitch port metadata::
206
207    $ devlink dev param show pci/0000:06:00.0 name esw_port_metadata
208      pci/0000:06:00.0:
209        name esw_port_metadata type driver-specific
210          values:
211            cmode runtime value true
212
213- Disable eswitch port metadata::
214
215    $ devlink dev param set pci/0000:06:00.0 name esw_port_metadata value false cmode runtime
216
217- Change eswitch mode to switchdev mode where after choosing the metadata value::
218
219    $ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
220
221Bridge offload
222==============
223The mlx5 driver implements support for offloading bridge rules when in switchdev
224mode. Linux bridge FDBs are automatically offloaded when mlx5 switchdev
225representor is attached to bridge.
226
227- Change device to switchdev mode::
228
229    $ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
230
231- Attach mlx5 switchdev representor 'enp8s0f0' to bridge netdev 'bridge1'::
232
233    $ ip link set enp8s0f0 master bridge1
234
235VLANs
236-----
237Following bridge VLAN functions are supported by mlx5:
238
239- VLAN filtering (including multiple VLANs per port)::
240
241    $ ip link set bridge1 type bridge vlan_filtering 1
242    $ bridge vlan add dev enp8s0f0 vid 2-3
243
244- VLAN push on bridge ingress::
245
246    $ bridge vlan add dev enp8s0f0 vid 3 pvid
247
248- VLAN pop on bridge egress::
249
250    $ bridge vlan add dev enp8s0f0 vid 3 untagged
251
252mlx5 subfunction
253================
254mlx5 supports subfunction management using devlink port (see :ref:`Documentation/networking/devlink/devlink-port.rst <devlink_port>`) interface.
255
256A Subfunction has its own function capabilities and its own resources. This
257means a subfunction has its own dedicated queues (txq, rxq, cq, eq). These
258queues are neither shared nor stolen from the parent PCI function.
259
260When a subfunction is RDMA capable, it has its own QP1, GID table and rdma
261resources neither shared nor stolen from the parent PCI function.
262
263A subfunction has a dedicated window in PCI BAR space that is not shared
264with ther other subfunctions or the parent PCI function. This ensures that all
265devices (netdev, rdma, vdpa etc.) of the subfunction accesses only assigned
266PCI BAR space.
267
268A Subfunction supports eswitch representation through which it supports tc
269offloads. The user configures eswitch to send/receive packets from/to
270the subfunction port.
271
272Subfunctions share PCI level resources such as PCI MSI-X IRQs with
273other subfunctions and/or with its parent PCI function.
274
275Example mlx5 software, system and device view::
276
277       _______
278      | admin |
279      | user  |----------
280      |_______|         |
281          |             |
282      ____|____       __|______            _________________
283     |         |     |         |          |                 |
284     | devlink |     | tc tool |          |    user         |
285     | tool    |     |_________|          | applications    |
286     |_________|         |                |_________________|
287           |             |                   |          |
288           |             |                   |          |         Userspace
289 +---------|-------------|-------------------|----------|--------------------+
290           |             |           +----------+   +----------+   Kernel
291           |             |           |  netdev  |   | rdma dev |
292           |             |           +----------+   +----------+
293   (devlink port add/del |              ^               ^
294    port function set)   |              |               |
295           |             |              +---------------|
296      _____|___          |              |        _______|_______
297     |         |         |              |       | mlx5 class    |
298     | devlink |   +------------+       |       |   drivers     |
299     | kernel  |   | rep netdev |       |       |(mlx5_core,ib) |
300     |_________|   +------------+       |       |_______________|
301           |             |              |               ^
302   (devlink ops)         |              |          (probe/remove)
303  _________|________     |              |           ____|________
304 | subfunction      |    |     +---------------+   | subfunction |
305 | management driver|-----     | subfunction   |---|  driver     |
306 | (mlx5_core)      |          | auxiliary dev |   | (mlx5_core) |
307 |__________________|          +---------------+   |_____________|
308           |                                            ^
309  (sf add/del, vhca events)                             |
310           |                                      (device add/del)
311      _____|____                                    ____|________
312     |          |                                  | subfunction |
313     |  PCI NIC |---- activate/deactive events---->| host driver |
314     |__________|                                  | (mlx5_core) |
315                                                   |_____________|
316
317Subfunction is created using devlink port interface.
318
319- Change device to switchdev mode::
320
321    $ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
322
323- Add a devlink port of subfunction flaovur::
324
325    $ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
326    pci/0000:06:00.0/32768: type eth netdev eth6 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
327      function:
328        hw_addr 00:00:00:00:00:00 state inactive opstate detached
329
330- Show a devlink port of the subfunction::
331
332    $ devlink port show pci/0000:06:00.0/32768
333    pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcisf pfnum 0 sfnum 88
334      function:
335        hw_addr 00:00:00:00:00:00 state inactive opstate detached
336
337- Delete a devlink port of subfunction after use::
338
339    $ devlink port del pci/0000:06:00.0/32768
340
341mlx5 function attributes
342========================
343The mlx5 driver provides a mechanism to setup PCI VF/SF function attributes in
344a unified way for SmartNIC and non-SmartNIC.
345
346This is supported only when the eswitch mode is set to switchdev. Port function
347configuration of the PCI VF/SF is supported through devlink eswitch port.
348
349Port function attributes should be set before PCI VF/SF is enumerated by the
350driver.
351
352MAC address setup
353-----------------
354mlx5 driver provides mechanism to setup the MAC address of the PCI VF/SF.
355
356The configured MAC address of the PCI VF/SF will be used by netdevice and rdma
357device created for the PCI VF/SF.
358
359- Get the MAC address of the VF identified by its unique devlink port index::
360
361    $ devlink port show pci/0000:06:00.0/2
362    pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1
363      function:
364        hw_addr 00:00:00:00:00:00
365
366- Set the MAC address of the VF identified by its unique devlink port index::
367
368    $ devlink port function set pci/0000:06:00.0/2 hw_addr 00:11:22:33:44:55
369
370    $ devlink port show pci/0000:06:00.0/2
371    pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1
372      function:
373        hw_addr 00:11:22:33:44:55
374
375- Get the MAC address of the SF identified by its unique devlink port index::
376
377    $ devlink port show pci/0000:06:00.0/32768
378    pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcisf pfnum 0 sfnum 88
379      function:
380        hw_addr 00:00:00:00:00:00
381
382- Set the MAC address of the VF identified by its unique devlink port index::
383
384    $ devlink port function set pci/0000:06:00.0/32768 hw_addr 00:00:00:00:88:88
385
386    $ devlink port show pci/0000:06:00.0/32768
387    pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcivf pfnum 0 sfnum 88
388      function:
389        hw_addr 00:00:00:00:88:88
390
391SF state setup
392--------------
393To use the SF, the user must active the SF using the SF function state
394attribute.
395
396- Get the state of the SF identified by its unique devlink port index::
397
398   $ devlink port show ens2f0npf0sf88
399   pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
400     function:
401       hw_addr 00:00:00:00:88:88 state inactive opstate detached
402
403- Activate the function and verify its state is active::
404
405   $ devlink port function set ens2f0npf0sf88 state active
406
407   $ devlink port show ens2f0npf0sf88
408   pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
409     function:
410       hw_addr 00:00:00:00:88:88 state active opstate detached
411
412Upon function activation, the PF driver instance gets the event from the device
413that a particular SF was activated. It's the cue to put the device on bus, probe
414it and instantiate the devlink instance and class specific auxiliary devices
415for it.
416
417- Show the auxiliary device and port of the subfunction::
418
419    $ devlink dev show
420    devlink dev show auxiliary/mlx5_core.sf.4
421
422    $ devlink port show auxiliary/mlx5_core.sf.4/1
423    auxiliary/mlx5_core.sf.4/1: type eth netdev p0sf88 flavour virtual port 0 splittable false
424
425    $ rdma link show mlx5_0/1
426    link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev p0sf88
427
428    $ rdma dev show
429    8: rocep6s0f1: node_type ca fw 16.29.0550 node_guid 248a:0703:00b3:d113 sys_image_guid 248a:0703:00b3:d112
430    13: mlx5_0: node_type ca fw 16.29.0550 node_guid 0000:00ff:fe00:8888 sys_image_guid 248a:0703:00b3:d112
431
432- Subfunction auxiliary device and class device hierarchy::
433
434                 mlx5_core.sf.4
435          (subfunction auxiliary device)
436                       /\
437                      /  \
438                     /    \
439                    /      \
440                   /        \
441      mlx5_core.eth.4     mlx5_core.rdma.4
442     (sf eth aux dev)     (sf rdma aux dev)
443         |                      |
444         |                      |
445      p0sf88                  mlx5_0
446     (sf netdev)          (sf rdma device)
447
448Additionally, the SF port also gets the event when the driver attaches to the
449auxiliary device of the subfunction. This results in changing the operational
450state of the function. This provides visiblity to the user to decide when is it
451safe to delete the SF port for graceful termination of the subfunction.
452
453- Show the SF port operational state::
454
455    $ devlink port show ens2f0npf0sf88
456    pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
457      function:
458        hw_addr 00:00:00:00:88:88 state active opstate attached
459
460Devlink health reporters
461========================
462
463tx reporter
464-----------
465The tx reporter is responsible for reporting and recovering of the following two error scenarios:
466
467- TX timeout
468    Report on kernel tx timeout detection.
469    Recover by searching lost interrupts.
470- TX error completion
471    Report on error tx completion.
472    Recover by flushing the TX queue and reset it.
473
474TX reporter also support on demand diagnose callback, on which it provides
475real time information of its send queues status.
476
477User commands examples:
478
479- Diagnose send queues status::
480
481    $ devlink health diagnose pci/0000:82:00.0 reporter tx
482
483NOTE: This command has valid output only when interface is up, otherwise the command has empty output.
484
485- Show number of tx errors indicated, number of recover flows ended successfully,
486  is autorecover enabled and graceful period from last recover::
487
488    $ devlink health show pci/0000:82:00.0 reporter tx
489
490rx reporter
491-----------
492The rx reporter is responsible for reporting and recovering of the following two error scenarios:
493
494- RX queues initialization (population) timeout
495    RX queues descriptors population on ring initialization is done in
496    napi context via triggering an irq, in case of a failure to get
497    the minimum amount of descriptors, a timeout would occur and it
498    could be recoverable by polling the EQ (Event Queue).
499- RX completions with errors (reported by HW on interrupt context)
500    Report on rx completion error.
501    Recover (if needed) by flushing the related queue and reset it.
502
503RX reporter also supports on demand diagnose callback, on which it
504provides real time information of its receive queues status.
505
506- Diagnose rx queues status, and corresponding completion queue::
507
508    $ devlink health diagnose pci/0000:82:00.0 reporter rx
509
510NOTE: This command has valid output only when interface is up, otherwise the command has empty output.
511
512- Show number of rx errors indicated, number of recover flows ended successfully,
513  is autorecover enabled and graceful period from last recover::
514
515    $ devlink health show pci/0000:82:00.0 reporter rx
516
517fw reporter
518-----------
519The fw reporter implements diagnose and dump callbacks.
520It follows symptoms of fw error such as fw syndrome by triggering
521fw core dump and storing it into the dump buffer.
522The fw reporter diagnose command can be triggered any time by the user to check
523current fw status.
524
525User commands examples:
526
527- Check fw heath status::
528
529    $ devlink health diagnose pci/0000:82:00.0 reporter fw
530
531- Read FW core dump if already stored or trigger new one::
532
533    $ devlink health dump show pci/0000:82:00.0 reporter fw
534
535NOTE: This command can run only on the PF which has fw tracer ownership,
536running it on other PF or any VF will return "Operation not permitted".
537
538fw fatal reporter
539-----------------
540The fw fatal reporter implements dump and recover callbacks.
541It follows fatal errors indications by CR-space dump and recover flow.
542The CR-space dump uses vsc interface which is valid even if the FW command
543interface is not functional, which is the case in most FW fatal errors.
544The recover function runs recover flow which reloads the driver and triggers fw
545reset if needed.
546
547User commands examples:
548
549- Run fw recover flow manually::
550
551    $ devlink health recover pci/0000:82:00.0 reporter fw_fatal
552
553- Read FW CR-space dump if already strored or trigger new one::
554
555    $ devlink health dump show pci/0000:82:00.1 reporter fw_fatal
556
557NOTE: This command can run only on PF.
558
559mlx5 tracepoints
560================
561
562mlx5 driver provides internal trace points for tracking and debugging using
563kernel tracepoints interfaces (refer to Documentation/trace/ftrace.rst).
564
565For the list of support mlx5 events check /sys/kernel/debug/tracing/events/mlx5/
566
567tc and eswitch offloads tracepoints:
568
569- mlx5e_configure_flower: trace flower filter actions and cookies offloaded to mlx5::
570
571    $ echo mlx5:mlx5e_configure_flower >> /sys/kernel/debug/tracing/set_event
572    $ cat /sys/kernel/debug/tracing/trace
573    ...
574    tc-6535  [019] ...1  2672.404466: mlx5e_configure_flower: cookie=0000000067874a55 actions= REDIRECT
575
576- mlx5e_delete_flower: trace flower filter actions and cookies deleted from mlx5::
577
578    $ echo mlx5:mlx5e_delete_flower >> /sys/kernel/debug/tracing/set_event
579    $ cat /sys/kernel/debug/tracing/trace
580    ...
581    tc-6569  [010] .N.1  2686.379075: mlx5e_delete_flower: cookie=0000000067874a55 actions= NULL
582
583- mlx5e_stats_flower: trace flower stats request::
584
585    $ echo mlx5:mlx5e_stats_flower >> /sys/kernel/debug/tracing/set_event
586    $ cat /sys/kernel/debug/tracing/trace
587    ...
588    tc-6546  [010] ...1  2679.704889: mlx5e_stats_flower: cookie=0000000060eb3d6a bytes=0 packets=0 lastused=4295560217
589
590- mlx5e_tc_update_neigh_used_value: trace tunnel rule neigh update value offloaded to mlx5::
591
592    $ echo mlx5:mlx5e_tc_update_neigh_used_value >> /sys/kernel/debug/tracing/set_event
593    $ cat /sys/kernel/debug/tracing/trace
594    ...
595    kworker/u48:4-8806  [009] ...1 55117.882428: mlx5e_tc_update_neigh_used_value: netdev: ens1f0 IPv4: 1.1.1.10 IPv6: ::ffff:1.1.1.10 neigh_used=1
596
597- mlx5e_rep_neigh_update: trace neigh update tasks scheduled due to neigh state change events::
598
599    $ echo mlx5:mlx5e_rep_neigh_update >> /sys/kernel/debug/tracing/set_event
600    $ cat /sys/kernel/debug/tracing/trace
601    ...
602    kworker/u48:7-2221  [009] ...1  1475.387435: mlx5e_rep_neigh_update: netdev: ens1f0 MAC: 24:8a:07:9a:17:9a IPv4: 1.1.1.10 IPv6: ::ffff:1.1.1.10 neigh_connected=1
603
604Bridge offloads tracepoints:
605
606- mlx5_esw_bridge_fdb_entry_init: trace bridge FDB entry offloaded to mlx5::
607
608    $ echo mlx5:mlx5_esw_bridge_fdb_entry_init >> set_event
609    $ cat /sys/kernel/debug/tracing/trace
610    ...
611    kworker/u20:9-2217    [003] ...1   318.582243: mlx5_esw_bridge_fdb_entry_init: net_device=enp8s0f0_0 addr=e4:fd:05:08:00:02 vid=0 flags=0 used=0
612
613- mlx5_esw_bridge_fdb_entry_cleanup: trace bridge FDB entry deleted from mlx5::
614
615    $ echo mlx5:mlx5_esw_bridge_fdb_entry_cleanup >> set_event
616    $ cat /sys/kernel/debug/tracing/trace
617    ...
618    ip-2581    [005] ...1   318.629871: mlx5_esw_bridge_fdb_entry_cleanup: net_device=enp8s0f0_1 addr=e4:fd:05:08:00:03 vid=0 flags=0 used=16
619
620- mlx5_esw_bridge_fdb_entry_refresh: trace bridge FDB entry offload refreshed in
621  mlx5::
622
623    $ echo mlx5:mlx5_esw_bridge_fdb_entry_refresh >> set_event
624    $ cat /sys/kernel/debug/tracing/trace
625    ...
626    kworker/u20:8-3849    [003] ...1       466716: mlx5_esw_bridge_fdb_entry_refresh: net_device=enp8s0f0_0 addr=e4:fd:05:08:00:02 vid=3 flags=0 used=0
627
628- mlx5_esw_bridge_vlan_create: trace bridge VLAN object add on mlx5
629  representor::
630
631    $ echo mlx5:mlx5_esw_bridge_vlan_create >> set_event
632    $ cat /sys/kernel/debug/tracing/trace
633    ...
634    ip-2560    [007] ...1   318.460258: mlx5_esw_bridge_vlan_create: vid=1 flags=6
635
636- mlx5_esw_bridge_vlan_cleanup: trace bridge VLAN object delete from mlx5
637  representor::
638
639    $ echo mlx5:mlx5_esw_bridge_vlan_cleanup >> set_event
640    $ cat /sys/kernel/debug/tracing/trace
641    ...
642    bridge-2582    [007] ...1   318.653496: mlx5_esw_bridge_vlan_cleanup: vid=2 flags=8
643
644- mlx5_esw_bridge_vport_init: trace mlx5 vport assigned with bridge upper
645  device::
646
647    $ echo mlx5:mlx5_esw_bridge_vport_init >> set_event
648    $ cat /sys/kernel/debug/tracing/trace
649    ...
650    ip-2560    [007] ...1   318.458915: mlx5_esw_bridge_vport_init: vport_num=1
651
652- mlx5_esw_bridge_vport_cleanup: trace mlx5 vport removed from bridge upper
653  device::
654
655    $ echo mlx5:mlx5_esw_bridge_vport_cleanup >> set_event
656    $ cat /sys/kernel/debug/tracing/trace
657    ...
658    ip-5387    [000] ...1       573713: mlx5_esw_bridge_vport_cleanup: vport_num=1
659
660Eswitch QoS tracepoints:
661
662- mlx5_esw_vport_qos_create: trace creation of transmit scheduler arbiter for vport::
663
664    $ echo mlx5:mlx5_esw_vport_qos_create >> /sys/kernel/debug/tracing/set_event
665    $ cat /sys/kernel/debug/tracing/trace
666    ...
667    <...>-23496   [018] .... 73136.838831: mlx5_esw_vport_qos_create: (0000:82:00.0) vport=2 tsar_ix=4 bw_share=0, max_rate=0 group=000000007b576bb3
668
669- mlx5_esw_vport_qos_config: trace configuration of transmit scheduler arbiter for vport::
670
671    $ echo mlx5:mlx5_esw_vport_qos_config >> /sys/kernel/debug/tracing/set_event
672    $ cat /sys/kernel/debug/tracing/trace
673    ...
674    <...>-26548   [023] .... 75754.223823: mlx5_esw_vport_qos_config: (0000:82:00.0) vport=1 tsar_ix=3 bw_share=34, max_rate=10000 group=000000007b576bb3
675
676- mlx5_esw_vport_qos_destroy: trace deletion of transmit scheduler arbiter for vport::
677
678    $ echo mlx5:mlx5_esw_vport_qos_destroy >> /sys/kernel/debug/tracing/set_event
679    $ cat /sys/kernel/debug/tracing/trace
680    ...
681    <...>-27418   [004] .... 76546.680901: mlx5_esw_vport_qos_destroy: (0000:82:00.0) vport=1 tsar_ix=3
682
683- mlx5_esw_group_qos_create: trace creation of transmit scheduler arbiter for rate group::
684
685    $ echo mlx5:mlx5_esw_group_qos_create >> /sys/kernel/debug/tracing/set_event
686    $ cat /sys/kernel/debug/tracing/trace
687    ...
688    <...>-26578   [008] .... 75776.022112: mlx5_esw_group_qos_create: (0000:82:00.0) group=000000008dac63ea tsar_ix=5
689
690- mlx5_esw_group_qos_config: trace configuration of transmit scheduler arbiter for rate group::
691
692    $ echo mlx5:mlx5_esw_group_qos_config >> /sys/kernel/debug/tracing/set_event
693    $ cat /sys/kernel/debug/tracing/trace
694    ...
695    <...>-27303   [020] .... 76461.455356: mlx5_esw_group_qos_config: (0000:82:00.0) group=000000008dac63ea tsar_ix=5 bw_share=100 max_rate=20000
696
697- mlx5_esw_group_qos_destroy: trace deletion of transmit scheduler arbiter for group::
698
699    $ echo mlx5:mlx5_esw_group_qos_destroy >> /sys/kernel/debug/tracing/set_event
700    $ cat /sys/kernel/debug/tracing/trace
701    ...
702    <...>-27418   [006] .... 76547.187258: mlx5_esw_group_qos_destroy: (0000:82:00.0) group=000000007b576bb3 tsar_ix=1
703