1.. SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2
3=================================================
4Mellanox ConnectX(R) mlx5 core VPI Network Driver
5=================================================
6
7Copyright (c) 2019, Mellanox Technologies LTD.
8
9Contents
10========
11
12- `Enabling the driver and kconfig options`_
13- `Devlink info`_
14- `Devlink parameters`_
15- `Bridge offload`_
16- `mlx5 subfunction`_
17- `mlx5 function attributes`_
18- `Devlink health reporters`_
19- `mlx5 tracepoints`_
20
21Enabling the driver and kconfig options
22=======================================
23
24| mlx5 core is modular and most of the major mlx5 core driver features can be selected (compiled in/out)
25| at build time via kernel Kconfig flags.
26| Basic features, ethernet net device rx/tx offloads and XDP, are available with the most basic flags
27| CONFIG_MLX5_CORE=y/m and CONFIG_MLX5_CORE_EN=y.
28| For the list of advanced features please see below.
29
30**CONFIG_MLX5_CORE=(y/m/n)** (module mlx5_core.ko)
31
32|    The driver can be enabled by choosing CONFIG_MLX5_CORE=y/m in kernel config.
33|    This will provide mlx5 core driver for mlx5 ulps to interface with (mlx5e, mlx5_ib).
34
35
36**CONFIG_MLX5_CORE_EN=(y/n)**
37
38|    Choosing this option will allow basic ethernet netdevice support with all of the standard rx/tx offloads.
39|    mlx5e is the mlx5 ulp driver which provides netdevice kernel interface, when chosen, mlx5e will be
40|    built-in into mlx5_core.ko.
41
42
43**CONFIG_MLX5_EN_ARFS=(y/n)**
44
45|     Enables Hardware-accelerated receive flow steering (arfs) support, and ntuple filtering.
46|     https://community.mellanox.com/s/article/howto-configure-arfs-on-connectx-4
47
48
49**CONFIG_MLX5_EN_RXNFC=(y/n)**
50
51|    Enables ethtool receive network flow classification, which allows user defined
52|    flow rules to direct traffic into arbitrary rx queue via ethtool set/get_rxnfc API.
53
54
55**CONFIG_MLX5_CORE_EN_DCB=(y/n)**:
56
57|    Enables `Data Center Bridging (DCB) Support <https://community.mellanox.com/s/article/howto-auto-config-pfc-and-ets-on-connectx-4-via-lldp-dcbx>`_.
58
59
60**CONFIG_MLX5_MPFS=(y/n)**
61
62|    Ethernet Multi-Physical Function Switch (MPFS) support in ConnectX NIC.
63|    MPFs is required for when `Multi-Host <http://www.mellanox.com/page/multihost>`_ configuration is enabled to allow passing
64|    user configured unicast MAC addresses to the requesting PF.
65
66
67**CONFIG_MLX5_ESWITCH=(y/n)**
68
69|    Ethernet SRIOV E-Switch support in ConnectX NIC. E-Switch provides internal SRIOV packet steering
70|    and switching for the enabled VFs and PF in two available modes:
71|           1) `Legacy SRIOV mode (L2 mac vlan steering based) <https://community.mellanox.com/s/article/howto-configure-sr-iov-for-connectx-4-connectx-5-with-kvm--ethernet-x>`_.
72|           2) `Switchdev mode (eswitch offloads) <https://www.mellanox.com/related-docs/prod_software/ASAP2_Hardware_Offloading_for_vSwitches_User_Manual_v4.4.pdf>`_.
73
74
75**CONFIG_MLX5_CORE_IPOIB=(y/n)**
76
77|    IPoIB offloads & acceleration support.
78|    Requires CONFIG_MLX5_CORE_EN to provide an accelerated interface for the rdma
79|    IPoIB ulp netdevice.
80
81
82**CONFIG_MLX5_FPGA=(y/n)**
83
84|    Build support for the Innova family of network cards by Mellanox Technologies.
85|    Innova network cards are comprised of a ConnectX chip and an FPGA chip on one board.
86|    If you select this option, the mlx5_core driver will include the Innova FPGA core and allow
87|    building sandbox-specific client drivers.
88
89
90**CONFIG_MLX5_EN_IPSEC=(y/n)**
91
92|    Enables `IPSec XFRM cryptography-offload accelaration <http://www.mellanox.com/related-docs/prod_software/Mellanox_Innova_IPsec_Ethernet_Adapter_Card_User_Manual.pdf>`_.
93
94**CONFIG_MLX5_EN_TLS=(y/n)**
95
96|   TLS cryptography-offload accelaration.
97
98
99**CONFIG_MLX5_INFINIBAND=(y/n/m)** (module mlx5_ib.ko)
100
101|   Provides low-level InfiniBand/RDMA and `RoCE <https://community.mellanox.com/s/article/recommended-network-configuration-examples-for-roce-deployment>`_ support.
102
103**CONFIG_MLX5_SF=(y/n)**
104
105|   Build support for subfunction.
106|   Subfunctons are more light weight than PCI SRIOV VFs. Choosing this option
107|   will enable support for creating subfunction devices.
108
109**External options** ( Choose if the corresponding mlx5 feature is required )
110
111- CONFIG_PTP_1588_CLOCK: When chosen, mlx5 ptp support will be enabled
112- CONFIG_VXLAN: When chosen, mlx5 vxlan support will be enabled.
113- CONFIG_MLXFW: When chosen, mlx5 firmware flashing support will be enabled (via devlink and ethtool).
114
115Devlink info
116============
117
118The devlink info reports the running and stored firmware versions on device.
119It also prints the device PSID which represents the HCA board type ID.
120
121User command example::
122
123   $ devlink dev info pci/0000:00:06.0
124      pci/0000:00:06.0:
125      driver mlx5_core
126      versions:
127         fixed:
128            fw.psid MT_0000000009
129         running:
130            fw.version 16.26.0100
131         stored:
132            fw.version 16.26.0100
133
134Devlink parameters
135==================
136
137flow_steering_mode: Device flow steering mode
138---------------------------------------------
139The flow steering mode parameter controls the flow steering mode of the driver.
140Two modes are supported:
1411. 'dmfs' - Device managed flow steering.
1422. 'smfs  - Software/Driver managed flow steering.
143
144In DMFS mode, the HW steering entities are created and managed through the
145Firmware.
146In SMFS mode, the HW steering entities are created and managed though by
147the driver directly into Hardware without firmware intervention.
148
149SMFS mode is faster and provides better rule inserstion rate compared to default DMFS mode.
150
151User command examples:
152
153- Set SMFS flow steering mode::
154
155    $ devlink dev param set pci/0000:06:00.0 name flow_steering_mode value "smfs" cmode runtime
156
157- Read device flow steering mode::
158
159    $ devlink dev param show pci/0000:06:00.0 name flow_steering_mode
160      pci/0000:06:00.0:
161      name flow_steering_mode type driver-specific
162      values:
163         cmode runtime value smfs
164
165enable_roce: RoCE enablement state
166----------------------------------
167RoCE enablement state controls driver support for RoCE traffic.
168When RoCE is disabled, there is no gid table, only raw ethernet QPs are supported and traffic on the well known UDP RoCE port is handled as raw ethernet traffic.
169
170To change RoCE enablement state a user must change the driverinit cmode value and run devlink reload.
171
172User command examples:
173
174- Disable RoCE::
175
176    $ devlink dev param set pci/0000:06:00.0 name enable_roce value false cmode driverinit
177    $ devlink dev reload pci/0000:06:00.0
178
179- Read RoCE enablement state::
180
181    $ devlink dev param show pci/0000:06:00.0 name enable_roce
182      pci/0000:06:00.0:
183      name enable_roce type generic
184      values:
185         cmode driverinit value true
186
187esw_port_metadata: Eswitch port metadata state
188----------------------------------------------
189When applicable, disabling Eswitch metadata can increase packet rate
190up to 20% depending on the use case and packet sizes.
191
192Eswitch port metadata state controls whether to internally tag packets with
193metadata. Metadata tagging must be enabled for multi-port RoCE, failover
194between representors and stacked devices.
195By default metadata is enabled on the supported devices in E-switch.
196Metadata is applicable only for E-switch in switchdev mode and
197users may disable it when NONE of the below use cases will be in use:
1981. HCA is in Dual/multi-port RoCE mode.
1992. VF/SF representor bonding (Usually used for Live migration)
2003. Stacked devices
201
202When metadata is disabled, the above use cases will fail to initialize if
203users try to enable them.
204
205- Show eswitch port metadata::
206
207    $ devlink dev param show pci/0000:06:00.0 name esw_port_metadata
208      pci/0000:06:00.0:
209        name esw_port_metadata type driver-specific
210          values:
211            cmode runtime value true
212
213- Disable eswitch port metadata::
214
215    $ devlink dev param set pci/0000:06:00.0 name esw_port_metadata value false cmode runtime
216
217- Change eswitch mode to switchdev mode where after choosing the metadata value::
218
219    $ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
220
221Bridge offload
222==============
223The mlx5 driver implements support for offloading bridge rules when in switchdev
224mode. Linux bridge FDBs are automatically offloaded when mlx5 switchdev
225representor is attached to bridge.
226
227- Change device to switchdev mode::
228
229    $ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
230
231- Attach mlx5 switchdev representor 'enp8s0f0' to bridge netdev 'bridge1'::
232
233    $ ip link set enp8s0f0 master bridge1
234
235VLANs
236-----
237Following bridge VLAN functions are supported by mlx5:
238
239- VLAN filtering (including multiple VLANs per port)::
240
241    $ ip link set bridge1 type bridge vlan_filtering 1
242    $ bridge vlan add dev enp8s0f0 vid 2-3
243
244- VLAN push on bridge ingress::
245
246    $ bridge vlan add dev enp8s0f0 vid 3 pvid
247
248- VLAN pop on bridge egress::
249
250    $ bridge vlan add dev enp8s0f0 vid 3 untagged
251
252mlx5 subfunction
253================
254mlx5 supports subfunction management using devlink port (see :ref:`Documentation/networking/devlink/devlink-port.rst <devlink_port>`) interface.
255
256A Subfunction has its own function capabilities and its own resources. This
257means a subfunction has its own dedicated queues (txq, rxq, cq, eq). These
258queues are neither shared nor stolen from the parent PCI function.
259
260When a subfunction is RDMA capable, it has its own QP1, GID table and rdma
261resources neither shared nor stolen from the parent PCI function.
262
263A subfunction has a dedicated window in PCI BAR space that is not shared
264with ther other subfunctions or the parent PCI function. This ensures that all
265devices (netdev, rdma, vdpa etc.) of the subfunction accesses only assigned
266PCI BAR space.
267
268A Subfunction supports eswitch representation through which it supports tc
269offloads. The user configures eswitch to send/receive packets from/to
270the subfunction port.
271
272Subfunctions share PCI level resources such as PCI MSI-X IRQs with
273other subfunctions and/or with its parent PCI function.
274
275Example mlx5 software, system and device view::
276
277       _______
278      | admin |
279      | user  |----------
280      |_______|         |
281          |             |
282      ____|____       __|______            _________________
283     |         |     |         |          |                 |
284     | devlink |     | tc tool |          |    user         |
285     | tool    |     |_________|          | applications    |
286     |_________|         |                |_________________|
287           |             |                   |          |
288           |             |                   |          |         Userspace
289 +---------|-------------|-------------------|----------|--------------------+
290           |             |           +----------+   +----------+   Kernel
291           |             |           |  netdev  |   | rdma dev |
292           |             |           +----------+   +----------+
293   (devlink port add/del |              ^               ^
294    port function set)   |              |               |
295           |             |              +---------------|
296      _____|___          |              |        _______|_______
297     |         |         |              |       | mlx5 class    |
298     | devlink |   +------------+       |       |   drivers     |
299     | kernel  |   | rep netdev |       |       |(mlx5_core,ib) |
300     |_________|   +------------+       |       |_______________|
301           |             |              |               ^
302   (devlink ops)         |              |          (probe/remove)
303  _________|________     |              |           ____|________
304 | subfunction      |    |     +---------------+   | subfunction |
305 | management driver|-----     | subfunction   |---|  driver     |
306 | (mlx5_core)      |          | auxiliary dev |   | (mlx5_core) |
307 |__________________|          +---------------+   |_____________|
308           |                                            ^
309  (sf add/del, vhca events)                             |
310           |                                      (device add/del)
311      _____|____                                    ____|________
312     |          |                                  | subfunction |
313     |  PCI NIC |---- activate/deactive events---->| host driver |
314     |__________|                                  | (mlx5_core) |
315                                                   |_____________|
316
317Subfunction is created using devlink port interface.
318
319- Change device to switchdev mode::
320
321    $ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
322
323- Add a devlink port of subfunction flaovur::
324
325    $ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
326    pci/0000:06:00.0/32768: type eth netdev eth6 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
327      function:
328        hw_addr 00:00:00:00:00:00 state inactive opstate detached
329
330- Show a devlink port of the subfunction::
331
332    $ devlink port show pci/0000:06:00.0/32768
333    pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcisf pfnum 0 sfnum 88
334      function:
335        hw_addr 00:00:00:00:00:00 state inactive opstate detached
336
337- Delete a devlink port of subfunction after use::
338
339    $ devlink port del pci/0000:06:00.0/32768
340
341mlx5 function attributes
342========================
343The mlx5 driver provides a mechanism to setup PCI VF/SF function attributes in
344a unified way for SmartNIC and non-SmartNIC.
345
346This is supported only when the eswitch mode is set to switchdev. Port function
347configuration of the PCI VF/SF is supported through devlink eswitch port.
348
349Port function attributes should be set before PCI VF/SF is enumerated by the
350driver.
351
352MAC address setup
353-----------------
354mlx5 driver provides mechanism to setup the MAC address of the PCI VF/SF.
355
356The configured MAC address of the PCI VF/SF will be used by netdevice and rdma
357device created for the PCI VF/SF.
358
359- Get the MAC address of the VF identified by its unique devlink port index::
360
361    $ devlink port show pci/0000:06:00.0/2
362    pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1
363      function:
364        hw_addr 00:00:00:00:00:00
365
366- Set the MAC address of the VF identified by its unique devlink port index::
367
368    $ devlink port function set pci/0000:06:00.0/2 hw_addr 00:11:22:33:44:55
369
370    $ devlink port show pci/0000:06:00.0/2
371    pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1
372      function:
373        hw_addr 00:11:22:33:44:55
374
375- Get the MAC address of the SF identified by its unique devlink port index::
376
377    $ devlink port show pci/0000:06:00.0/32768
378    pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcisf pfnum 0 sfnum 88
379      function:
380        hw_addr 00:00:00:00:00:00
381
382- Set the MAC address of the VF identified by its unique devlink port index::
383
384    $ devlink port function set pci/0000:06:00.0/32768 hw_addr 00:00:00:00:88:88
385
386    $ devlink port show pci/0000:06:00.0/32768
387    pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcivf pfnum 0 sfnum 88
388      function:
389        hw_addr 00:00:00:00:88:88
390
391SF state setup
392--------------
393To use the SF, the user must active the SF using the SF function state
394attribute.
395
396- Get the state of the SF identified by its unique devlink port index::
397
398   $ devlink port show ens2f0npf0sf88
399   pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
400     function:
401       hw_addr 00:00:00:00:88:88 state inactive opstate detached
402
403- Activate the function and verify its state is active::
404
405   $ devlink port function set ens2f0npf0sf88 state active
406
407   $ devlink port show ens2f0npf0sf88
408   pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
409     function:
410       hw_addr 00:00:00:00:88:88 state active opstate detached
411
412Upon function activation, the PF driver instance gets the event from the device
413that a particular SF was activated. It's the cue to put the device on bus, probe
414it and instantiate the devlink instance and class specific auxiliary devices
415for it.
416
417- Show the auxiliary device and port of the subfunction::
418
419    $ devlink dev show
420    devlink dev show auxiliary/mlx5_core.sf.4
421
422    $ devlink port show auxiliary/mlx5_core.sf.4/1
423    auxiliary/mlx5_core.sf.4/1: type eth netdev p0sf88 flavour virtual port 0 splittable false
424
425    $ rdma link show mlx5_0/1
426    link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev p0sf88
427
428    $ rdma dev show
429    8: rocep6s0f1: node_type ca fw 16.29.0550 node_guid 248a:0703:00b3:d113 sys_image_guid 248a:0703:00b3:d112
430    13: mlx5_0: node_type ca fw 16.29.0550 node_guid 0000:00ff:fe00:8888 sys_image_guid 248a:0703:00b3:d112
431
432- Subfunction auxiliary device and class device hierarchy::
433
434                 mlx5_core.sf.4
435          (subfunction auxiliary device)
436                       /\
437                      /  \
438                     /    \
439                    /      \
440                   /        \
441      mlx5_core.eth.4     mlx5_core.rdma.4
442     (sf eth aux dev)     (sf rdma aux dev)
443         |                      |
444         |                      |
445      p0sf88                  mlx5_0
446     (sf netdev)          (sf rdma device)
447
448Additionally, the SF port also gets the event when the driver attaches to the
449auxiliary device of the subfunction. This results in changing the operational
450state of the function. This provides visiblity to the user to decide when is it
451safe to delete the SF port for graceful termination of the subfunction.
452
453- Show the SF port operational state::
454
455    $ devlink port show ens2f0npf0sf88
456    pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
457      function:
458        hw_addr 00:00:00:00:88:88 state active opstate attached
459
460Devlink health reporters
461========================
462
463tx reporter
464-----------
465The tx reporter is responsible for reporting and recovering of the following two error scenarios:
466
467- TX timeout
468    Report on kernel tx timeout detection.
469    Recover by searching lost interrupts.
470- TX error completion
471    Report on error tx completion.
472    Recover by flushing the TX queue and reset it.
473
474TX reporter also support on demand diagnose callback, on which it provides
475real time information of its send queues status.
476
477User commands examples:
478
479- Diagnose send queues status::
480
481    $ devlink health diagnose pci/0000:82:00.0 reporter tx
482
483NOTE: This command has valid output only when interface is up, otherwise the command has empty output.
484
485- Show number of tx errors indicated, number of recover flows ended successfully,
486  is autorecover enabled and graceful period from last recover::
487
488    $ devlink health show pci/0000:82:00.0 reporter tx
489
490rx reporter
491-----------
492The rx reporter is responsible for reporting and recovering of the following two error scenarios:
493
494- RX queues initialization (population) timeout
495    RX queues descriptors population on ring initialization is done in
496    napi context via triggering an irq, in case of a failure to get
497    the minimum amount of descriptors, a timeout would occur and it
498    could be recoverable by polling the EQ (Event Queue).
499- RX completions with errors (reported by HW on interrupt context)
500    Report on rx completion error.
501    Recover (if needed) by flushing the related queue and reset it.
502
503RX reporter also supports on demand diagnose callback, on which it
504provides real time information of its receive queues status.
505
506- Diagnose rx queues status, and corresponding completion queue::
507
508    $ devlink health diagnose pci/0000:82:00.0 reporter rx
509
510NOTE: This command has valid output only when interface is up, otherwise the command has empty output.
511
512- Show number of rx errors indicated, number of recover flows ended successfully,
513  is autorecover enabled and graceful period from last recover::
514
515    $ devlink health show pci/0000:82:00.0 reporter rx
516
517fw reporter
518-----------
519The fw reporter implements diagnose and dump callbacks.
520It follows symptoms of fw error such as fw syndrome by triggering
521fw core dump and storing it into the dump buffer.
522The fw reporter diagnose command can be triggered any time by the user to check
523current fw status.
524
525User commands examples:
526
527- Check fw heath status::
528
529    $ devlink health diagnose pci/0000:82:00.0 reporter fw
530
531- Read FW core dump if already stored or trigger new one::
532
533    $ devlink health dump show pci/0000:82:00.0 reporter fw
534
535NOTE: This command can run only on the PF which has fw tracer ownership,
536running it on other PF or any VF will return "Operation not permitted".
537
538fw fatal reporter
539-----------------
540The fw fatal reporter implements dump and recover callbacks.
541It follows fatal errors indications by CR-space dump and recover flow.
542The CR-space dump uses vsc interface which is valid even if the FW command
543interface is not functional, which is the case in most FW fatal errors.
544The recover function runs recover flow which reloads the driver and triggers fw
545reset if needed.
546On firmware error, the health buffer is dumped into the dmesg. The log
547level is derived from the error's severity (given in health buffer).
548
549User commands examples:
550
551- Run fw recover flow manually::
552
553    $ devlink health recover pci/0000:82:00.0 reporter fw_fatal
554
555- Read FW CR-space dump if already strored or trigger new one::
556
557    $ devlink health dump show pci/0000:82:00.1 reporter fw_fatal
558
559NOTE: This command can run only on PF.
560
561mlx5 tracepoints
562================
563
564mlx5 driver provides internal trace points for tracking and debugging using
565kernel tracepoints interfaces (refer to Documentation/trace/ftrace.rst).
566
567For the list of support mlx5 events check /sys/kernel/debug/tracing/events/mlx5/
568
569tc and eswitch offloads tracepoints:
570
571- mlx5e_configure_flower: trace flower filter actions and cookies offloaded to mlx5::
572
573    $ echo mlx5:mlx5e_configure_flower >> /sys/kernel/debug/tracing/set_event
574    $ cat /sys/kernel/debug/tracing/trace
575    ...
576    tc-6535  [019] ...1  2672.404466: mlx5e_configure_flower: cookie=0000000067874a55 actions= REDIRECT
577
578- mlx5e_delete_flower: trace flower filter actions and cookies deleted from mlx5::
579
580    $ echo mlx5:mlx5e_delete_flower >> /sys/kernel/debug/tracing/set_event
581    $ cat /sys/kernel/debug/tracing/trace
582    ...
583    tc-6569  [010] .N.1  2686.379075: mlx5e_delete_flower: cookie=0000000067874a55 actions= NULL
584
585- mlx5e_stats_flower: trace flower stats request::
586
587    $ echo mlx5:mlx5e_stats_flower >> /sys/kernel/debug/tracing/set_event
588    $ cat /sys/kernel/debug/tracing/trace
589    ...
590    tc-6546  [010] ...1  2679.704889: mlx5e_stats_flower: cookie=0000000060eb3d6a bytes=0 packets=0 lastused=4295560217
591
592- mlx5e_tc_update_neigh_used_value: trace tunnel rule neigh update value offloaded to mlx5::
593
594    $ echo mlx5:mlx5e_tc_update_neigh_used_value >> /sys/kernel/debug/tracing/set_event
595    $ cat /sys/kernel/debug/tracing/trace
596    ...
597    kworker/u48:4-8806  [009] ...1 55117.882428: mlx5e_tc_update_neigh_used_value: netdev: ens1f0 IPv4: 1.1.1.10 IPv6: ::ffff:1.1.1.10 neigh_used=1
598
599- mlx5e_rep_neigh_update: trace neigh update tasks scheduled due to neigh state change events::
600
601    $ echo mlx5:mlx5e_rep_neigh_update >> /sys/kernel/debug/tracing/set_event
602    $ cat /sys/kernel/debug/tracing/trace
603    ...
604    kworker/u48:7-2221  [009] ...1  1475.387435: mlx5e_rep_neigh_update: netdev: ens1f0 MAC: 24:8a:07:9a:17:9a IPv4: 1.1.1.10 IPv6: ::ffff:1.1.1.10 neigh_connected=1
605
606Bridge offloads tracepoints:
607
608- mlx5_esw_bridge_fdb_entry_init: trace bridge FDB entry offloaded to mlx5::
609
610    $ echo mlx5:mlx5_esw_bridge_fdb_entry_init >> set_event
611    $ cat /sys/kernel/debug/tracing/trace
612    ...
613    kworker/u20:9-2217    [003] ...1   318.582243: mlx5_esw_bridge_fdb_entry_init: net_device=enp8s0f0_0 addr=e4:fd:05:08:00:02 vid=0 flags=0 used=0
614
615- mlx5_esw_bridge_fdb_entry_cleanup: trace bridge FDB entry deleted from mlx5::
616
617    $ echo mlx5:mlx5_esw_bridge_fdb_entry_cleanup >> set_event
618    $ cat /sys/kernel/debug/tracing/trace
619    ...
620    ip-2581    [005] ...1   318.629871: mlx5_esw_bridge_fdb_entry_cleanup: net_device=enp8s0f0_1 addr=e4:fd:05:08:00:03 vid=0 flags=0 used=16
621
622- mlx5_esw_bridge_fdb_entry_refresh: trace bridge FDB entry offload refreshed in
623  mlx5::
624
625    $ echo mlx5:mlx5_esw_bridge_fdb_entry_refresh >> set_event
626    $ cat /sys/kernel/debug/tracing/trace
627    ...
628    kworker/u20:8-3849    [003] ...1       466716: mlx5_esw_bridge_fdb_entry_refresh: net_device=enp8s0f0_0 addr=e4:fd:05:08:00:02 vid=3 flags=0 used=0
629
630- mlx5_esw_bridge_vlan_create: trace bridge VLAN object add on mlx5
631  representor::
632
633    $ echo mlx5:mlx5_esw_bridge_vlan_create >> set_event
634    $ cat /sys/kernel/debug/tracing/trace
635    ...
636    ip-2560    [007] ...1   318.460258: mlx5_esw_bridge_vlan_create: vid=1 flags=6
637
638- mlx5_esw_bridge_vlan_cleanup: trace bridge VLAN object delete from mlx5
639  representor::
640
641    $ echo mlx5:mlx5_esw_bridge_vlan_cleanup >> set_event
642    $ cat /sys/kernel/debug/tracing/trace
643    ...
644    bridge-2582    [007] ...1   318.653496: mlx5_esw_bridge_vlan_cleanup: vid=2 flags=8
645
646- mlx5_esw_bridge_vport_init: trace mlx5 vport assigned with bridge upper
647  device::
648
649    $ echo mlx5:mlx5_esw_bridge_vport_init >> set_event
650    $ cat /sys/kernel/debug/tracing/trace
651    ...
652    ip-2560    [007] ...1   318.458915: mlx5_esw_bridge_vport_init: vport_num=1
653
654- mlx5_esw_bridge_vport_cleanup: trace mlx5 vport removed from bridge upper
655  device::
656
657    $ echo mlx5:mlx5_esw_bridge_vport_cleanup >> set_event
658    $ cat /sys/kernel/debug/tracing/trace
659    ...
660    ip-5387    [000] ...1       573713: mlx5_esw_bridge_vport_cleanup: vport_num=1
661
662Eswitch QoS tracepoints:
663
664- mlx5_esw_vport_qos_create: trace creation of transmit scheduler arbiter for vport::
665
666    $ echo mlx5:mlx5_esw_vport_qos_create >> /sys/kernel/debug/tracing/set_event
667    $ cat /sys/kernel/debug/tracing/trace
668    ...
669    <...>-23496   [018] .... 73136.838831: mlx5_esw_vport_qos_create: (0000:82:00.0) vport=2 tsar_ix=4 bw_share=0, max_rate=0 group=000000007b576bb3
670
671- mlx5_esw_vport_qos_config: trace configuration of transmit scheduler arbiter for vport::
672
673    $ echo mlx5:mlx5_esw_vport_qos_config >> /sys/kernel/debug/tracing/set_event
674    $ cat /sys/kernel/debug/tracing/trace
675    ...
676    <...>-26548   [023] .... 75754.223823: mlx5_esw_vport_qos_config: (0000:82:00.0) vport=1 tsar_ix=3 bw_share=34, max_rate=10000 group=000000007b576bb3
677
678- mlx5_esw_vport_qos_destroy: trace deletion of transmit scheduler arbiter for vport::
679
680    $ echo mlx5:mlx5_esw_vport_qos_destroy >> /sys/kernel/debug/tracing/set_event
681    $ cat /sys/kernel/debug/tracing/trace
682    ...
683    <...>-27418   [004] .... 76546.680901: mlx5_esw_vport_qos_destroy: (0000:82:00.0) vport=1 tsar_ix=3
684
685- mlx5_esw_group_qos_create: trace creation of transmit scheduler arbiter for rate group::
686
687    $ echo mlx5:mlx5_esw_group_qos_create >> /sys/kernel/debug/tracing/set_event
688    $ cat /sys/kernel/debug/tracing/trace
689    ...
690    <...>-26578   [008] .... 75776.022112: mlx5_esw_group_qos_create: (0000:82:00.0) group=000000008dac63ea tsar_ix=5
691
692- mlx5_esw_group_qos_config: trace configuration of transmit scheduler arbiter for rate group::
693
694    $ echo mlx5:mlx5_esw_group_qos_config >> /sys/kernel/debug/tracing/set_event
695    $ cat /sys/kernel/debug/tracing/trace
696    ...
697    <...>-27303   [020] .... 76461.455356: mlx5_esw_group_qos_config: (0000:82:00.0) group=000000008dac63ea tsar_ix=5 bw_share=100 max_rate=20000
698
699- mlx5_esw_group_qos_destroy: trace deletion of transmit scheduler arbiter for group::
700
701    $ echo mlx5:mlx5_esw_group_qos_destroy >> /sys/kernel/debug/tracing/set_event
702    $ cat /sys/kernel/debug/tracing/trace
703    ...
704    <...>-27418   [006] .... 76547.187258: mlx5_esw_group_qos_destroy: (0000:82:00.0) group=000000007b576bb3 tsar_ix=1
705
706SF tracepoints:
707
708- mlx5_sf_add: trace addition of the SF port::
709
710    $ echo mlx5:mlx5_sf_add >> /sys/kernel/debug/tracing/set_event
711    $ cat /sys/kernel/debug/tracing/trace
712    ...
713    devlink-9363    [031] ..... 24610.188722: mlx5_sf_add: (0000:06:00.0) port_index=32768 controller=0 hw_id=0x8000 sfnum=88
714
715- mlx5_sf_free: trace freeing of the SF port::
716
717    $ echo mlx5:mlx5_sf_free >> /sys/kernel/debug/tracing/set_event
718    $ cat /sys/kernel/debug/tracing/trace
719    ...
720    devlink-9830    [038] ..... 26300.404749: mlx5_sf_free: (0000:06:00.0) port_index=32768 controller=0 hw_id=0x8000
721
722- mlx5_sf_hwc_alloc: trace allocating of the hardware SF context::
723
724    $ echo mlx5:mlx5_sf_hwc_alloc >> /sys/kernel/debug/tracing/set_event
725    $ cat /sys/kernel/debug/tracing/trace
726    ...
727    devlink-9775    [031] ..... 26296.385259: mlx5_sf_hwc_alloc: (0000:06:00.0) controller=0 hw_id=0x8000 sfnum=88
728
729- mlx5_sf_hwc_free: trace freeing of the hardware SF context::
730
731    $ echo mlx5:mlx5_sf_hwc_free >> /sys/kernel/debug/tracing/set_event
732    $ cat /sys/kernel/debug/tracing/trace
733    ...
734    kworker/u128:3-9093    [046] ..... 24625.365771: mlx5_sf_hwc_free: (0000:06:00.0) hw_id=0x8000
735
736- mlx5_sf_hwc_deferred_free : trace deferred freeing of the hardware SF context::
737
738    $ echo mlx5:mlx5_sf_hwc_deferred_free >> /sys/kernel/debug/tracing/set_event
739    $ cat /sys/kernel/debug/tracing/trace
740    ...
741    devlink-9519    [046] ..... 24624.400271: mlx5_sf_hwc_deferred_free: (0000:06:00.0) hw_id=0x8000
742
743- mlx5_sf_vhca_event: trace SF vhca event and state::
744
745    $ echo mlx5:mlx5_sf_vhca_event >> /sys/kernel/debug/tracing/set_event
746    $ cat /sys/kernel/debug/tracing/trace
747    ...
748    kworker/u128:3-9093    [046] ..... 24625.365525: mlx5_sf_vhca_event: (0000:06:00.0) hw_id=0x8000 sfnum=88 vhca_state=1
749
750- mlx5_sf_dev_add : trace SF device add event::
751
752    $ echo mlx5:mlx5_sf_dev_add>> /sys/kernel/debug/tracing/set_event
753    $ cat /sys/kernel/debug/tracing/trace
754    ...
755    kworker/u128:3-9093    [000] ..... 24616.524495: mlx5_sf_dev_add: (0000:06:00.0) sfdev=00000000fc5d96fd aux_id=4 hw_id=0x8000 sfnum=88
756
757- mlx5_sf_dev_del : trace SF device delete event::
758
759    $ echo mlx5:mlx5_sf_dev_del >> /sys/kernel/debug/tracing/set_event
760    $ cat /sys/kernel/debug/tracing/trace
761    ...
762    kworker/u128:3-9093    [044] ..... 24624.400749: mlx5_sf_dev_del: (0000:06:00.0) sfdev=00000000fc5d96fd aux_id=4 hw_id=0x8000 sfnum=88
763