1.. _tracing:
2
3Tracing
4#######
5
6Overview
7********
8
9The tracing feature provides hooks that permits you to collect data from
10your application and allows :ref:`tools` running on a host to visualize the inner-working of
11the kernel and various subsystems.
12
13Every system has application-specific events to trace out.  Historically,
14that has implied:
15
161. Determining the application-specific payload,
172. Choosing suitable serialization-format,
183. Writing the on-target serialization code,
194. Deciding on and writing the I/O transport mechanics,
205. Writing the PC-side deserializer/parser,
216. Writing custom ad-hoc tools for filtering and presentation.
22
23An application can use one of the existing formats or define a custom format by
24overriding the macros declared in :zephyr_file:`include/zephyr/tracing/tracing.h`.
25
26Different formats, transports and host tools are available and supported in
27Zephyr.
28
29In fact, I/O varies greatly from system to system.  Therefore, it is
30instructive to create a taxonomy for I/O types when we must ensure the
31interface between payload/format (Top Layer) and the transport mechanics
32(bottom Layer) is generic and efficient enough to model these. See the
33*I/O taxonomy* section below.
34
35Named Trace Events
36******************
37
38Although the user can extend any of the supported serialization formats
39to enable additional tracing functions (or provide their own backend), zephyr
40also provides one generic named tracing function for convenience purposes,
41as well as to demonstrate how tracing frameworks could be extended.
42
43Users can generate a custom trace event by calling
44:c:func:`sys_tracing_named_event`, which takes an event name as well as two
45arbitrary 4 byte arguments. Tracing backends may truncate the provided event
46name if it is too long for the serialization format they support.
47
48Serialization Formats
49**********************
50
51.. _ctf:
52
53Common Trace Format (CTF) Support
54=================================
55
56Common Trace Format, CTF, is an open format and language to describe trace
57formats. This enables tool reuse, of which line-textual (babeltrace) and
58graphical (TraceCompass) variants already exist.
59
60CTF should look familiar to C programmers but adds stronger typing.
61See `CTF - A Flexible, High-performance Binary Trace Format
62<http://diamon.org/ctf/>`_.
63
64
65CTF allows us to formally describe application specific payload and the
66serialization format, which enables common infrastructure for host tools
67and parsers and tools for filtering and presentation.
68
69
70A Generic Interface
71--------------------
72
73In CTF, an event is serialized to a packet containing one or more fields.
74As seen from *I/O taxonomy* section below, a bottom layer may:
75
76- perform actions at transaction-start (e.g. mutex-lock),
77- process each field in some way (e.g. sync-push emit, concat, enqueue to
78  thread-bound FIFO),
79- perform actions at transaction-stop (e.g. mutex-release, emit of concat
80  buffer).
81
82CTF Top-Layer Example
83----------------------
84
85The CTF_EVENT macro will serialize each argument to a field::
86
87  /* Example for illustration */
88  static inline void ctf_top_foo(uint32_t thread_id, ctf_bounded_string_t name)
89  {
90    CTF_EVENT(
91      CTF_LITERAL(uint8_t, 42),
92      thread_id,
93      name,
94      "hello, I was emitted from function: ",
95      __func__  /* __func__ is standard since C99 */
96    );
97  }
98
99How to serialize and emit fields as well as handling alignment, can be done
100internally and statically at compile-time in the bottom-layer.
101
102
103The CTF top layer is enabled using the configuration option
104:kconfig:option:`CONFIG_TRACING_CTF` and can be used with the different transport
105backends both in synchronous and asynchronous modes.
106
107.. _tools:
108
109Tracing Tools
110*************
111
112Zephyr includes support for several popular tracing tools, presented below in alphabetical order.
113
114Percepio Tracealyzer Support
115============================
116
117Zephyr includes support for `Percepio Tracealyzer`_ that offers trace visualization for
118simplified analysis, report generation and other analysis features. Tracealyzer allows for trace
119streaming over various interfaces and also snapshot tracing, where the events are kept in a RAM
120buffer.
121
122.. _Percepio Tracealyzer: https://percepio.com/tracealyzer
123
124.. figure:: percepio_tracealyzer.png
125    :align: center
126    :alt: Percepio Tracealyzer
127    :figclass: align-center
128    :width: 80%
129
130Zephyr kernel events are captured automatically when Tracealyzer tracing is enabled.
131Tracealyzer also provides extensive support for application logging, where you call the tracing
132library from your application code. This lets you visualize kernel events and application events
133together, for example as data plots or state diagrams on logged variables.
134Learn more in the Tracealyzer User Manual provided with the application.
135
136Percepio TraceRecorder and Stream Ports
137---------------------------------------
138The tracing library for Tracealyzer (TraceRecorder) is included in the Zephyr manifest and
139provided under the same license (Apache 2.0). This is enabled by adding the following
140configuration options in your prj.conf:
141
142.. code-block:: cfg
143
144    CONFIG_TRACING=y
145    CONFIG_PERCEPIO_TRACERECORDER=y
146
147Or using menuconfig:
148
149* Enable :menuselection:`Subsystems and OS Services --> Tracing Support`
150* Under :menuselection:`Subsystems and OS Services --> Tracing Support --> Tracing Format`, select
151  :guilabel:`Percepio Tracealyzer`
152
153Some additional settings are needed to configure TraceRecorder. The most important configuration
154is to select the right "stream port". This specifies how to output the trace data.
155As of July 2024, the following stream ports are available in the Zephyr configuration system:
156
157* **Ring Buffer**: The trace data is kept in a circular RAM buffer.
158* **RTT**: Trace streaming via Segger RTT on J-Link debug probes.
159* **ITM**: Trace streaming via the ITM function on Arm Cortex-M devices.
160* **Semihost**: For tracing on QEMU. Streams the trace data to a host file.
161
162Select the stream port in menuconfig under
163:menuselection:`Modules --> percepio --> TraceRecorder --> Stream Port`.
164
165Or simply add one of the following options in your prj.conf:
166
167.. code-block:: cfg
168
169    CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_RINGBUFFER=y
170    CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_RTT=y
171    CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_ITM=y
172    CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_ZEPHYR_SEMIHOST=y
173
174Make sure to only include ONE of these configuration options.
175
176The stream port modules have individual configuration options. In menuconfig these are found
177under :menuselection:`Modules --> percepio --> TraceRecorder --> (Stream Port) Config`.
178The most important options for each stream port are described below.
179
180Tracealyzer Snapshot Tracing (Ring Buffer)
181------------------------------------------
182
183The "Ring Buffer" stream port keeps the trace data in a RAM buffer on the device.
184By default, this is a circular buffer, meaning that it always contains the most recent data.
185This is used to dump "snapshots" of the trace data, e.g. by using the debugger. This usually only
186allows for short traces, unless you have megabytes of RAM to spare, so it is not suitable for
187profiling. However, it can be quite useful for debugging in combination with breakpoints.
188For example, if you set a breakpoint in an error handler, a snapshot trace can show the sequence
189of events leading up to the error. Snapshot tracing is also easy to begin with, since it doesn't
190depend on any particular debug probe or other development tool.
191
192To use the Ring Buffer option, make sure to have the following configuration options in your
193prj.cnf:
194
195.. code-block:: cfg
196
197    CONFIG_TRACING=y
198    CONFIG_PERCEPIO_TRACERECORDER=y
199    CONFIG_PERCEPIO_TRC_START_MODE_START=y
200    CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_RINGBUFFER=y
201    CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_RINGBUFFER_SIZE=<size in bytes>
202
203Or if using menuconfig:
204
205* Enable :menuselection:`Subsystems and OS Services --> Tracing Support`
206* Under :menuselection:`Subsystems and OS Services --> Tracing Support --> Tracing Format`, select
207  :guilabel:`Percepio Tracealyzer`
208* Under :menuselection:`Modules --> percepio --> TraceRecorder --> Recorder Start Mode`, select
209  :guilabel:`Start`
210* Under :menuselection:`Modules --> percepio --> TraceRecorder --> Stream Port`, select
211  :guilabel:`Ring Buffer`
212* Under :menuselection:`Modules --> percepio --> TraceRecorder --> Ring Buffer Config --> Buffer Size`,
213  set the buffer size in bytes.
214
215The default buffer size can be reduced if you are tight on RAM, or increased if you have RAM to
216spare and want longer traces. You may also optimize the Tracing Configuration settings to get
217longer traces by filtering out less important events.
218In menuconfig, see
219:menuselection:`Subsystems and OS Services --> Tracing Support --> Tracing Configuration`.
220
221To view the trace data, the easiest way is to start your debugger (west debug) and run the
222following GDB command::
223
224    dump binary value trace.bin *RecorderDataPtr
225
226The resulting file is typically found in the root of the build folder, unless a different path is
227specified. Open this file in Tracealyzer by selecting :menuselection:`File --> Open --> Open File`.
228
229Tracealyzer Streaming with SEGGER RTT
230-------------------------------------
231
232Tracealyzer has built-in support for SEGGER RTT to receive trace data using a J-Link probe.
233This allows for recording very long traces. To configure Zephyr for RTT streaming to Tracealyzer,
234add the following configuration options in your prj.cnf:
235
236.. code-block:: cfg
237
238    CONFIG_TRACING=y
239    CONFIG_PERCEPIO_TRACERECORDER=y
240    CONFIG_PERCEPIO_TRC_START_MODE_START_FROM_HOST=y
241    CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_RTT=y
242    CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_RTT_UP_BUFFER_SIZE=<size in bytes>
243
244Or if using menuconfig:
245
246* Enable :menuselection:`Subsystems and OS Services --> Tracing Support`
247* Under :menuselection:`Subsystems and OS Services --> Tracing Support --> Tracing Format`, select
248  :guilabel:`Percepio Tracealyzer`
249* Under :menuselection:`Modules --> percepio --> TraceRecorder --> Recorder Start Mode`, select
250  :guilabel:`Start From Host`
251* Under :menuselection:`Modules --> percepio --> TraceRecorder --> Stream Port`, select
252  :guilabel:`RTT`
253* Under :menuselection:`Modules --> percepio --> TraceRecorder --> RTT Config`, set the size of the
254  RTT "up" buffer in bytes.
255
256The setting :guilabel:`RTT buffer size up` sets the size of the RTT transmission buffer. This is important
257for throughput. By default this buffer is quite large, 5000 bytes, to give decent performance
258also on onboard J-Link debuggers (they are not as fast as the stand-alone probes).
259If you are tight on RAM, you may consider reducing this setting. If using a regular J-Link probe
260it is often sufficient with a much smaller buffer, e.g. 1 KB or less.
261
262Learn more about RTT streaming in the Tracealyzer User Manual.
263See Creating and Loading Traces -> Percepio TraceRecorder -> Using TraceRecorder v4.6 or later ->
264Stream ports (or search for RTT).
265
266Tracealyzer Streaming with Arm ITM
267----------------------------------
268
269This stream port is for Arm Cortex-M devices featuring the ITM unit. It is recommended to use a
270fast debug probe that allows for SWO speeds of 10 MHz or higher. To use this stream port,
271apply the following configuration options:
272
273.. code-block:: cfg
274
275    CONFIG_TRACING=y
276    CONFIG_PERCEPIO_TRACERECORDER=y
277    CONFIG_PERCEPIO_TRC_START_MODE_START=y
278    CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_ITM=y
279    CONFIG_PERCEPIO_TRC_CFG_ITM_PORT=1
280
281Or if using menuconfig:
282
283* Enable :menuselection:`Subsystems and OS Services --> Tracing Support`
284* Under :menuselection:`Subsystems and OS Services --> Tracing Support --> Tracing Format`, select
285  :guilabel:`Percepio Tracealyzer`
286* Under :menuselection:`Modules --> percepio --> TraceRecorder --> Recorder Start Mode`, select
287  :guilabel:`Start`
288* Under :menuselection:`Modules --> percepio --> TraceRecorder --> Stream Port`, select
289  :guilabel:`ITM`
290* Under :menuselection:`Modules --> percepio --> TraceRecorder --> ITM Config`, set the ITM port to
291  1.
292
293The main setting for the ITM stream port is the ITM port (0-31). A dedicated channel is needed
294for Tracealyzer. Port 0 is usually reserved for printf logging, so channel 1 is used by default.
295
296The option :guilabel:`Use internal buffer` should typically remain disabled. It buffers the data in RAM
297before transmission and defers the data transmission to the periodic TzCtrl thread.
298
299The host-side setup depends on what debug probe you are using. Learn more in the Tracealyzer
300User Manual.
301See :menuselection:`Creating and Loading Traces --> Percepio TraceRecorder --> Using TraceRecorder v4.6 or later --> Stream ports (or search for ITM)`.
302
303Tracealyzer Streaming from QEMU (Semihost)
304------------------------------------------
305
306This stream port is designed for Zephyr tracing in QEMU. This can be an easy way to get started
307with tracing and try out streaming trace without needing a fast debug probe. The data is streamed
308to a host file using semihosting. To use this option, apply the following configuration options:
309
310.. code-block:: cfg
311
312    CONFIG_SEMIHOST=y
313    CONFIG_TRACING=y
314    CONFIG_PERCEPIO_TRACERECORDER=y
315    CONFIG_PERCEPIO_TRC_START_MODE_START=y
316    CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_ZEPHYR_SEMIHOST=y
317
318Using menuconfig
319
320* Enable :menuselection:`General Architecture Options --> Semihosting support for Arm and RISC-V targets`
321* Enable :menuselection:`Subsystems and OS Services --> Tracing Support`
322* Under :menuselection:`Subsystems and OS Services --> Tracing Support --> Tracing Format`, select
323  :guilabel:`Percepio Tracealyzer`
324* Under :menuselection:`Modules --> percepio --> TraceRecorder --> Recorder Start Mode`, select
325  :guilabel:`Start`
326* Under :menuselection:`Modules --> percepio --> TraceRecorder --> Stream Port`, select
327  :guilabel:`Semihost`
328
329By default, the resulting trace file is found in :file:`./trace.psf` in the root of the build folder,
330unless a different path is specified. Open this file in `Percepio Tracealyzer`_ by selecting
331:menuselection:`File --> Open --> Open File`.
332
333Recorder Start Mode
334-------------------
335
336You may have noticed the :guilabel:`Recorder Start Mode` option in the Tracealyzer examples above.
337This decides when the tracing starts. With the option :guilabel:`Start`, the tracing begins directly
338at startup, once the TraceRecorder library has been initialized. This is recommended when using the
339Ring Buffer and Semihost stream ports.
340
341For streaming via RTT or ITM you may also use :guilabel:`Start From Host` or
342:guilabel:`Start Await Host`. Both listens for start commands from the Tracealyzer application. The
343latter option, :guilabel:`Start Await Host`, causes the TraceRecorder initialization to block until
344the start command is received from the Tracealyzer application.
345
346Custom Stream Ports for Tracealyzer
347-----------------------------------
348
349The stream ports are small modules within TraceRecorder that define what functions to call to
350output the trace data and (optionally) how to read start/stop commands from Tracealyzer.
351It is fairly easy to make custom stream ports to implement your own data transport and
352Tracealyzer can receive trace streams over various interfaces, including files, sockets,
353COM ports, named pipes and more. Note that additional stream port modules are available in the
354TraceRecorder repo (e.g. lwIP), although they might require modifications to work with Zephyr.
355
356Learning More
357-------------
358
359Learn more about how to get started in the `Tracealyzer Getting Started Guides`_.
360
361.. _Tracealyzer Getting Started Guides: https://percepio.com/tracealyzer/gettingstarted/
362
363
364SEGGER SystemView Support
365=========================
366
367Zephyr provides built-in support for `SEGGER SystemView`_ that can be enabled in
368any application for platforms that have the required hardware support.
369
370The payload and format used with SystemView is custom to the application and
371relies on RTT as a transport. Newer versions of SystemView support other
372transports, for example UART or using snapshot mode (both still not
373supported in Zephyr).
374
375To enable tracing support with `SEGGER SystemView`_ add the
376:ref:`snippet-rtt-tracing` to your build command:
377
378    .. zephyr-app-commands::
379        :zephyr-app: samples/synchronization
380        :board: <board>
381        :snippets: rtt-tracing
382        :goals: build
383        :compact:
384
385SystemView can also be used for post-mortem tracing, which can be enabled with
386:kconfig:option:`CONFIG_SEGGER_SYSVIEW_POST_MORTEM_MODE`. In this mode, a debugger can
387be attached after the system has crashed using ``west attach`` after which the
388latest data from the internal RAM buffer can be loaded into SystemView.
389
390.. figure:: segger_systemview.png
391    :align: center
392    :alt: SEGGER SystemView
393    :figclass: align-center
394    :width: 80%
395
396.. _SEGGER SystemView: https://www.segger.com/products/development-tools/systemview/
397
398
399Recent versions of `SEGGER SystemView`_ come with an API translation table for
400Zephyr which is incomplete and does not match the current level of support
401available in Zephyr. To use the latest Zephyr API description table, copy the
402file available in the tree to your local configuration directory to override the
403builtin table::
404
405        # On Linux and MacOS
406        cp $ZEPHYR_BASE/subsys/tracing/sysview/SYSVIEW_Zephyr.txt ~/.config/SEGGER/
407
408TraceCompass
409=============
410
411TraceCompass is an open source tool that visualizes CTF events such as thread
412scheduling and interrupts, and is helpful to find unintended interactions and
413resource conflicts on complex systems.
414
415See also the presentation by Ericsson,
416`Advanced Trouble-shooting Of Real-time Systems
417<https://wiki.eclipse.org/images/0/0e/TechTalkOnlineDemoFeb2017_v1.pdf>`_.
418
419
420User-Defined Tracing
421====================
422
423This tracing format allows the user to define functions to perform any work desired
424when a task is switched in or out, when an interrupt is entered or exited, and when the cpu
425is idle.
426
427Examples include:
428- simple toggling of GPIO for external scope tracing while minimizing extra cpu load
429- generating/outputting trace data in a non-standard or proprietary format that can
430not be supported by the other tracing systems
431
432The following functions can be defined by the user:
433
434.. code-block:: c
435
436   void sys_trace_thread_create_user(struct k_thread *thread);
437   void sys_trace_thread_abort_user(struct k_thread *thread);
438   void sys_trace_thread_suspend_user(struct k_thread *thread);
439   void sys_trace_thread_resume_user(struct k_thread *thread);
440   void sys_trace_thread_name_set_user(struct k_thread *thread);
441   void sys_trace_thread_switched_in_user(struct k_thread *thread);
442   void sys_trace_thread_switched_out_user(struct k_thread *thread);
443   void sys_trace_thread_info_user(struct k_thread *thread);
444   void sys_trace_thread_sched_ready_user(struct k_thread *thread);
445   void sys_trace_thread_pend_user(struct k_thread *thread);
446   void sys_trace_thread_priority_set_user(struct k_thread *thread, int prio);
447   void sys_trace_isr_enter_user(int nested_interrupts);
448   void sys_trace_isr_exit_user(int nested_interrupts);
449   void sys_trace_idle_user();
450
451Enable this format with the :kconfig:option:`CONFIG_TRACING_USER` option.
452
453Transport Backends
454******************
455
456The following backends are currently supported:
457
458* UART
459* USB
460* File (Using the native port with POSIX architecture based targets)
461* RTT (With SystemView)
462* RAM (buffer to be retrieved by a debugger)
463
464Using Tracing
465*************
466
467The sample :zephyr_file:`samples/subsys/tracing` demonstrates tracing with
468different formats and backends.
469
470To get started, the simplest way is to use the CTF format with the :ref:`native_sim <native_sim>`
471port, build the sample as follows:
472
473.. zephyr-app-commands::
474   :tool: all
475   :zephyr-app: samples/subsys/tracing
476   :board: native_sim
477   :gen-args: -DCONF_FILE=prj_native_ctf.conf
478   :goals: build
479
480You can then run the resulting binary with the option ``-trace-file`` to generate
481the tracing data::
482
483    mkdir data
484    cp $ZEPHYR_BASE/subsys/tracing/ctf/tsdl/metadata data/
485    ./build/zephyr/zephyr.exe -trace-file=data/channel0_0
486
487The resulting CTF output can be visualized using babeltrace or TraceCompass
488by pointing the tool to the ``data`` directory with the metadata and trace files.
489
490Using RAM backend
491=================
492
493For devices that do not have available I/O for tracing such as USB or UART but have
494enough RAM to collect trace data, the ram backend can be enabled with configuration
495:kconfig:option:`CONFIG_TRACING_BACKEND_RAM`.
496Adjust :kconfig:option:`CONFIG_RAM_TRACING_BUFFER_SIZE` to be able to record enough traces for your needs.
497Then thanks to a runtime debugger such as gdb this buffer can be fetched from the target
498to an host computer::
499
500    (gdb) dump binary memory data/channel0_0 <ram_tracing_start> <ram_tracing_end>
501
502The resulting channel0_0 file have to be placed in a directory with the ``metadata``
503file like the other backend.
504
505Future LTTng Inspiration
506************************
507
508Currently, the top-layer provided here is quite simple and bare-bones,
509and needlessly copied from Zephyr's Segger SystemView debug module.
510
511For an OS like Zephyr, it would make sense to draw inspiration from
512Linux's LTTng and change the top-layer to serialize to the same format.
513Doing this would enable direct reuse of TraceCompass' canned analyses
514for Linux.  Alternatively, LTTng-analyses in TraceCompass could be
515customized to Zephyr.  It is ongoing work to enable TraceCompass
516visibility of Zephyr in a target-agnostic and open source way.
517
518
519I/O Taxonomy
520=============
521
522- Atomic Push/Produce/Write/Enqueue:
523
524  - synchronous:
525                  means data-transmission has completed with the return of the
526                  call.
527
528  - asynchronous:
529                  means data-transmission is pending or ongoing with the return
530                  of the call. Usually, interrupts/callbacks/signals or polling
531                  is used to determine completion.
532
533  - buffered:
534                  means data-transmissions are copied and grouped together to
535                  form a larger ones. Usually for amortizing overhead (burst
536                  dequeue) or jitter-mitigation (steady dequeue).
537
538  Examples:
539    - sync  unbuffered
540        E.g. PIO via GPIOs having steady stream, no extra FIFO memory needed.
541        Low jitter but may be less efficient (can't amortize the overhead of
542        writing).
543
544    - sync  buffered
545        E.g. ``fwrite()`` or enqueuing into FIFO.
546        Blockingly burst the FIFO when its buffer-waterlevel exceeds threshold.
547        Jitter due to bursts may lead to missed deadlines.
548
549    - async unbuffered
550        E.g. DMA, or zero-copying in shared memory.
551        Be careful of data hazards, race conditions, etc!
552
553    - async buffered
554        E.g. enqueuing into FIFO.
555
556
557
558- Atomic Pull/Consume/Read/Dequeue:
559
560  - synchronous:
561                  means data-reception has completed with the return of the call.
562
563  - asynchronous:
564                  means data-reception is pending or ongoing with the return of
565                  the call. Usually, interrupts/callbacks/signals or polling is
566                  used to determine completion.
567
568  - buffered:
569                  means data is copied-in in larger chunks than request-size.
570                  Usually for amortizing wait-time.
571
572  Examples:
573    - sync  unbuffered
574        E.g. Blocking read-call, ``fread()`` or SPI-read, zero-copying in shared
575        memory.
576
577    - sync  buffered
578        E.g. Blocking read-call with caching applied.
579        Makes sense if read pattern exhibits spatial locality.
580
581    - async unbuffered
582        E.g. zero-copying in shared memory.
583        Be careful of data hazards, race conditions, etc!
584
585    - async buffered
586        E.g. ``aio_read()`` or DMA.
587
588
589
590Unfortunately, I/O may not be atomic and may, therefore, require locking.
591Locking may not be needed if multiple independent channels are available.
592
593  - The system has non-atomic write and one shared channel
594        E.g. UART. Locking required.
595
596        ``lock(); emit(a); emit(b); emit(c); release();``
597
598  - The system has non-atomic write but many channels
599        E.g. Multi-UART. Lock-free if the bottom-layer maps each Zephyr
600        thread+ISR to its own channel, thus alleviating races as each
601        thread is sequentially consistent with itself.
602
603        ``emit(a,thread_id); emit(b,thread_id); emit(c,thread_id);``
604
605  - The system has atomic write     but one shared channel
606        E.g. ``native_sim`` or board with DMA. May or may not need locking.
607
608        ``emit(a ## b ## c); /* Concat to buffer */``
609
610        ``lock(); emit(a); emit(b); emit(c); release(); /* No extra mem */``
611
612  - The system has atomic write     and many channels
613        E.g. native_sim or board with multi-channel DMA. Lock-free.
614
615        ``emit(a ## b ## c, thread_id);``
616
617
618Object tracking
619***************
620
621The kernel can also maintain lists of objects that can be used to track
622their usage. Currently, the following lists can be enabled::
623
624  struct k_timer *_track_list_k_timer;
625  struct k_mem_slab *_track_list_k_mem_slab;
626  struct k_sem *_track_list_k_sem;
627  struct k_mutex *_track_list_k_mutex;
628  struct k_stack *_track_list_k_stack;
629  struct k_msgq *_track_list_k_msgq;
630  struct k_mbox *_track_list_k_mbox;
631  struct k_pipe *_track_list_k_pipe;
632  struct k_queue *_track_list_k_queue;
633  struct k_event *_track_list_k_event;
634
635Those global variables are the head of each list - they can be traversed
636with the help of macro ``SYS_PORT_TRACK_NEXT``. For instance, to traverse
637all initialized mutexes, one can write::
638
639  struct k_mutex *cur = _track_list_k_mutex;
640  while (cur != NULL) {
641    /* Do something */
642
643    cur = SYS_PORT_TRACK_NEXT(cur);
644  }
645
646To enable object tracking, enable :kconfig:option:`CONFIG_TRACING_OBJECT_TRACKING`.
647Note that each list can be enabled or disabled via their tracing
648configuration. For example, to disable tracking of semaphores, one can
649disable :kconfig:option:`CONFIG_TRACING_SEMAPHORE`.
650
651Object tracking is behind tracing configuration as it currently leverages
652tracing infrastructure to perform the tracking.
653
654API
655***
656
657
658Common
659======
660
661.. doxygengroup:: subsys_tracing_apis
662
663Threads
664=======
665
666.. doxygengroup:: subsys_tracing_apis_thread
667
668Work Queues
669===========
670
671.. doxygengroup:: subsys_tracing_apis_work
672
673Poll
674====
675
676.. doxygengroup:: subsys_tracing_apis_poll
677
678Semaphore
679=========
680
681.. doxygengroup:: subsys_tracing_apis_sem
682
683Mutex
684=====
685
686.. doxygengroup:: subsys_tracing_apis_mutex
687
688Condition Variables
689===================
690
691.. doxygengroup:: subsys_tracing_apis_condvar
692
693Queues
694======
695
696.. doxygengroup:: subsys_tracing_apis_queue
697
698FIFO
699====
700
701.. doxygengroup:: subsys_tracing_apis_fifo
702
703LIFO
704====
705.. doxygengroup:: subsys_tracing_apis_lifo
706
707Stacks
708======
709
710.. doxygengroup:: subsys_tracing_apis_stack
711
712Message Queues
713==============
714
715.. doxygengroup:: subsys_tracing_apis_msgq
716
717Mailbox
718=======
719
720.. doxygengroup:: subsys_tracing_apis_mbox
721
722Pipes
723======
724
725.. doxygengroup:: subsys_tracing_apis_pipe
726
727Heaps
728=====
729
730.. doxygengroup:: subsys_tracing_apis_heap
731
732Memory Slabs
733============
734
735.. doxygengroup:: subsys_tracing_apis_mslab
736
737Timers
738======
739
740.. doxygengroup:: subsys_tracing_apis_timer
741
742Object tracking
743===============
744
745.. doxygengroup:: subsys_tracing_object_tracking
746
747Syscalls
748========
749
750.. doxygengroup:: subsys_tracing_apis_syscall
751
752Network tracing
753===============
754
755.. doxygengroup:: subsys_tracing_apis_net
756
757Network socket tracing
758======================
759
760.. doxygengroup:: subsys_tracing_apis_socket
761