1# Blocking the Log Processing Thread
2
3## Overview
4
5When the core log buffer becomes full, the logging subsystem can be configured to
6
7* Drop older log messages with `CONFIG_LOG_MODE_OVERFLOW=y` (**default**)
8* Drop newer log messages with `CONFIG_LOG_MODE_OVERFLOW=n`, or
9* Drop no log messages at all with `CONFIG_LOG_BLOCK_IN_THREAD=y`, `CONFIG_LOG_BLOCK_IN_THREAD_TIMEOUT_MS=-1`.
10
11In the last configuration, the log processing thread will block until space
12becomes available again in the core log buffer.
13
14> Warning ⚠️: Blocking the log processing thread is generally not recommended
15> and should only be attempted in advanced use cases.
16
17## Logging and Flow Rates
18
19There are roughly 4 scenarios we care about testing with
20`CONFIG_LOG_BLOCK_IN_THREAD`, and they can all be characterized by comparing
21log message flow rates. Typically, one would describe log message flow rates
22with units such as `[msg/s]`. However, in the table below, we are mainly
23concerned with the ratio of the Output Rate to the Input Rate, and in that
24case, the units themselves cancel-out. In the table we assume there exists an
25`N` such that `N > 1`.
26
27| Name           | Input Rate | Output Rate | Rate |
28|----------------|------------|-------------|------|
29| Input-Limited  | 1          | N           | 1    |
30| Matched        | 1          | 1           | 1    |
31| Output-Limited | 1          | 1/N         | 1/N  |
32| Stalled        | 0          | 0           | 0    |
33
34The resultant _Rate_ is always `Rate = MIN(Input Rate, Output Rate)`.
35
36Rate-limiting of any kind can be described approximately as _back pressure_.
37Back pressure is fine in short bursts but it can cause delays in application
38and driver code if the pressure is not relieved promptly.
39
40## Physical Sources of Backpressure
41
42Many log backends, such as UARTs, have a built-in hardware FIFO that
43inherently provides back-pressure; output log processing is rate-limited
44based on the baud rate of the UART. Other backends, such as UDP sockets or
45DMA, can provide significantly higher throughput but are still inherently
46rate-limited by the physical layer over which they operate, be it Gigabit
47Ethernet or PCI express.
48
49Even a trivial _message source_ or _message sink_ is still rate-limited by
50memory or the CPU. From that perspective, we can infer that there is a finite
51limit in the log processing rate for practical systems. That may be
52comforting to know, even if it is something astronomical like 1G `[msg/s]`.
53
54## Input-Limited Log Rate
55
56The ideal scenario is when the output "bandwidth" exceeds the input rate. If
57so configured, we minimize the liklihood that the log processing thread will
58stall. We can also be sure that the output will be able to relieve
59backpressure (i.e. the core log buffer usage will tend to zero over time).
60
61## Rate-Matched Input and Output
62
63When the input rate and output rates are equal, one might think this is the
64ideal scenario. In reality, it is not. The rates could be matched, but a
65sustained increase (or several small increases) in the input log rate, could
66cause the core log buffer to approach 100% capacity. Since the output log rate
67is still only matched with the input log rate, the core log buffer capacity
68would not decrease from 100%, and it would remain saturated.
69
70Logging has a tendency to be bursty, so it is definitely preferable to
71operate in the _Input-limited Log Rate_ regime.
72
73## Output-Limited Log Rate
74
75If the rate of output processing is less than the rate of input processing,
76the core log buffer will approach 100% capacity and, eventually, stall the
77log processing thread.
78
79## Stalling the Log Processing Thread
80
81When any log backend is unable to process logs for whatever reason,
82the output rate approaches 0 `[msg/s]`. If application or
83driver code continue to submit logs, the core log buffer approaches 100%
84capacity. Once the core log buffer is full, the log processing thread is
85unable to allocate new log messages and it will be stalled.
86
87Stalling a real-time application produces unexpected behaviour, so it is
88advised to avoid this for any non-negligible amount of time.
89
90It is absolutely critical that the log backend is capable of operating
91correctly _even when the log processing thread is blocking_ in order to
92automatically recover from a stall.
93
94On a live system, it may be necessary to manually perform remediation of log
95backends that are unable to recover from stalling the log processing thread.
96Remediation could involve disabling the log backend and freeing any in-use
97buffers.
98