Lines Matching refs:the
3 # Licensed under the GNU Free Documentation License, Version 1.2
9 This document tries to describe the design of the rtmutex.c implementation.
10 It doesn't describe the reasons why rtmutex.c exists. For that please see
12 that happen without this code, but that is in the concept to understand
13 what the code actually is doing.
15 The goal of this document is to help others understand the priority
16 inheritance (PI) algorithm that is used, as well as reasons for the
17 decisions that were made to implement PI in the manner that was done.
25 most of the time it can't be helped. Anytime a high priority process wants
27 the high priority process must wait until the lower priority process is done
28 with the resource. This is a priority inversion. What we want to prevent
29 is something called unbounded priority inversion. That is when the high
34 processes, let's call them processes A, B, and C, where A is the highest
35 priority process, C is the lowest, and B is in between. A tries to grab a lock
36 that C owns and must wait and lets C run to release the lock. But in the
40 to release the lock, because for all we know, B is a CPU hog and will
41 never give C a chance to release the lock. This is called unbounded priority
44 Here's a little ASCII art to show the problem.
63 PI is where a process inherits the priority of another process if the other
64 process blocks on a lock owned by the current process. To make this easier
65 to understand, let's use the previous example, with processes A, B, and C again.
67 This time, when A blocks on the lock owned by C, C would inherit the priority
69 the high priority of A. As soon as C releases the lock, it loses its
70 inherited priority, and A then can continue with the resource that C had.
76 the design that is used to implement PI.
84 PI and spin locks that are used in the PI code, from now on
85 the PI locks will be called a mutex.
87 lock - In this document from now on, I will use the term lock when
88 referring to spin locks that are used to protect parts of the PI
95 waiter - A waiter is a struct that is stored on the stack of a blocked
96 process. Since the scope of the waiter is within the code for
97 a process being blocked on the mutex, it is fine to allocate
98 the waiter on the process's stack (local variable). This
99 structure holds a pointer to the task, as well as the mutex that
100 the task is blocked on. It also has rbtree node structures to
101 place the task in the waiters rbtree of a mutex as well as the
104 waiter is sometimes used in reference to the task that is waiting
105 on a mutex. This is the same as waiter->task.
111 top pi waiter - The highest priority process waiting on one of the mutexes
152 one, the chains merge.
162 For PI to work, the processes at the right end of these chains (or we may
163 also call it the Top of the chain) must be equal to or higher in priority
164 than the processes to the left or below in the chain.
172 And once again, to show how this can grow I will show the merging chains
182 If process G has the highest priority in the chain, then all the tasks up
183 the chain (A and B in this example), must have their priorities increased
189 Every mutex keeps track of all the waiters that are blocked on itself. The
191 by a spin lock that is located in the struct of the mutex. This lock is called
198 To keep track of the PI chains, each process has its own PI rbtree. This is
199 a tree of all top waiters of the mutexes that are owned by the process.
200 Note that this tree only holds the top waiters and not all waiters that are
201 blocked on mutexes owned by the process.
203 The top of the task's PI tree is always the highest priority task that
204 is waiting on a mutex that is owned by the task. So if the task has
205 inherited a priority, it will always be the priority of the task that is
206 at the top of this tree.
208 This tree is stored in the task structure of a process as a rbtree called
209 pi_waiters. It is protected by a spin lock also in the task structure,
211 locking the pi_lock, interrupts must be disabled.
214 Depth of the PI Chain
217 The maximum depth of the PI chain is not dynamic, and could actually be
219 the nesting of mutexes. Let's look at the example where we have 3 mutexes,
267 in func4 in the "do something again" area, we have a locking that follows:
276 And thus we have the chain A->L1->B->L2->C->L3->D.
278 This gives us a PI depth of 4 (four processes), but looking at any of the
280 depth of two. So, although the locking depth is defined at compile time,
281 it still is very difficult to find the possibilities of that depth.
285 PI chain, and have the code holding spin locks while looking at a large
286 amount of data. So to prevent this, the implementation not only implements
288 time, as it walks the PI chain. More about this below.
294 The mutex structure contains a pointer to the owner of the mutex. If the
296 have the task structure on at least a two byte alignment (and if this is
297 not true, the rtmutex.c code will be broken!), this allows for the least
298 significant bit to be used as a flag. Bit 0 is used as the "Has Waiters"
307 is used (when applicable) to keep the fast path of grabbing and releasing
310 cmpxchg is basically the following function performed atomically:
323 if the variable is what you expect it to be. You know if it succeeded if
324 the return value (the old value of A) is equal to B.
327 the architecture does not support CMPXCHG, then this macro is simply set
329 help out extremely to keep the fast path short.
331 The use of rt_mutex_cmpxchg with the flags in the owner field help optimize
332 the system for architectures that support it. This will also be explained
339 The implementation of the PI code in rtmutex.c has several places that a
340 process must adjust its priority. With the help of the pi_waiters of a
343 The functions implementing the task adjustments are rt_mutex_adjust_prio
346 rt_mutex_adjust_prio examines the priority of the task, and the highest
347 priority process that is waiting any of mutexes owned by the task. Since
348 the pi_waiters of a task holds an order by priority of all the top waiters
349 of all the mutexes that the task owns, we simply need to compare the top
350 pi waiter to its own normal/deadline priority and take the higher one.
351 Then rt_mutex_setprio is called to adjust the priority of the task to the
353 to implement the actual change in priority.
355 (Note: For the "prio" field in task_struct, the lower the number, the
356 higher the priority. A "prio" of 5 is of higher priority than a
360 or decrease the priority of the task. In the case that a higher priority
361 process has just blocked on a mutex owned by the task, rt_mutex_adjust_prio
362 would increase/boost the task's priority. But if a higher priority task
363 were for some reason to leave the mutex (timeout or signal), this same function
364 would decrease/unboost the priority of the task. That is because the pi_waiters
365 always contains the highest priority task that is waiting on a mutex owned
366 by the task, so we only need to compare the priority of that top pi waiter
367 to the normal priority of the given task.
370 High level overview of the PI chain walk
373 The PI chain walk is implemented by the function rt_mutex_adjust_prio_chain.
376 with what we believe is the best. It walks the PI chain by only grabbing
383 (de)boosting (the owner of a mutex that a process is blocking on), a flag to
384 check for deadlocking, the mutex that the task owns, a pointer to a waiter
385 that is the process's waiter struct that is blocked on the mutex (although this
386 parameter may be NULL for deboosting), a pointer to the mutex on which the task
387 is blocked, and a top_task as the top waiter of the mutex.
393 that the state of the owner and lock can change when entered into this function.
395 Before this function is called, the task has already had rt_mutex_adjust_prio
396 performed on it. This means that the task is set to the priority that it
397 should be at, but the rbtree nodes of the task's waiter have not been updated
398 with the new priorities, and this task may not be in the proper locations
399 in the pi_waiters and waiters trees that the task is blocked on. This function
403 rtmutex.c. See the 'Chain walk basics and protection scope' comment for further
409 OK, now let's take a look at the detailed walk through of what happens when
412 The first thing that is tried is the fast taking of the mutex. This is
413 done when we have CMPXCHG enabled (otherwise the fast taking automatically
414 fails). Only when the owner field of the mutex is NULL can the lock be
415 taken with the CMPXCHG and nothing else needs to be done.
417 If there is contention on the lock, we go about the slow path
420 The slow path function is where the task's waiter structure is created on
421 the stack. This is because the waiter structure is only needed for the
422 scope of this function. The waiter structure holds the nodes to store
423 the task on the waiters tree of the mutex, and if need be, the pi_waiters
424 tree of the owner.
426 The wait_lock of the mutex is taken since the slow path of unlocking the
429 We then call try_to_take_rt_mutex. This is where the architecture that
430 does not implement CMPXCHG would always grab the lock (if there's no
433 try_to_take_rt_mutex is used every time the task tries to grab a mutex in the
435 the "Has Waiters" flag of the mutex's owner field. By setting this flag
436 now, the current owner of the mutex being contended for can't release the mutex
437 without going into the slow unlock path, and it would then need to grab the
438 wait_lock, which this code currently holds. So setting the "Has Waiters" flag
439 forces the current owner to synchronize with this code.
441 The lock is taken if the following are true:
443 2) The current task is the highest priority against all other
444 waiters of the lock
446 If the task succeeds to acquire the lock, then the task is set as the
447 owner of the lock, and if the lock still has waiters, the top_waiter
448 (highest priority task waiting on the lock) is added to this task's
451 If the lock is not taken by try_to_take_rt_mutex(), then the
452 task_blocks_on_rt_mutex() function is called. This will add the task to
453 the lock's waiter tree and propagate the pi chain of the lock as well
454 as the lock's owner's pi_waiters tree. This is described in the next
460 The accounting of a mutex and process is done with the waiter structure of
461 the process. The "task" field is set to the process, and the "lock" field
462 to the mutex. The rbtree node of waiter are initialized to the processes
465 Since the wait_lock was taken at the entry of the slow lock, we can safely
466 add the waiter to the task waiter tree. If the current process is the
467 highest priority process currently waiting on this mutex, then we remove the
468 previous top waiter process (if it exists) from the pi_waiters of the owner,
469 and add the current process to that tree. Since the pi_waiter of the owner
470 has changed, we call rt_mutex_adjust_prio on the owner to see if the owner
473 If the owner is also blocked on a lock, and had its pi_waiters changed
474 (or deadlock checking is on), we unlock the wait_lock of the mutex and go ahead
475 and run rt_mutex_adjust_prio_chain on the owner, as described earlier.
477 Now all locks are released, and if the current process is still blocked on a
480 Waking up in the loop
484 1) The previous lock owner released the lock, and the task now is top_waiter
487 In both cases, the task will try again to acquire the lock. If it
488 does, then it will take itself off the waiters tree and set itself back
489 to the TASK_RUNNING state.
491 In first case, if the lock was acquired by another task before this task
492 could get the lock, then it will go back to sleep and wait to be woken again.
495 that can wake up before getting the lock, either due to a signal or
497 take the lock again, if it succeeds, then the task will return with the
498 lock held, otherwise it will return with -EINTR if the task was woken
502 Unlocking the Mutex
506 CMPXCHG. Since the taking of a mutex on contention always sets the
507 "Has Waiters" flag of the mutex's owner, we use this to know if we need to
508 take the slow path when unlocking the mutex. If the mutex doesn't have any
509 waiters, the owner field of the mutex would equal the current process and
510 the mutex can be unlocked by just replacing the owner field with NULL.
512 If the owner field has the "Has Waiters" bit set (or CMPXCHG is not available),
513 the slow unlock path is taken.
515 The first thing done in the slow unlock path is to take the wait_lock of the
516 mutex. This synchronizes the locking and unlocking of the mutex.
518 A check is made to see if the mutex has waiters or not. On architectures that
519 do not have CMPXCHG, this is the location that the owner of the mutex will
521 do have CMPXCHG, that check is done in the fast path, but it is still needed
522 in the slow path too. If a waiter of a mutex woke up because of a signal
523 or timeout between the time the owner failed the fast path CMPXCHG check and
524 the grabbing of the wait_lock, the mutex may not have any waiters, thus the
525 owner still needs to make this check. If there are no waiters then the mutex
526 owner field is set to NULL, the wait_lock is released and nothing more is
531 On the wake up code, the pi_lock of the current owner is taken. The top
532 waiter of the lock is found and removed from the waiters tree of the mutex
533 as well as the pi_waiters tree of the current owner. The "Has Waiters" bit is
534 marked to prevent lower priority tasks from stealing the lock.
536 Finally we unlock the pi_lock of the pending owner and wake it up.