Lines Matching +full:cpu +full:- +full:idle +full:- +full:states
1 .. SPDX-License-Identifier: GPL-2.0
5 .. |cpufreq| replace:: :doc:`CPU Performance Scaling <cpufreq>`
8 CPU Idle Time Management
19 Modern processors are generally able to enter states in which the execution of
21 memory or executed. Those states are the *idle* states of the processor.
23 Since part of the processor hardware is not used in idle states, entering them
27 CPU idle time management is an energy-efficiency feature concerned about using
28 the idle states of processors for this purpose.
31 ------------
33 CPU idle time management operates on CPUs as seen by the *CPU scheduler* (that
37 software as individual single-core processors. In other words, a CPU is an
43 program) at a time, it is a CPU. In that case, if the hardware is asked to
44 enter an idle state, that applies to the processor as a whole.
46 Second, if the processor is multi-core, each core in it is able to follow at
52 enter an idle state, that applies to the core that asked for it in the first
56 except for one have been put into idle states at the "core level" and the
57 remaining core asks the processor to enter an idle state, that may trigger it
58 to put the whole larger unit into an idle state which also will affect the
61 Finally, each core in a multi-core processor may be able to follow more than one
66 multiple individual single-core "processors", referred to as *hardware threads*
67 (or hyper-threads specifically on Intel hardware), that each can follow one
68 sequence of instructions. Then, the hardware threads are CPUs from the CPU idle
69 time management perspective and if the processor is asked to enter an idle state
70 by one of them, the hardware thread (or CPU) that asked for it is stopped, but
72 core also have asked the processor to enter an idle state. In that situation,
73 the core may be put into an idle state individually or a larger unit containing
74 it may be put into an idle state as a whole (if the other cores within the
75 larger unit are in idle states already).
77 Idle CPUs
78 ---------
81 *idle* by the Linux kernel when there are no tasks to run on them except for the
82 special "idle" task.
84 Tasks are the CPU scheduler's representation of work. Each task consists of a
87 processor every time the task's code is run by a CPU. The CPU scheduler
90 Tasks can be in various states. In particular, they are *runnable* if there are
91 no specific conditions preventing their code from being run by a CPU as long as
92 there is a CPU available for that (for example, they are not waiting for any
93 events to occur or similar). When a task becomes runnable, the CPU scheduler
95 tasks assigned to it, the CPU will load the given task's context and run its
97 another CPU). [If there are multiple runnable tasks assigned to one CPU
101 The special "idle" task becomes runnable if there are no other runnable tasks
102 assigned to the given CPU and the CPU is then regarded as idle. In other words,
103 in Linux idle CPUs run the code of the "idle" task called *the idle loop*. That
104 code may cause the processor to be put into one of its idle states, if they are
106 idle states, or there is not enough time to spend in an idle state before the
108 available idle states from being used, the CPU will simply execute more or less
112 .. _idle-loop:
114 The Idle Loop
117 The idle loop code takes two major steps in every iteration of it. First, it
118 calls into a code module referred to as the *governor* that belongs to the CPU
119 idle time management subsystem called ``CPUIdle`` to select an idle state for
120 the CPU to ask the hardware to enter. Second, it invokes another code module
122 processor hardware to enter the idle state selected by the governor.
124 The role of the governor is to find an idle state most suitable for the
125 conditions at hand. For this purpose, idle states that the hardware can be
127 the platform or the processor architecture and organized in a one-dimensional
133 Each idle state present in that array is characterized by two parameters to be
134 taken into account by the governor, the *target residency* and the (worst-case)
138 the shallower idle states instead. [The "depth" of an idle state roughly
140 latency, in turn, is the maximum time it will take a CPU asking the processor
141 hardware to enter an idle state to start executing the first instruction after a
151 CPU depends on can spend in an idle state, including the time necessary to enter
152 and exit it. However, the CPU may be woken up by a non-timer event at any time
154 when that may happen. The governor can only see how much time the CPU actually
155 was idle after it has been woken up (that time will be referred to as the *idle
157 time until the closest timer to estimate the idle duration in future. How the
162 There are four ``CPUIdle`` governors available, ``menu``, `TEO <teo-gov_>`_,
165 tick can be `stopped by the idle loop <idle-cpus-and-tick_>`_. Available
169 :file:`current_governor` file under :file:`/sys/devices/system/cpu/cpuidle/`
176 hardcoded idle states information and the other able to read that information
183 file under :file:`/sys/devices/system/cpu/cpuidle/` in ``sysfs``.
186 .. _idle-cpus-and-tick:
188 Idle CPUs and The Scheduler Tick
192 the time sharing strategy of the CPU scheduler. Of course, if there are
193 multiple runnable tasks assigned to one CPU at the same time, the only way to
195 share the available CPU time. Namely, in rough approximation, each task is
196 given a slice of the CPU time to run its code, subject to the scheduling class,
197 prioritization and so on and when that time slice is used up, the CPU should be
199 may not want to give the CPU away voluntarily, however, and the scheduler tick
203 The scheduler tick is problematic from the CPU idle time management perspective,
206 Thus, if the tick is allowed to trigger on idle CPUs, it will not make sense
207 for them to ask the hardware to enter idle states with target residencies above
208 the tick period length. Moreover, in that case the idle duration of any CPU
210 exiting idle states due to the tick wakeups on idle CPUs will be wasted.
212 Fortunately, it is not really necessary to allow the tick to trigger on idle
214 "idle" one. In other words, from the CPU scheduler perspective, the only user
215 of the CPU time on them is the idle loop. Since the time of an idle CPU need
217 tick goes away if the given CPU is idle. Consequently, it is possible to stop
218 the scheduler tick entirely on idle CPUs in principle, even though that may not
221 Whether or not it makes sense to stop the scheduler tick in the idle loop
223 (non-tick) timer due to trigger within the tick range, stopping the tick clearly
225 reprogrammed in that case. Second, if the governor is expecting a non-timer
227 be harmful. Namely, in that case the governor will select an idle state with
229 going to be relatively shallow. The governor really cannot select a deep idle
235 in the shallow idle state selected by the governor, which will be a waste of
238 governor will select a relatively deep idle state, so the tick should be stopped
239 so that it does not wake up the CPU too early.
246 The kernel can be configured to disable stopping the scheduler tick in the idle
247 loop altogether. That can be done through the build-time configuration of it
251 ignored by the idle loop code and the tick is never stopped.
254 stopped on idle CPUs are referred to as *tickless* systems and they are
255 generally regarded as more energy-efficient than the systems running kernels in
261 .. _menu-gov:
268 Namely, when invoked to select an idle state for a CPU (i.e. an idle state that
269 the CPU will ask the processor hardware to enter), it attempts to predict the
270 idle duration and uses the predicted value for idle state selection.
274 length* in what follows, is the upper bound on the time before the next CPU
279 One of them is used when tasks previously running on the given CPU are waiting
286 selecting the idle state for the CPU) is updated after the CPU has been woken
287 up and the closer the sleep length is to the observed idle duration, the closer
290 falls into to obtain the first approximation of the predicted idle duration.
293 idle duration prediction. Namely, it saves the last 8 observed idle duration
294 values and, when predicting the idle duration next time, it computes the average
298 interval" value. Otherwise, the longest of the saved observed idle duration
306 taken as the predicted idle duration.
310 idle state is comparable with the predicted idle duration, the total time spent
315 of the extra latency limit is the predicted idle duration itself which
317 previously ran on the given CPU and now they are waiting for I/O operations to
319 from the power management quality of service, or `PM QoS <cpu-pm-qos_>`_,
320 framework and the minimum of the two is taken as the limit for the idle states'
323 Now, the governor is ready to walk the list of idle states and choose one of
325 the predicted idle duration and the exit latency of it with the computed latency
327 idle duration, but still below it, and exit latency that does not exceed the
330 In the final step the governor may still need to refine the idle state selection
331 if it has not decided to `stop the scheduler tick <idle-cpus-and-tick_>`_. That
332 happens if the idle duration predicted by it is less than the tick period and
333 the tick has not been stopped already (in a previous iteration of the idle
340 .. _teo-gov:
347 <menu-gov_>`_: it always tries to find the deepest idle state suitable for the
351 to correlate the observed idle duration values with the available idle states
352 and use that information to pick up the idle state that is most likely to
353 "match" the upcoming CPU idle interval. Second, it does not take the tasks
354 that were running on the given CPU in the past and are waiting on some I/O
356 the same CPU when they become runnable again) and the pattern detection code in
357 it avoids taking timer wakeups into account. It also only uses idle duration
361 Like in the ``menu`` governor `case <menu-gov_>`_, the first step is to obtain
364 on the time until the next CPU wakeup). That value is then used to preselect an
365 idle state on the basis of three metrics maintained for each idle state provided
368 The ``hits`` and ``misses`` metrics measure the likelihood that a given idle
369 state will "match" the observed (post-wakeup) idle duration if it "matches" the
370 sleep length. They both are subject to decay (after a CPU wakeup) every time
371 the target residency of the idle state corresponding to them is less than or
372 equal to the sleep length and the target residency of the next idle state is
373 greater than the sleep length (that is, when the idle state corresponding to
375 former condition is satisfied and the target residency of the given idle state
376 is less than or equal to the observed idle duration and the target residency of
377 the next idle state is greater than the observed idle duration at the same time
378 (that is, it is increased when the given idle state "matches" both the sleep
379 length and the observed idle duration). In turn, the ``misses`` metric is
380 increased when the given idle state "matches" the sleep length only and the
381 observed idle duration is too short for its target residency.
383 The ``early_hits`` metric measures the likelihood that a given idle state will
384 "match" the observed (post-wakeup) idle duration if it does not "match" the
385 sleep length. It is subject to decay on every CPU wakeup and it is increased
386 when the idle state corresponding to it "matches" the observed (post-wakeup)
387 idle duration and the target residency of the next idle state is less than or
388 equal to the sleep length (i.e. the idle state "matching" the sleep length is
391 The governor walks the list of idle states provided by the ``CPUIdle`` driver
393 to the sleep length. Then, the ``hits`` and ``misses`` metrics of that idle
395 greater (which means that that idle state is likely to "match" the observed idle
396 duration after CPU wakeup). If the ``misses`` one is greater, the governor
397 preselects the shallower idle state with the maximum ``early_hits`` metric
398 (or if there are multiple shallower idle states with equal ``early_hits``
401 <cpu-pm-qos_>`_ which is hit before reaching the deepest idle state with the
402 target residency within the sleep length, the deepest idle state with the exit
406 Next, the governor takes several idle duration values observed most recently
408 the target residency of the preselected idle state, that idle state becomes the
409 final candidate to ask for. Otherwise, the average of the most recent idle
410 duration values below the target residency of the preselected idle state is
411 computed and the governor walks the idle states shallower than the preselected
413 That idle state is then taken as the final candidate to ask for.
415 Still, at this point the governor may need to refine the idle state selection if
416 it has not decided to `stop the scheduler tick <idle-cpus-and-tick_>`_. That
417 generally happens if the target residency of the idle state selected so far is
419 previous iteration of the idle loop). Then, like in the ``menu`` governor
420 `case <menu-gov_>`_, the sleep length used in the previous computations may not
426 .. _idle-states-representation:
428 Representation of Idle States
431 For the CPU idle time management purposes all of the physical idle states
432 supported by the processor have to be represented as a one-dimensional array of
433 |struct cpuidle_state| objects each allowing an individual (logical) CPU to ask
434 the processor hardware to enter an idle state of certain properties. If there
436 cover a combination of idle states supported by the units at different levels of
438 of it <idle-loop_>`_, must reflect the properties of the idle state at the
439 deepest level (i.e. the idle state of the unit containing all of the other
443 a "module" and suppose that asking the hardware to enter a specific idle state
445 enter a specific idle state of its own (say "MX") if the other core is in idle
446 state "X" already. In other words, asking for idle state "X" at the "core"
447 level gives the hardware a license to go as deep as to idle state "MX" at the
449 asking for idle state "X" may just end up in that state by itself instead).
451 idle state "X" must reflect the minimum time to spend in idle state "MX" of
453 time the CPU needs to be idle to save any energy in case the hardware enters
455 the exit time of idle state "MX" of the module (and usually its entry time too),
456 because that is the maximum delay between a wakeup signal and the time the CPU
462 hierarchy of units inside them, however. In those cases asking for an idle
465 handling of the hierarchy. Then, the definition of the idle state objects is
466 entirely up to the driver, but still the physical properties of the idle state
468 used by the governor for idle state selection (for instance, the actual exit
469 latency of that idle state must not exceed the exit latency parameter of the
470 idle state object selected by the governor).
472 In addition to the target residency and exit latency idle state parameters
473 discussed above, the objects representing idle states each contain a few other
474 parameters describing the idle state and a pointer to the function to run in
478 statistics of the given idle state. That information is exposed by the kernel
481 For each CPU in the system, there is a :file:`/sys/devices/system/cpu/cpu<N>/cpuidle/`
483 CPU at the initialization time. That directory contains a set of subdirectories
484 called :file:`state0`, :file:`state1` and so on, up to the number of idle state
485 objects defined for the given CPU minus one. Each of these directories
486 corresponds to one idle state object and the larger the number in its name, the
487 deeper the (effective) idle state represented by it. Each of them contains
488 a number of files (attributes) representing the properties of the idle state
492 Total number of times this idle state had been asked for, but the
493 observed idle duration was certainly too short to match its target
497 Total number of times this idle state had been asked for, but certainly
498 a deeper idle state would have been a better match for the observed idle
502 Description of the idle state.
505 Whether or not this idle state is disabled.
511 Exit latency of the idle state in microseconds.
514 Name of the idle state.
517 Power drawn by hardware in this idle state in milliwatts (if specified,
521 Target residency of the idle state in microseconds.
524 Total time spent in this idle state by the given CPU (as measured by the
528 Total number of times the hardware has been asked by the given CPU to
529 enter this idle state.
532 Total number of times a request to enter this idle state on the given
533 CPU was rejected.
541 given idle state is disabled for this particular CPU, which means that the
542 governor will never select it for this particular CPU and the ``CPUIdle``
543 driver will never ask the hardware to enter it for that CPU as a result.
544 However, disabling an idle state for one CPU does not prevent it from being
547 governor is implemented, disabling an idle state prevents that governor from
548 selecting any idle states deeper than the disabled one too.]
550 If the :file:`disable` attribute contains 0, the given idle state is enabled for
551 this particular CPU, but it still may be disabled for some or all of the other
552 CPUs in the system at the same time. Writing 1 to it causes the idle state to
553 be disabled for this particular CPU and writing 0 to it allows the governor to
554 take it into consideration for the given CPU and the driver to ask for it,
558 The :file:`power` attribute is not defined very well, especially for idle state
559 objects representing combinations of idle states at different levels of the
560 hierarchy of units in the processor, and it generally is hard to obtain idle
566 really spent by the given CPU in the given idle state, because it is measured by
568 this idle state and entered a shallower one instead of it (or even it did not
569 enter any idle state at all). The kernel can only measure the time span between
570 asking the hardware to enter an idle state and the subsequent wakeup of the CPU
572 Moreover, if the idle state object in question represents a combination of idle
573 states at different levels of the hierarchy of units in the processor,
576 much time has been spent by the hardware in different idle states supported by
577 it is to use idle state residency counters in the hardware, if available.
579 Generally, an interrupt received when trying to enter an idle state causes the
580 idle state entry request to be rejected, in which case the ``CPUIdle`` driver
582 and :file:`rejected` files report the number of times the given idle state
585 .. _cpu-pm-qos:
592 energy-efficiency features of the kernel to prevent performance from dropping
595 CPU idle time management can be affected by PM QoS in two ways, through the
596 global CPU latency limit and through the resume latency constraints for
601 signed 32-bit integer) to it. In turn, the resume latency constraint for a CPU
603 32-bit integer) to the :file:`power/pm_qos_resume_latency_us` file under
604 :file:`/sys/devices/system/cpu/cpu<N>/` in ``sysfs``, where the CPU number
613 global CPU latency limit and for each individual CPU, aggregates them and
618 PM QoS request to be created and added to a global priority list of CPU latency
624 that effective value will be set as a new CPU latency limit. Thus requesting a
636 with that file descriptor to be removed from the global priority list of CPU
641 In turn, for each CPU there is one resume latency PM QoS request associated with
643 :file:`/sys/devices/system/cpu/cpu<N>/` in ``sysfs`` and writing to it causes
648 practice is to pin a process to the CPU in question and let it use the
652 CPU in question every time the list of requests is updated this way or another
655 CPU idle time governors are expected to regard the minimum of the global
656 (effective) CPU latency limit and the effective resume latency constraint for
657 the given CPU as the upper limit for the exit latency of the idle states that
658 they are allowed to select for that CPU. They should never select any idle
659 states with exit latency beyond that limit.
662 Idle States Control Via Kernel Command Line
665 In addition to the ``sysfs`` interface allowing individual idle states to be
666 `disabled for individual CPUs <idle-states-representation_>`_, there are kernel
667 command line parameters affecting CPU idle time management.
670 CPU idle time management entirely. It does not prevent the idle loop from
671 running on idle CPUs, but it prevents the CPU idle time governors and drivers
672 from being invoked. If it is added to the kernel command line, the idle loop
673 will ask the hardware to enter idle states on idle CPUs via the CPU architecture
676 processors implementing the architecture (i.e. CPU instruction set) in question,
677 however, so it is rather crude and not very energy-efficient. For this reason,
687 The other kernel command line parameters controlling CPU idle time management
692 options related to CPU idle time management: ``idle=poll``, ``idle=halt``,
693 and ``idle=nomwait``. The first two of them disable the ``acpi_idle`` and
695 ``CPUIdle`` subsystem to be disabled and makes the idle loop invoke the
696 architecture support code to deal with idle CPUs. How it does that depends on
698 ``idle=halt`` case, the architecture support code will use the ``HLT``
700 and causes the hardware to attempt to enter the shallowest available idle state)
701 for this purpose, and if ``idle=poll`` is used, idle CPUs will execute a
703 that using ``idle=poll`` is somewhat drastic in many cases, as preventing idle
706 P-states (see |cpufreq|) that require any number of CPUs in a package to be
707 idle, so it very well may hurt single-thread computations performance as well as
708 energy-efficiency. Thus using it for performance reasons may not be a good idea
711 The ``idle=nomwait`` option disables the ``intel_idle`` driver and causes
714 ``MWAIT`` instruction of the CPUs to ask the hardware to enter idle states.
716 In addition to the architecture-level kernel command line options affecting CPU
717 idle time management, there are parameters affecting individual ``CPUIdle``
720 where ``<n>`` is an idle state index also used in the name of the given
722 `Representation of Idle States <idle-states-representation_>`_), causes the
724 idle states deeper than idle state ``<n>``. In that case, they will never ask
725 for any of those idle states or expose them to the governor. [The behavior of