Lines Matching +full:in +full:- +full:memory
4 Memory Hot(Un)Plug
7 This document describes generic Linux support for memory hot(un)plug with
15 Memory hot(un)plug allows for increasing and decreasing the size of physical
16 memory available to a machine at runtime. In the simplest case, it consists of
20 Memory hot(un)plug is used for various purposes:
22 - The physical memory available to a machine can be adjusted at runtime, up- or
23 downgrading the memory capacity. This dynamic memory resizing, sometimes
27 - Replacing hardware, such as DIMMs or whole NUMA nodes, without downtime. One
28 example is replacing failing memory modules.
30 - Reducing energy consumption either by physically unplugging memory modules or
31 by logically unplugging (parts of) memory modules from Linux.
33 Further, the basic memory hot(un)plug infrastructure in Linux is nowadays also
34 used to expose persistent memory, other performance-differentiated memory and
35 reserved memory regions as ordinary system RAM to Linux.
37 Linux only supports memory hot(un)plug on selected 64 bit architectures, such as
40 Memory Hot(Un)Plug Granularity
41 ------------------------------
43 Memory hot(un)plug in Linux uses the SPARSEMEM memory model, which divides the
44 physical memory address space into chunks of the same size: memory sections. The
45 size of a memory section is architecture dependent. For example, x86_64 uses
48 Memory sections are combined into chunks referred to as "memory blocks". The
49 size of a memory block is architecture dependent and corresponds to the smallest
50 granularity that can be hot(un)plugged. The default size of a memory block is
51 the same as memory section size, unless an architecture specifies otherwise.
53 All memory blocks have the same size.
55 Phases of Memory Hotplug
56 ------------------------
58 Memory hotplug consists of two phases:
60 (1) Adding the memory to Linux
61 (2) Onlining memory blocks
63 In the first phase, metadata, such as the memory map ("memmap") and page tables
64 for the direct mapping, is allocated and initialized, and memory blocks are
65 created; the latter also creates sysfs files for managing newly created memory
68 In the second phase, added memory is exposed to the page allocator. After this
69 phase, the memory is visible in memory statistics, such as free and total
70 memory, of the system.
72 Phases of Memory Hotunplug
73 --------------------------
75 Memory hotunplug consists of two phases:
77 (1) Offlining memory blocks
78 (2) Removing the memory from Linux
80 In the fist phase, memory is "hidden" from the page allocator again, for
81 example, by migrating busy memory to other memory locations and removing all
82 relevant free pages from the page allocator After this phase, the memory is no
83 longer visible in memory statistics of the system.
85 In the second phase, the memory blocks are removed and metadata is freed.
87 Memory Hotplug Notifications
90 There are various ways how Linux is notified about memory hotplug events such
91 that it can start adding hotplugged memory. This description is limited to
96 ------------------
98 Platforms that support ACPI, such as x86_64, can support memory hotplug
101 In general, a firmware supporting memory hotplug defines a memory class object
102 HID "PNP0C80". When notified about hotplug of a new memory device, the ACPI
103 driver will hotplug the memory to Linux.
107 assigned memory devices are added to Linux by the ACPI driver.
109 Similarly, Linux can be notified about requests to hotunplug a memory device or
110 a NUMA node via ACPI. The ACPI driver will try offlining all relevant memory
111 blocks, and, if successful, hotunplug the memory from Linux.
114 --------------
117 system about a memory hotplug event. Instead, the memory has to be manually
122 /sys/devices/system/memory/probe
124 Only complete memory blocks can be probed. Individual memory blocks are probed
125 by providing the physical start address of the memory block::
127 % echo addr > /sys/devices/system/memory/probe
129 Which results in a memory block for the range [addr, addr + memory_block_size)
135 because Linux cannot validate user input; this interface might be removed in
138 Onlining and Offlining Memory Blocks
141 After a memory block has been created, Linux has to be instructed to actually
142 make use of that memory: the memory block has to be "online".
144 Before a memory block can be removed, Linux has to stop using any memory part of
145 the memory block: the memory block has to be "offlined".
147 The Linux kernel can be configured to automatically online added memory blocks
148 and drivers automatically trigger offlining of memory blocks when trying
149 hotunplug of memory. Memory blocks can only be removed once offlining succeeded
150 and drivers may trigger offlining of memory blocks when attempting hotunplug of
151 memory.
153 Onlining Memory Blocks Manually
154 -------------------------------
156 If auto-onlining of memory blocks isn't enabled, user-space has to manually
157 trigger onlining of memory blocks. Often, udev rules are used to automate this
158 task in user space.
160 Onlining of a memory block can be triggered via::
162 % echo online > /sys/devices/system/memory/memoryXXX/state
166 % echo 1 > /sys/devices/system/memory/memoryXXX/online
170 command line or if the memory block would intersect the ZONE_MOVABLE already.
172 One can explicitly request to associate an offline memory block with
175 % echo online_movable > /sys/devices/system/memory/memoryXXX/state
179 % echo online_kernel > /sys/devices/system/memory/memoryXXX/state
181 In any case, if onlining succeeds, the state of the memory block is changed to
182 be "online". If it fails, the state of the memory block will remain unchanged
185 Onlining Memory Blocks Automatically
186 ------------------------------------
188 The kernel can be configured to try auto-onlining of newly added memory blocks.
189 If this feature is disabled, the memory blocks will stay offline until
192 The configured auto-online behavior can be observed via::
194 % cat /sys/devices/system/memory/auto_online_blocks
196 Auto-onlining can be enabled by writing ``online``, ``online_kernel`` or
199 % echo online > /sys/devices/system/memory/auto_online_blocks
201 Modifying the auto-online behavior will only affect all subsequently added
202 memory blocks only.
206 In corner cases, auto-onlining can fail. The kernel won't retry. Note that
207 auto-onlining is not expected to fail in default configurations.
212 memory blocks; if onlining fails, memory blocks are removed again.
214 Offlining Memory Blocks
215 -----------------------
217 In the current implementation, Linux's memory offlining will try migrating all
218 movable pages off the affected memory block. As most kernel allocations, such as
220 memory offlining from succeeding.
222 Having the memory provided by memory block managed by ZONE_MOVABLE significantly
223 increases memory offlining reliability; still, memory offlining can fail in
226 Further, memory offlining might retry for a long time (or even forever), until
229 Offlining of a memory block can be triggered via::
231 % echo offline > /sys/devices/system/memory/memoryXXX/state
235 % echo 0 > /sys/devices/system/memory/memoryXXX/online
237 If offlining succeeds, the state of the memory block is changed to be "offline".
238 If it fails, the state of the memory block will remain unchanged and the above
247 Observing the State of Memory Blocks
248 ------------------------------------
250 The state (online/offline/going-offline) of a memory block can be observed
253 % cat /sys/device/system/memory/memoryXXX/state
257 % cat /sys/device/system/memory/memoryXXX/online
259 For an online memory block, the managing zone can be observed via::
261 % cat /sys/device/system/memory/memoryXXX/valid_zones
263 Configuring Memory Hot(Un)Plug
266 There are various ways how system administrators can configure memory
267 hot(un)plug and interact with memory blocks, especially, to online them.
269 Memory Hot(Un)Plug Configuration via Sysfs
270 ------------------------------------------
272 Some memory hot(un)plug properties can be configured or inspected via sysfs in::
274 /sys/devices/system/memory/
279 ``auto_online_blocks`` read-write: set or get the default state of new memory
280 blocks; configure auto-onlining.
286 See the ``state`` property of memory blocks for details.
287 ``block_size_bytes`` read-only: the size in bytes of a memory block.
288 ``probe`` write-only: add (probe) selected memory blocks manually
293 ``uevent`` read-write: generic udev file for device subsystems.
301 this functionality is not really related to memory hot(un)plug or actual
302 offlining of memory blocks.
304 Memory Block Configuration via Sysfs
305 ------------------------------------
307 Each memory block is represented as a memory block device that can be
308 onlined or offlined. All memory blocks have their device information located in
309 sysfs. Each present memory block is listed under
310 ``/sys/devices/system/memory`` as::
312 /sys/devices/system/memory/memoryXXX
314 where XXX is the memory block id; the number of digits is variable.
316 A present memory block indicates that some memory in the range is present;
317 however, a memory block might span memory holes. A memory block spanning memory
320 For example, assume 1 GiB memory block size. A device for a memory starting at
321 0x100000000 is ``/sys/device/system/memory/memory4``::
330 ``online`` read-write: simplified interface to trigger onlining /
331 offlining and to observe the state of a memory block.
333 ``phys_device`` read-only: legacy interface only ever used on s390x to
335 ``phys_index`` read-only: the memory block id (XXX).
336 ``removable`` read-only: legacy interface that indicated whether a memory
338 kernel return ``1`` if and only if it supports memory
340 ``state`` read-write: advanced interface to trigger onlining /
341 offlining and to observe the state of a memory block.
348 zone for the memory block, such as ZONE_NORMAL.
351 When reading, ``online``, ``offline`` and ``going-offline``
353 ``uevent`` read-write: generic uevent file for devices.
354 ``valid_zones`` read-only: when a block is online, shows the zone it
358 For online memory blocks, ``DMA``, ``DMA32``, ``Normal``,
360 that memory provided by a memory block is managed by
361 multiple zones or spans multiple nodes; such memory blocks
365 For offline memory blocks, the first column shows the
366 zone the kernel would select when onlining the memory block
376 directories can also be accessed via symbolic links located in the
381 /sys/devices/system/node/node0/memory9 -> ../../memory/memory9
385 /sys/devices/system/memory/memory9/node0 -> ../../node/node0
388 -----------------------
390 Some command line parameters affect memory hot(un)plug handling. The following
394 ``memhp_default_state`` configure auto-onlining by essentially setting
395 ``/sys/devices/system/memory/auto_online_blocks``.
402 ------------------
418 ``memmap_on_memory`` read-write: Allocate memory for the memmap from the
419 added memory block itself. Even if enabled, actual
424 While allocating the memmap from the memory block
425 itself makes memory hotplug less likely to fail and
426 keeps the memmap on the same NUMA node in any case, it
427 can fragment physical memory in a way that huge pages
428 in bigger granularity cannot be formed on hotplugged
429 memory.
435 ZONE_MOVABLE is an important mechanism for more reliable memory offlining.
440 Most kernel allocations are unmovable. Important examples include the memory
441 map (usually 1/64ths of memory), page tables, and kmalloc(). Such allocations
444 Most user space pages, such as anonymous memory, and page cache pages are
447 Only movable allocations are served from ZONE_MOVABLE, resulting in unmovable
449 absolutely no guarantee whether a memory block can be offlined successfully.
452 ---------------
456 might crash because it runs out of free memory for unmovable allocations,
457 although there is still plenty of free memory left in ZONE_MOVABLE.
460 are definitely impossible due to the overhead for the memory map.
463 long-term pinning of pages, might not be able to deal with ZONE_MOVABLE at all.
467 CMA memory part of a kernel zone essentially behaves like memory in
472 ----------------------------------
475 be consumed by user space, either directly or indirectly via the page cache. In
478 With that in mind, it makes sense that we can have a big portion of system RAM
480 ZONE_MOVABLE, especially when fine-tuning zone ratios:
482 - Having a lot of offline memory blocks. Even offline memory blocks consume
483 memory for metadata and page tables in the direct map; having a lot of offline
484 memory blocks is not a typical case, though.
486 - Memory ballooning without balloon compaction is incompatible with
487 ZONE_MOVABLE. Only some implementations, such as virtio-balloon and
491 disabled. In that case, balloon inflation will only perform unmovable
495 - Gigantic pages are unmovable, resulting in user space consuming a
496 lot of unmovable memory.
498 - Huge pages are unmovable when an architectures does not support huge
499 page migration, resulting in a similar issue as with gigantic pages.
501 - Page tables are unmovable. Excessive swapping, mapping extremely large
502 files or ZONE_DEVICE memory can be problematic, although only really relevant
503 in corner cases. When we manage a lot of user space memory that has been
504 swapped out or is served from a file/persistent memory/... we still need a lot
505 of page tables to manage that memory once user space accessed that memory.
507 - In certain DAX configurations the memory map for the device memory will be
510 - KASAN can have a significant memory overhead, for example, consuming 1/8th of
511 the total system memory size as (unmovable) tracking metadata.
513 - Long-term pinning of pages. Techniques that rely on long-term pinnings
515 ZONE_MOVABLE, and therefore, memory offlining. Pinned pages cannot reside
518 even if there is plenty of free memory in ZONE_MOVABLE.
520 In addition, using ZONE_MOVABLE might make page pinning more expensive,
523 By default, all the memory configured at boot time is managed by the kernel
526 To enable ZONE_MOVABLE to include the memory present at boot and to control the
529 Documentation/admin-guide/kernel-parameters.rst for their description.
531 Memory Offlining and ZONE_MOVABLE
532 ---------------------------------
534 Even with ZONE_MOVABLE, there are some corner cases where offlining a memory
537 - Memory blocks with memory holes; this applies to memory blocks present during
538 boot and can apply to memory blocks hotplugged via the XEN balloon and the
539 Hyper-V balloon.
541 - Mixed NUMA nodes and mixed zones within a single memory block prevent memory
542 offlining; this applies to memory blocks present during boot only.
544 - Special memory blocks prevented by the system from getting offlined. Examples
545 include any memory available during boot on arm64 or memory blocks spanning
546 the crashkernel area on s390x; this usually applies to memory blocks present
549 - Memory blocks overlapping with CMA areas cannot be offlined, this applies to
550 memory blocks present during boot only.
552 - Concurrent activity that operates on the same physical memory area, such as
553 allocating gigantic pages, can result in temporary offlining failures.
555 - Out of memory when dissolving huge pages, especially when freeing unused
560 for the vmemmap, because the system might not have free memory in the kernel
563 Users that depend on memory offlining to succeed for movable zones should
564 carefully consider whether the memory savings gained from this feature are
565 worth the risk of possibly not being able to offline memory in certain
568 Further, when running into out of memory situations while migrating pages, or
570 (-> BUG), memory offlining will keep retrying until it eventually succeeds.