1.. _admin_guide_memory_hotplug: 2 3================== 4Memory Hot(Un)Plug 5================== 6 7This document describes generic Linux support for memory hot(un)plug with 8a focus on System RAM, including ZONE_MOVABLE support. 9 10.. contents:: :local: 11 12Introduction 13============ 14 15Memory hot(un)plug allows for increasing and decreasing the size of physical 16memory available to a machine at runtime. In the simplest case, it consists of 17physically plugging or unplugging a DIMM at runtime, coordinated with the 18operating system. 19 20Memory hot(un)plug is used for various purposes: 21 22- The physical memory available to a machine can be adjusted at runtime, up- or 23 downgrading the memory capacity. This dynamic memory resizing, sometimes 24 referred to as "capacity on demand", is frequently used with virtual machines 25 and logical partitions. 26 27- Replacing hardware, such as DIMMs or whole NUMA nodes, without downtime. One 28 example is replacing failing memory modules. 29 30- Reducing energy consumption either by physically unplugging memory modules or 31 by logically unplugging (parts of) memory modules from Linux. 32 33Further, the basic memory hot(un)plug infrastructure in Linux is nowadays also 34used to expose persistent memory, other performance-differentiated memory and 35reserved memory regions as ordinary system RAM to Linux. 36 37Linux only supports memory hot(un)plug on selected 64 bit architectures, such as 38x86_64, arm64, ppc64, s390x and ia64. 39 40Memory Hot(Un)Plug Granularity 41------------------------------ 42 43Memory hot(un)plug in Linux uses the SPARSEMEM memory model, which divides the 44physical memory address space into chunks of the same size: memory sections. The 45size of a memory section is architecture dependent. For example, x86_64 uses 46128 MiB and ppc64 uses 16 MiB. 47 48Memory sections are combined into chunks referred to as "memory blocks". The 49size of a memory block is architecture dependent and corresponds to the smallest 50granularity that can be hot(un)plugged. The default size of a memory block is 51the same as memory section size, unless an architecture specifies otherwise. 52 53All memory blocks have the same size. 54 55Phases of Memory Hotplug 56------------------------ 57 58Memory hotplug consists of two phases: 59 60(1) Adding the memory to Linux 61(2) Onlining memory blocks 62 63In the first phase, metadata, such as the memory map ("memmap") and page tables 64for the direct mapping, is allocated and initialized, and memory blocks are 65created; the latter also creates sysfs files for managing newly created memory 66blocks. 67 68In the second phase, added memory is exposed to the page allocator. After this 69phase, the memory is visible in memory statistics, such as free and total 70memory, of the system. 71 72Phases of Memory Hotunplug 73-------------------------- 74 75Memory hotunplug consists of two phases: 76 77(1) Offlining memory blocks 78(2) Removing the memory from Linux 79 80In the fist phase, memory is "hidden" from the page allocator again, for 81example, by migrating busy memory to other memory locations and removing all 82relevant free pages from the page allocator After this phase, the memory is no 83longer visible in memory statistics of the system. 84 85In the second phase, the memory blocks are removed and metadata is freed. 86 87Memory Hotplug Notifications 88============================ 89 90There are various ways how Linux is notified about memory hotplug events such 91that it can start adding hotplugged memory. This description is limited to 92systems that support ACPI; mechanisms specific to other firmware interfaces or 93virtual machines are not described. 94 95ACPI Notifications 96------------------ 97 98Platforms that support ACPI, such as x86_64, can support memory hotplug 99notifications via ACPI. 100 101In general, a firmware supporting memory hotplug defines a memory class object 102HID "PNP0C80". When notified about hotplug of a new memory device, the ACPI 103driver will hotplug the memory to Linux. 104 105If the firmware supports hotplug of NUMA nodes, it defines an object _HID 106"ACPI0004", "PNP0A05", or "PNP0A06". When notified about an hotplug event, all 107assigned memory devices are added to Linux by the ACPI driver. 108 109Similarly, Linux can be notified about requests to hotunplug a memory device or 110a NUMA node via ACPI. The ACPI driver will try offlining all relevant memory 111blocks, and, if successful, hotunplug the memory from Linux. 112 113Manual Probing 114-------------- 115 116On some architectures, the firmware may not be able to notify the operating 117system about a memory hotplug event. Instead, the memory has to be manually 118probed from user space. 119 120The probe interface is located at:: 121 122 /sys/devices/system/memory/probe 123 124Only complete memory blocks can be probed. Individual memory blocks are probed 125by providing the physical start address of the memory block:: 126 127 % echo addr > /sys/devices/system/memory/probe 128 129Which results in a memory block for the range [addr, addr + memory_block_size) 130being created. 131 132.. note:: 133 134 Using the probe interface is discouraged as it is easy to crash the kernel, 135 because Linux cannot validate user input; this interface might be removed in 136 the future. 137 138Onlining and Offlining Memory Blocks 139==================================== 140 141After a memory block has been created, Linux has to be instructed to actually 142make use of that memory: the memory block has to be "online". 143 144Before a memory block can be removed, Linux has to stop using any memory part of 145the memory block: the memory block has to be "offlined". 146 147The Linux kernel can be configured to automatically online added memory blocks 148and drivers automatically trigger offlining of memory blocks when trying 149hotunplug of memory. Memory blocks can only be removed once offlining succeeded 150and drivers may trigger offlining of memory blocks when attempting hotunplug of 151memory. 152 153Onlining Memory Blocks Manually 154------------------------------- 155 156If auto-onlining of memory blocks isn't enabled, user-space has to manually 157trigger onlining of memory blocks. Often, udev rules are used to automate this 158task in user space. 159 160Onlining of a memory block can be triggered via:: 161 162 % echo online > /sys/devices/system/memory/memoryXXX/state 163 164Or alternatively:: 165 166 % echo 1 > /sys/devices/system/memory/memoryXXX/online 167 168The kernel will select the target zone automatically, usually defaulting to 169``ZONE_NORMAL`` unless ``movablecore=1`` has been specified on the kernel 170command line or if the memory block would intersect the ZONE_MOVABLE already. 171 172One can explicitly request to associate an offline memory block with 173ZONE_MOVABLE by:: 174 175 % echo online_movable > /sys/devices/system/memory/memoryXXX/state 176 177Or one can explicitly request a kernel zone (usually ZONE_NORMAL) by:: 178 179 % echo online_kernel > /sys/devices/system/memory/memoryXXX/state 180 181In any case, if onlining succeeds, the state of the memory block is changed to 182be "online". If it fails, the state of the memory block will remain unchanged 183and the above commands will fail. 184 185Onlining Memory Blocks Automatically 186------------------------------------ 187 188The kernel can be configured to try auto-onlining of newly added memory blocks. 189If this feature is disabled, the memory blocks will stay offline until 190explicitly onlined from user space. 191 192The configured auto-online behavior can be observed via:: 193 194 % cat /sys/devices/system/memory/auto_online_blocks 195 196Auto-onlining can be enabled by writing ``online``, ``online_kernel`` or 197``online_movable`` to that file, like:: 198 199 % echo online > /sys/devices/system/memory/auto_online_blocks 200 201Modifying the auto-online behavior will only affect all subsequently added 202memory blocks only. 203 204.. note:: 205 206 In corner cases, auto-onlining can fail. The kernel won't retry. Note that 207 auto-onlining is not expected to fail in default configurations. 208 209.. note:: 210 211 DLPAR on ppc64 ignores the ``offline`` setting and will still online added 212 memory blocks; if onlining fails, memory blocks are removed again. 213 214Offlining Memory Blocks 215----------------------- 216 217In the current implementation, Linux's memory offlining will try migrating all 218movable pages off the affected memory block. As most kernel allocations, such as 219page tables, are unmovable, page migration can fail and, therefore, inhibit 220memory offlining from succeeding. 221 222Having the memory provided by memory block managed by ZONE_MOVABLE significantly 223increases memory offlining reliability; still, memory offlining can fail in 224some corner cases. 225 226Further, memory offlining might retry for a long time (or even forever), until 227aborted by the user. 228 229Offlining of a memory block can be triggered via:: 230 231 % echo offline > /sys/devices/system/memory/memoryXXX/state 232 233Or alternatively:: 234 235 % echo 0 > /sys/devices/system/memory/memoryXXX/online 236 237If offlining succeeds, the state of the memory block is changed to be "offline". 238If it fails, the state of the memory block will remain unchanged and the above 239commands will fail, for example, via:: 240 241 bash: echo: write error: Device or resource busy 242 243or via:: 244 245 bash: echo: write error: Invalid argument 246 247Observing the State of Memory Blocks 248------------------------------------ 249 250The state (online/offline/going-offline) of a memory block can be observed 251either via:: 252 253 % cat /sys/device/system/memory/memoryXXX/state 254 255Or alternatively (1/0) via:: 256 257 % cat /sys/device/system/memory/memoryXXX/online 258 259For an online memory block, the managing zone can be observed via:: 260 261 % cat /sys/device/system/memory/memoryXXX/valid_zones 262 263Configuring Memory Hot(Un)Plug 264============================== 265 266There are various ways how system administrators can configure memory 267hot(un)plug and interact with memory blocks, especially, to online them. 268 269Memory Hot(Un)Plug Configuration via Sysfs 270------------------------------------------ 271 272Some memory hot(un)plug properties can be configured or inspected via sysfs in:: 273 274 /sys/devices/system/memory/ 275 276The following files are currently defined: 277 278====================== ========================================================= 279``auto_online_blocks`` read-write: set or get the default state of new memory 280 blocks; configure auto-onlining. 281 282 The default value depends on the 283 CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE kernel configuration 284 option. 285 286 See the ``state`` property of memory blocks for details. 287``block_size_bytes`` read-only: the size in bytes of a memory block. 288``probe`` write-only: add (probe) selected memory blocks manually 289 from user space by supplying the physical start address. 290 291 Availability depends on the CONFIG_ARCH_MEMORY_PROBE 292 kernel configuration option. 293``uevent`` read-write: generic udev file for device subsystems. 294====================== ========================================================= 295 296.. note:: 297 298 When the CONFIG_MEMORY_FAILURE kernel configuration option is enabled, two 299 additional files ``hard_offline_page`` and ``soft_offline_page`` are available 300 to trigger hwpoisoning of pages, for example, for testing purposes. Note that 301 this functionality is not really related to memory hot(un)plug or actual 302 offlining of memory blocks. 303 304Memory Block Configuration via Sysfs 305------------------------------------ 306 307Each memory block is represented as a memory block device that can be 308onlined or offlined. All memory blocks have their device information located in 309sysfs. Each present memory block is listed under 310``/sys/devices/system/memory`` as:: 311 312 /sys/devices/system/memory/memoryXXX 313 314where XXX is the memory block id; the number of digits is variable. 315 316A present memory block indicates that some memory in the range is present; 317however, a memory block might span memory holes. A memory block spanning memory 318holes cannot be offlined. 319 320For example, assume 1 GiB memory block size. A device for a memory starting at 3210x100000000 is ``/sys/device/system/memory/memory4``:: 322 323 (0x100000000 / 1Gib = 4) 324 325This device covers address range [0x100000000 ... 0x140000000) 326 327The following files are currently defined: 328 329=================== ============================================================ 330``online`` read-write: simplified interface to trigger onlining / 331 offlining and to observe the state of a memory block. 332 When onlining, the zone is selected automatically. 333``phys_device`` read-only: legacy interface only ever used on s390x to 334 expose the covered storage increment. 335``phys_index`` read-only: the memory block id (XXX). 336``removable`` read-only: legacy interface that indicated whether a memory 337 block was likely to be offlineable or not. Nowadays, the 338 kernel return ``1`` if and only if it supports memory 339 offlining. 340``state`` read-write: advanced interface to trigger onlining / 341 offlining and to observe the state of a memory block. 342 343 When writing, ``online``, ``offline``, ``online_kernel`` and 344 ``online_movable`` are supported. 345 346 ``online_movable`` specifies onlining to ZONE_MOVABLE. 347 ``online_kernel`` specifies onlining to the default kernel 348 zone for the memory block, such as ZONE_NORMAL. 349 ``online`` let's the kernel select the zone automatically. 350 351 When reading, ``online``, ``offline`` and ``going-offline`` 352 may be returned. 353``uevent`` read-write: generic uevent file for devices. 354``valid_zones`` read-only: when a block is online, shows the zone it 355 belongs to; when a block is offline, shows what zone will 356 manage it when the block will be onlined. 357 358 For online memory blocks, ``DMA``, ``DMA32``, ``Normal``, 359 ``Movable`` and ``none`` may be returned. ``none`` indicates 360 that memory provided by a memory block is managed by 361 multiple zones or spans multiple nodes; such memory blocks 362 cannot be offlined. ``Movable`` indicates ZONE_MOVABLE. 363 Other values indicate a kernel zone. 364 365 For offline memory blocks, the first column shows the 366 zone the kernel would select when onlining the memory block 367 right now without further specifying a zone. 368 369 Availability depends on the CONFIG_MEMORY_HOTREMOVE 370 kernel configuration option. 371=================== ============================================================ 372 373.. note:: 374 375 If the CONFIG_NUMA kernel configuration option is enabled, the memoryXXX/ 376 directories can also be accessed via symbolic links located in the 377 ``/sys/devices/system/node/node*`` directories. 378 379 For example:: 380 381 /sys/devices/system/node/node0/memory9 -> ../../memory/memory9 382 383 A backlink will also be created:: 384 385 /sys/devices/system/memory/memory9/node0 -> ../../node/node0 386 387Command Line Parameters 388----------------------- 389 390Some command line parameters affect memory hot(un)plug handling. The following 391command line parameters are relevant: 392 393======================== ======================================================= 394``memhp_default_state`` configure auto-onlining by essentially setting 395 ``/sys/devices/system/memory/auto_online_blocks``. 396``movablecore`` configure automatic zone selection of the kernel. When 397 set, the kernel will default to ZONE_MOVABLE, unless 398 other zones can be kept contiguous. 399======================== ======================================================= 400 401Module Parameters 402------------------ 403 404Instead of additional command line parameters or sysfs files, the 405``memory_hotplug`` subsystem now provides a dedicated namespace for module 406parameters. Module parameters can be set via the command line by predicating 407them with ``memory_hotplug.`` such as:: 408 409 memory_hotplug.memmap_on_memory=1 410 411and they can be observed (and some even modified at runtime) via:: 412 413 /sys/modules/memory_hotplug/parameters/ 414 415The following module parameters are currently defined: 416 417======================== ======================================================= 418``memmap_on_memory`` read-write: Allocate memory for the memmap from the 419 added memory block itself. Even if enabled, actual 420 support depends on various other system properties and 421 should only be regarded as a hint whether the behavior 422 would be desired. 423 424 While allocating the memmap from the memory block 425 itself makes memory hotplug less likely to fail and 426 keeps the memmap on the same NUMA node in any case, it 427 can fragment physical memory in a way that huge pages 428 in bigger granularity cannot be formed on hotplugged 429 memory. 430======================== ======================================================= 431 432ZONE_MOVABLE 433============ 434 435ZONE_MOVABLE is an important mechanism for more reliable memory offlining. 436Further, having system RAM managed by ZONE_MOVABLE instead of one of the 437kernel zones can increase the number of possible transparent huge pages and 438dynamically allocated huge pages. 439 440Most kernel allocations are unmovable. Important examples include the memory 441map (usually 1/64ths of memory), page tables, and kmalloc(). Such allocations 442can only be served from the kernel zones. 443 444Most user space pages, such as anonymous memory, and page cache pages are 445movable. Such allocations can be served from ZONE_MOVABLE and the kernel zones. 446 447Only movable allocations are served from ZONE_MOVABLE, resulting in unmovable 448allocations being limited to the kernel zones. Without ZONE_MOVABLE, there is 449absolutely no guarantee whether a memory block can be offlined successfully. 450 451Zone Imbalances 452--------------- 453 454Having too much system RAM managed by ZONE_MOVABLE is called a zone imbalance, 455which can harm the system or degrade performance. As one example, the kernel 456might crash because it runs out of free memory for unmovable allocations, 457although there is still plenty of free memory left in ZONE_MOVABLE. 458 459Usually, MOVABLE:KERNEL ratios of up to 3:1 or even 4:1 are fine. Ratios of 63:1 460are definitely impossible due to the overhead for the memory map. 461 462Actual safe zone ratios depend on the workload. Extreme cases, like excessive 463long-term pinning of pages, might not be able to deal with ZONE_MOVABLE at all. 464 465.. note:: 466 467 CMA memory part of a kernel zone essentially behaves like memory in 468 ZONE_MOVABLE and similar considerations apply, especially when combining 469 CMA with ZONE_MOVABLE. 470 471ZONE_MOVABLE Sizing Considerations 472---------------------------------- 473 474We usually expect that a large portion of available system RAM will actually 475be consumed by user space, either directly or indirectly via the page cache. In 476the normal case, ZONE_MOVABLE can be used when allocating such pages just fine. 477 478With that in mind, it makes sense that we can have a big portion of system RAM 479managed by ZONE_MOVABLE. However, there are some things to consider when using 480ZONE_MOVABLE, especially when fine-tuning zone ratios: 481 482- Having a lot of offline memory blocks. Even offline memory blocks consume 483 memory for metadata and page tables in the direct map; having a lot of offline 484 memory blocks is not a typical case, though. 485 486- Memory ballooning without balloon compaction is incompatible with 487 ZONE_MOVABLE. Only some implementations, such as virtio-balloon and 488 pseries CMM, fully support balloon compaction. 489 490 Further, the CONFIG_BALLOON_COMPACTION kernel configuration option might be 491 disabled. In that case, balloon inflation will only perform unmovable 492 allocations and silently create a zone imbalance, usually triggered by 493 inflation requests from the hypervisor. 494 495- Gigantic pages are unmovable, resulting in user space consuming a 496 lot of unmovable memory. 497 498- Huge pages are unmovable when an architectures does not support huge 499 page migration, resulting in a similar issue as with gigantic pages. 500 501- Page tables are unmovable. Excessive swapping, mapping extremely large 502 files or ZONE_DEVICE memory can be problematic, although only really relevant 503 in corner cases. When we manage a lot of user space memory that has been 504 swapped out or is served from a file/persistent memory/... we still need a lot 505 of page tables to manage that memory once user space accessed that memory. 506 507- In certain DAX configurations the memory map for the device memory will be 508 allocated from the kernel zones. 509 510- KASAN can have a significant memory overhead, for example, consuming 1/8th of 511 the total system memory size as (unmovable) tracking metadata. 512 513- Long-term pinning of pages. Techniques that rely on long-term pinnings 514 (especially, RDMA and vfio/mdev) are fundamentally problematic with 515 ZONE_MOVABLE, and therefore, memory offlining. Pinned pages cannot reside 516 on ZONE_MOVABLE as that would turn these pages unmovable. Therefore, they 517 have to be migrated off that zone while pinning. Pinning a page can fail 518 even if there is plenty of free memory in ZONE_MOVABLE. 519 520 In addition, using ZONE_MOVABLE might make page pinning more expensive, 521 because of the page migration overhead. 522 523By default, all the memory configured at boot time is managed by the kernel 524zones and ZONE_MOVABLE is not used. 525 526To enable ZONE_MOVABLE to include the memory present at boot and to control the 527ratio between movable and kernel zones there are two command line options: 528``kernelcore=`` and ``movablecore=``. See 529Documentation/admin-guide/kernel-parameters.rst for their description. 530 531Memory Offlining and ZONE_MOVABLE 532--------------------------------- 533 534Even with ZONE_MOVABLE, there are some corner cases where offlining a memory 535block might fail: 536 537- Memory blocks with memory holes; this applies to memory blocks present during 538 boot and can apply to memory blocks hotplugged via the XEN balloon and the 539 Hyper-V balloon. 540 541- Mixed NUMA nodes and mixed zones within a single memory block prevent memory 542 offlining; this applies to memory blocks present during boot only. 543 544- Special memory blocks prevented by the system from getting offlined. Examples 545 include any memory available during boot on arm64 or memory blocks spanning 546 the crashkernel area on s390x; this usually applies to memory blocks present 547 during boot only. 548 549- Memory blocks overlapping with CMA areas cannot be offlined, this applies to 550 memory blocks present during boot only. 551 552- Concurrent activity that operates on the same physical memory area, such as 553 allocating gigantic pages, can result in temporary offlining failures. 554 555- Out of memory when dissolving huge pages, especially when freeing unused 556 vmemmap pages associated with each hugetlb page is enabled. 557 558 Offlining code may be able to migrate huge page contents, but may not be able 559 to dissolve the source huge page because it fails allocating (unmovable) pages 560 for the vmemmap, because the system might not have free memory in the kernel 561 zones left. 562 563 Users that depend on memory offlining to succeed for movable zones should 564 carefully consider whether the memory savings gained from this feature are 565 worth the risk of possibly not being able to offline memory in certain 566 situations. 567 568Further, when running into out of memory situations while migrating pages, or 569when still encountering permanently unmovable pages within ZONE_MOVABLE 570(-> BUG), memory offlining will keep retrying until it eventually succeeds. 571 572When offlining is triggered from user space, the offlining context can be 573terminated by sending a fatal signal. A timeout based offlining can easily be 574implemented via:: 575 576 % timeout $TIMEOUT offline_block | failure_handling 577
