21  * Allocation is done by offsets into a unit's address space.  Ie., an
29  * the static percpu variables in the kernel image as allocation services
40  * memcg-awareness.  To make a percpu allocation memcg-aware the __GFP_ACCOUNT
46  * is managed by a bitmap with metadata blocks.  The allocation map is updated
47  * on every allocation and free to reflect the current state while the boundary
48  * map is only updated on allocation.  Each metadata block contains
153 /* group information, used for vm allocation */
314  * @bits: size of allocation
317  * Check to see if the allocation can fit in the block's contig hint.
333  * @alloc_bits: size of allocation
338  * cannot fulfill an allocation, we can begin scanning from there knowing
347 	 * contig_hint == scan_hint).  Third, is the allocation request  in pcpu_next_hint()
411  * pcpu_next_fit_region - finds fit areas for a given allocation request
413  * @alloc_bits: size of allocation
420  * allocation.  block->first_free is returned if the allocation request fits
479  * a fit is found for the allocation request.
498  * @gfp: allocation flags
551  * This function is called after an allocation or free changed @chunk.
706  * Finding the final allocation spot first goes through pcpu_find_block_fit()
707  * to find a block that can hold the allocation and then pcpu_alloc_area()
743  * A full scan can be avoided on the allocation path as this is triggered
803  * pcpu_block_update_hint_alloc - update hint on allocation path
808  * Updates metadata for the allocation path.  The metadata only has to be
818 	int s_index, e_index;	/* block indexes of the freed allocation */  in pcpu_block_update_hint_alloc()
819 	int s_off, e_off;	/* block offsets of the freed allocation */  in pcpu_block_update_hint_alloc()
837 	 * block->first_free must be updated if the allocation takes its place.  in pcpu_block_update_hint_alloc()
838 	 * If the allocation breaks the contig_hint, a scan is required to  in pcpu_block_update_hint_alloc()
883 		 * When the allocation is across blocks, the end is along  in pcpu_block_update_hint_alloc()
947  * Updates metadata for the allocation path.  This avoids a blind block
964 	int s_index, e_index;	/* block indexes of the freed allocation */  in pcpu_block_update_hint_free()
965 	int s_off, e_off;	/* block offsets of the freed allocation */  in pcpu_block_update_hint_free()
1090  * @alloc_bits: size of request in allocation units
1094  * Given a chunk and an allocation spec, find the offset to begin searching
1097  * not quite first fit as if the allocation does not fit in the contig hint
1114 	 * allocation cannot fit in the global hint, there is memory pressure  in pcpu_find_block_fit()
1169 	/* Align allocation */  in pcpu_find_zero_area()
1196  * @alloc_bits: size of request in allocation units
1201  * allocation of @alloc_bits with alignment @align.  It needs to scan
1202  * the allocation map because if it fits within the block's contig hint,
1204  * allocation prior to breaking the contig hint.  The allocation and
1266  * This function determines the size of an allocation to free using
1267  * the boundary bitmap and clears the allocation map.
1712  * @gfp: allocation flags
1715  * contain %GFP_KERNEL, the allocation is atomic. If @gfp has __GFP_NOWARN
1716  * then no warning will be triggered on invalid or failed allocation
1744 	 * There is now a minimum allocation size of PCPU_MIN_ALLOC_SIZE,  in pcpu_alloc()
1746 	 * An allocation may have internal fragmentation from rounding up  in pcpu_alloc()
1758 		WARN(do_warn, "illegal size (%zu) or align (%zu) for percpu allocation\n",  in pcpu_alloc()
1901 		pr_warn("allocation failed, size=%zu align=%zu atomic=%d, %s\n",  in pcpu_alloc()
1924  * @gfp: allocation flags
1927  * @gfp doesn't contain %GFP_KERNEL, the allocation doesn't block and can
1930  * allocation requests.
1961  * allocation is served from the same dynamic area.  Might sleep.
1965  * Does GFP_KERNEL allocation.
2034  * allocation causes the failure as it is possible that requests can be
2052 	 * allocs don't increase fragmentation.  If atomic allocation  in pcpu_balance_populated()
2422  * pcpu_alloc_alloc_info - allocate percpu allocation info
2466  * pcpu_free_alloc_info - free percpu allocation info
2479  * @ai: allocation info to dump
2550  * percpu allocation.  This is primarily used to serve module percpu
2556  * allocation in the first chunk.  The area between @ai->static_size +
2563  * @ai->atom_size is the allocation atom size and used as alignment
2566  * @ai->alloc_size is the allocation size and always multiple of
2583  * share the same vm, but use offset regions in the area allocation map.
2585  * and available for dynamic allocation like any other chunk.
2736 	 * minimum allocation size as this offsets the reserved and  in pcpu_setup_first_chunk()
2834  * @dyn_size: minimum free size for dynamic allocation in bytes
2835  * @atom_size: allocation atom size
2839  * and other parameters considering needed percpu size, allocation
2887 	/* determine the maximum # of units that can fit in an allocation */  in pcpu_build_alloc_info()
2979 		 * reflect actual allocation.  in pcpu_build_alloc_info()
3035  * @dyn_size: minimum free size for dynamic allocation in bytes
3036  * @atom_size: allocation atom size