1x86 Topology
2============
3
4This documents and clarifies the main aspects of x86 topology modelling and
5representation in the kernel. Update/change when doing changes to the
6respective code.
7
8The architecture-agnostic topology definitions are in
9Documentation/cputopology.txt. This file holds x86-specific
10differences/specialities which must not necessarily apply to the generic
11definitions. Thus, the way to read up on Linux topology on x86 is to start
12with the generic one and look at this one in parallel for the x86 specifics.
13
14Needless to say, code should use the generic functions - this file is *only*
15here to *document* the inner workings of x86 topology.
16
17Started by Thomas Gleixner <tglx@linutronix.de> and Borislav Petkov <bp@alien8.de>.
18
19The main aim of the topology facilities is to present adequate interfaces to
20code which needs to know/query/use the structure of the running system wrt
21threads, cores, packages, etc.
22
23The kernel does not care about the concept of physical sockets because a
24socket has no relevance to software. It's an electromechanical component. In
25the past a socket always contained a single package (see below), but with the
26advent of Multi Chip Modules (MCM) a socket can hold more than one package. So
27there might be still references to sockets in the code, but they are of
28historical nature and should be cleaned up.
29
30The topology of a system is described in the units of:
31
32    - packages
33    - cores
34    - threads
35
36* Package:
37
38  Packages contain a number of cores plus shared resources, e.g. DRAM
39  controller, shared caches etc.
40
41  AMD nomenclature for package is 'Node'.
42
43  Package-related topology information in the kernel:
44
45  - cpuinfo_x86.x86_max_cores:
46
47    The number of cores in a package. This information is retrieved via CPUID.
48
49  - cpuinfo_x86.phys_proc_id:
50
51    The physical ID of the package. This information is retrieved via CPUID
52    and deduced from the APIC IDs of the cores in the package.
53
54  - cpuinfo_x86.logical_id:
55
56    The logical ID of the package. As we do not trust BIOSes to enumerate the
57    packages in a consistent way, we introduced the concept of logical package
58    ID so we can sanely calculate the number of maximum possible packages in
59    the system and have the packages enumerated linearly.
60
61  - topology_max_packages():
62
63    The maximum possible number of packages in the system. Helpful for per
64    package facilities to preallocate per package information.
65
66  - cpu_llc_id:
67
68    A per-CPU variable containing:
69    - On Intel, the first APIC ID of the list of CPUs sharing the Last Level
70    Cache
71
72    - On AMD, the Node ID or Core Complex ID containing the Last Level
73    Cache. In general, it is a number identifying an LLC uniquely on the
74    system.
75
76* Cores:
77
78  A core consists of 1 or more threads. It does not matter whether the threads
79  are SMT- or CMT-type threads.
80
81  AMDs nomenclature for a CMT core is "Compute Unit". The kernel always uses
82  "core".
83
84  Core-related topology information in the kernel:
85
86  - smp_num_siblings:
87
88    The number of threads in a core. The number of threads in a package can be
89    calculated by:
90
91	threads_per_package = cpuinfo_x86.x86_max_cores * smp_num_siblings
92
93
94* Threads:
95
96  A thread is a single scheduling unit. It's the equivalent to a logical Linux
97  CPU.
98
99  AMDs nomenclature for CMT threads is "Compute Unit Core". The kernel always
100  uses "thread".
101
102  Thread-related topology information in the kernel:
103
104  - topology_core_cpumask():
105
106    The cpumask contains all online threads in the package to which a thread
107    belongs.
108
109    The number of online threads is also printed in /proc/cpuinfo "siblings."
110
111  - topology_sibling_cpumask():
112
113    The cpumask contains all online threads in the core to which a thread
114    belongs.
115
116   - topology_logical_package_id():
117
118    The logical package ID to which a thread belongs.
119
120   - topology_physical_package_id():
121
122    The physical package ID to which a thread belongs.
123
124   - topology_core_id();
125
126    The ID of the core to which a thread belongs. It is also printed in /proc/cpuinfo
127    "core_id."
128
129
130
131System topology examples
132
133Note:
134
135The alternative Linux CPU enumeration depends on how the BIOS enumerates the
136threads. Many BIOSes enumerate all threads 0 first and then all threads 1.
137That has the "advantage" that the logical Linux CPU numbers of threads 0 stay
138the same whether threads are enabled or not. That's merely an implementation
139detail and has no practical impact.
140
1411) Single Package, Single Core
142
143   [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
144
1452) Single Package, Dual Core
146
147   a) One thread per core
148
149	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
150		    -> [core 1] -> [thread 0] -> Linux CPU 1
151
152   b) Two threads per core
153
154	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
155				-> [thread 1] -> Linux CPU 1
156		    -> [core 1] -> [thread 0] -> Linux CPU 2
157				-> [thread 1] -> Linux CPU 3
158
159      Alternative enumeration:
160
161	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
162				-> [thread 1] -> Linux CPU 2
163		    -> [core 1] -> [thread 0] -> Linux CPU 1
164				-> [thread 1] -> Linux CPU 3
165
166      AMD nomenclature for CMT systems:
167
168	[node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0
169				     -> [Compute Unit Core 1] -> Linux CPU 1
170		 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2
171				     -> [Compute Unit Core 1] -> Linux CPU 3
172
1734) Dual Package, Dual Core
174
175   a) One thread per core
176
177	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
178		    -> [core 1] -> [thread 0] -> Linux CPU 1
179
180	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 2
181		    -> [core 1] -> [thread 0] -> Linux CPU 3
182
183   b) Two threads per core
184
185	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
186				-> [thread 1] -> Linux CPU 1
187		    -> [core 1] -> [thread 0] -> Linux CPU 2
188				-> [thread 1] -> Linux CPU 3
189
190	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 4
191				-> [thread 1] -> Linux CPU 5
192		    -> [core 1] -> [thread 0] -> Linux CPU 6
193				-> [thread 1] -> Linux CPU 7
194
195      Alternative enumeration:
196
197	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
198				-> [thread 1] -> Linux CPU 4
199		    -> [core 1] -> [thread 0] -> Linux CPU 1
200				-> [thread 1] -> Linux CPU 5
201
202	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 2
203				-> [thread 1] -> Linux CPU 6
204		    -> [core 1] -> [thread 0] -> Linux CPU 3
205				-> [thread 1] -> Linux CPU 7
206
207      AMD nomenclature for CMT systems:
208
209	[node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0
210				     -> [Compute Unit Core 1] -> Linux CPU 1
211		 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2
212				     -> [Compute Unit Core 1] -> Linux CPU 3
213
214	[node 1] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 4
215				     -> [Compute Unit Core 1] -> Linux CPU 5
216		 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 6
217				     -> [Compute Unit Core 1] -> Linux CPU 7
218