1======================================================
2Intel Many Integrated Core (MIC) architecture overview
3======================================================
4
5An Intel MIC X100 device is a PCIe form factor add-in coprocessor
6card based on the Intel Many Integrated Core (MIC) architecture
7that runs a Linux OS. It is a PCIe endpoint in a platform and therefore
8implements the three required standard address spaces i.e. configuration,
9memory and I/O. The host OS loads a device driver as is typical for
10PCIe devices. The card itself runs a bootstrap after reset that
11transfers control to the card OS downloaded from the host driver. The
12host driver supports OSPM suspend and resume operations. It shuts down
13the card during suspend and reboots the card OS during resume.
14The card OS as shipped by Intel is a Linux kernel with modifications
15for the X100 devices.
16
17Since it is a PCIe card, it does not have the ability to host hardware
18devices for networking, storage and console. We provide these devices
19on X100 coprocessors thus enabling a self-bootable equivalent
20environment for applications. A key benefit of our solution is that it
21leverages the standard virtio framework for network, disk and console
22devices, though in our case the virtio framework is used across a PCIe
23bus. A Virtio Over PCIe (VOP) driver allows creating user space
24backends or devices on the host which are used to probe virtio drivers
25for these devices on the MIC card. The existing VRINGH infrastructure
26in the kernel is used to access virtio rings from the host. The card
27VOP driver allows card virtio drivers to communicate with their user
28space backends on the host via a device page. Ring 3 apps on the host
29can add, remove and configure virtio devices. A thin MIC specific
30virtio_config_ops is implemented which is borrowed heavily from
31previous similar implementations in lguest and s390.
32
33MIC PCIe card has a dma controller with 8 channels. These channels are
34shared between the host s/w and the card s/w. 0 to 3 are used by host
35and 4 to 7 by card. As the dma device doesn't show up as PCIe device,
36a virtual bus called mic bus is created and virtual dma devices are
37created on it by the host/card drivers. On host the channels are private
38and used only by the host driver to transfer data for the virtio devices.
39
40The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a
41low level communications API across PCIe currently implemented for MIC.
42More details are available at scif_overview.txt.
43
44The Coprocessor State Management (COSM) driver on the host allows for
45boot, shutdown and reset of Intel MIC devices. It communicates with a COSM
46"client" driver on the MIC cards over SCIF to perform these functions.
47
48Here is a block diagram of the various components described above. The
49virtio backends are situated on the host rather than the card given better
50single threaded performance for the host compared to MIC, the ability of
51the host to initiate DMA's to/from the card using the MIC DMA engine and
52the fact that the virtio block storage backend can only be on the host::
53
54               +----------+           |             +----------+
55               | Card OS  |           |             | Host OS  |
56               +----------+           |             +----------+
57                                      |
58        +-------+ +--------+ +------+ | +---------+  +--------+ +--------+
59        | Virtio| |Virtio  | |Virtio| | |Virtio   |  |Virtio  | |Virtio  |
60        | Net   | |Console | |Block | | |Net      |  |Console | |Block   |
61        | Driver| |Driver  | |Driver| | |backend  |  |backend | |backend |
62        +---+---+ +---+----+ +--+---+ | +---------+  +----+---+ +--------+
63            |         |         |     |      |            |         |
64            |         |         |     |User  |            |         |
65            |         |         |     |------|------------|--+------|-------
66            +---------+---------+     |Kernel                |
67                      |               |                      |
68  +---------+     +---+----+ +------+ | +------+ +------+ +--+---+  +-------+
69  |MIC DMA  |     |  VOP   | | SCIF | | | SCIF | | COSM | | VOP  |  |MIC DMA|
70  +---+-----+     +---+----+ +--+---+ | +--+---+ +--+---+ +------+  +----+--+
71      |               |         |     |    |        |                    |
72  +---+-----+     +---+----+ +--+---+ | +--+---+ +--+---+ +------+  +----+--+
73  |MIC      |     |  VOP   | |SCIF  | | |SCIF  | | COSM | | VOP  |  | MIC   |
74  |HW Bus   |     |  HW Bus| |HW Bus| | |HW Bus| | Bus  | |HW Bus|  |HW Bus |
75  +---------+     +--------+ +--+---+ | +--+---+ +------+ +------+  +-------+
76      |               |         |     |       |     |                    |
77      |   +-----------+--+      |     |       |    +---------------+     |
78      |   |Intel MIC     |      |     |       |    |Intel MIC      |     |
79      |   |Card Driver   |      |     |       |    |Host Driver    |     |
80      +---+--------------+------+     |       +----+---------------+-----+
81                 |                    |                   |
82             +-------------------------------------------------------------+
83             |                                                             |
84             |                    PCIe Bus                                 |
85             +-------------------------------------------------------------+
86