1An Intel MIC X100 device is a PCIe form factor add-in coprocessor
2card based on the Intel Many Integrated Core (MIC) architecture
3that runs a Linux OS. It is a PCIe endpoint in a platform and therefore
4implements the three required standard address spaces i.e. configuration,
5memory and I/O. The host OS loads a device driver as is typical for
6PCIe devices. The card itself runs a bootstrap after reset that
7transfers control to the card OS downloaded from the host driver. The
8host driver supports OSPM suspend and resume operations. It shuts down
9the card during suspend and reboots the card OS during resume.
10The card OS as shipped by Intel is a Linux kernel with modifications
11for the X100 devices.
12
13Since it is a PCIe card, it does not have the ability to host hardware
14devices for networking, storage and console. We provide these devices
15on X100 coprocessors thus enabling a self-bootable equivalent
16environment for applications. A key benefit of our solution is that it
17leverages the standard virtio framework for network, disk and console
18devices, though in our case the virtio framework is used across a PCIe
19bus. A Virtio Over PCIe (VOP) driver allows creating user space
20backends or devices on the host which are used to probe virtio drivers
21for these devices on the MIC card. The existing VRINGH infrastructure
22in the kernel is used to access virtio rings from the host. The card
23VOP driver allows card virtio drivers to communicate with their user
24space backends on the host via a device page. Ring 3 apps on the host
25can add, remove and configure virtio devices. A thin MIC specific
26virtio_config_ops is implemented which is borrowed heavily from
27previous similar implementations in lguest and s390.
28
29MIC PCIe card has a dma controller with 8 channels. These channels are
30shared between the host s/w and the card s/w. 0 to 3 are used by host
31and 4 to 7 by card. As the dma device doesn't show up as PCIe device,
32a virtual bus called mic bus is created and virtual dma devices are
33created on it by the host/card drivers. On host the channels are private
34and used only by the host driver to transfer data for the virtio devices.
35
36The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a
37low level communications API across PCIe currently implemented for MIC.
38More details are available at scif_overview.txt.
39
40The Coprocessor State Management (COSM) driver on the host allows for
41boot, shutdown and reset of Intel MIC devices. It communicates with a COSM
42"client" driver on the MIC cards over SCIF to perform these functions.
43
44Here is a block diagram of the various components described above. The
45virtio backends are situated on the host rather than the card given better
46single threaded performance for the host compared to MIC, the ability of
47the host to initiate DMA's to/from the card using the MIC DMA engine and
48the fact that the virtio block storage backend can only be on the host.
49
50               +----------+           |             +----------+
51               | Card OS  |           |             | Host OS  |
52               +----------+           |             +----------+
53                                      |
54        +-------+ +--------+ +------+ | +---------+  +--------+ +--------+
55        | Virtio| |Virtio  | |Virtio| | |Virtio   |  |Virtio  | |Virtio  |
56        | Net   | |Console | |Block | | |Net      |  |Console | |Block   |
57        | Driver| |Driver  | |Driver| | |backend  |  |backend | |backend |
58        +---+---+ +---+----+ +--+---+ | +---------+  +----+---+ +--------+
59            |         |         |     |      |            |         |
60            |         |         |     |User  |            |         |
61            |         |         |     |------|------------|--+------|-------
62            +---------+---------+     |Kernel                |
63                      |               |                      |
64  +---------+     +---+----+ +------+ | +------+ +------+ +--+---+  +-------+
65  |MIC DMA  |     |  VOP   | | SCIF | | | SCIF | | COSM | | VOP  |  |MIC DMA|
66  +---+-----+     +---+----+ +--+---+ | +--+---+ +--+---+ +------+  +----+--+
67      |               |         |     |    |        |                    |
68  +---+-----+     +---+----+ +--+---+ | +--+---+ +--+---+ +------+  +----+--+
69  |MIC      |     |  VOP   | |SCIF  | | |SCIF  | | COSM | | VOP  |  | MIC   |
70  |HW Bus   |     |  HW Bus| |HW Bus| | |HW Bus| | Bus  | |HW Bus|  |HW Bus |
71  +---------+     +--------+ +--+---+ | +--+---+ +------+ +------+  +-------+
72      |               |         |     |       |     |                    |
73      |   +-----------+--+      |     |       |    +---------------+     |
74      |   |Intel MIC     |      |     |       |    |Intel MIC      |     |
75      |   |Card Driver   |      |     |       |    |Host Driver    |     |
76      +---+--------------+------+     |       +----+---------------+-----+
77                 |                    |                   |
78             +-------------------------------------------------------------+
79             |                                                             |
80             |                    PCIe Bus                                 |
81             +-------------------------------------------------------------+
82