1# CMSIS-Core Unit Tests
2
3This folder contains a unit test suite that validates CMSIS-Core implementation.
4It uses test tools from LLVM, namely LIT and FileCheck, to validate consistency across compilers and devices.
5
6Consult the manual of [FileCheck - Flexible pattern matching file verifier](https://llvm.org/docs/CommandGuide/FileCheck.html)
7for details.
8
9## Folder structure
10
11```txt
12    �� Test
13    ┣ �� src                       Test source files
14    ┣ �� build.py                  Build wrapper
15    ┣ �� lit.cfg.py                LIT test suite configuration
16    ┣ �� requirements.txt          Python dependencies required for build.py script
17    ┗ �� vcpkg-configuration.json  vcpkg configuration to create virtual environment required for running these tests
18```
19
20## Test matrix
21
22Currently, the following build configurations are provided:
23
241. Compiler
25   - Arm Compiler 6 (AC6)
26   - GNU Compiler (GCC)
27   - LLVM/Clang (Clang)
282. Devices
29   - Cortex-M0
30   - Cortex-M0+
31   - Cortex-M3
32   - Cortex-M4
33     - w/o FPU
34     - with FPU
35   - Cortex-M7
36     - w/o FPU
37     - with SP FPU
38     - with DP FPU
39   - Cortex-M23
40     - w/o security extensions (TrustZone)
41     - in secure mode
42     - in non-secure mode
43   - Cortex-M33 (with FPU and DSP extensions)
44     - w/o security extensions (TrustZone)
45     - in secure mode
46     - in non-secure mode
47   - Cortex-M35P (with FPU and DSP extensions)
48     - w/o security extensions (TrustZone)
49     - in secure mode
50     - in non-secure mode
51   - Cortex-M55 (with FPU and DSP extensions)
52     - in secure mode
53     - in non-secure mode
54   - Cortex-M85 (with FPU and DSP extensions)
55     - in secure mode
56     - in non-secure mode
57   - Cortex-A5
58     - w/o NEON extensions
59     - w NEON extensions
60   - Cortex-A7
61     - w/o NEON extensions
62     - w NEON extensions
63   - Cortex-A9
64     - w/o NEON extensions
65     - w NEON extensions
663. Optimization Levels
67   - none
68   - balanced
69   - size
70   - speed
71
72## Prerequisites
73
74The following tools are required to build and run the CoreValidation tests:
75
76- [CMSIS-Toolbox 2.1.0](https://artifacts.keil.arm.com/cmsis-toolbox/2.1.0/)*
77- [CMake 3.25.2](https://cmake.org/download/)*
78- [Ninja 1.10.2](https://github.com/ninja-build/ninja/releases)*
79- [Arm Compiler 6.20](https://artifacts.keil.arm.com/arm-compiler/6.20/21/)*
80- [GCC Compiler 13.2.1](https://artifacts.keil.arm.com/arm-none-eabi-gcc/13.2.1/)*
81- [Clang Compiler 17.0.1](https://github.com/ARM-software/LLVM-embedded-toolchain-for-Arm/releases/tag/release-17.0.1)*
82- [Arm Virtual Hardware for Cortex-M based on FastModels 11.22.39](https://artifacts.keil.arm.com/avh/11.22.39/)*
83- [Python 3.9](https://www.python.org/downloads/)
84- [LLVM FileCheck](https://github.com/llvm/llvm-project/releases/)
85  - Ubuntu package `llvm-<version>-tools`
86  - MacOS Homebrew formula `llvm`
87
88The executables need to be present on the `PATH`.
89For tools distributed via vcpkg (*) this can be achieved automatically:
90
91```bash
92 ./CMSIS/Core/Test $ vcpkg activate
93```
94
95Install the Python packages required by `build.py`:
96
97```bash
98 ./CMSIS/Core/Test $ pip install -r requirements.txt
99```
100
101## Execute LIT tests
102
103To build and run the CoreValidation tests for one or more configurations use the following command line.
104Select the `<compiler>`, `<device>`, and `<optimize>` level to execute `lit` for.
105
106```bash
107 ./CMSIS/Core/Test $ ./build.py -c <compiler> -d <device> -o <optimize> [lit]
108```
109
110For example, to execute the LIT tests using GCC for Cortex-M3 with no optimization, run:
111
112```bash
113 ./CMSIS/Core/Test $ ./build.py -c GCC -d CM3 -o none lit
114[GCC][Cortex-M3][none](lit:run_lit) /opt/homebrew/bin/lit --xunit-xml-output lit.xml -D toolchain=GCC -D device=CM3 -D optimize=none src
115[GCC][Cortex-M3][none](lit:run_lit) -- Testing: 49 tests, 10 workers --
116 :
117[GCC][Cortex-M3][none](lit:run_lit) /opt/homebrew/bin/lit succeeded with exit code 0
118
119Matrix Summary
120==============
121
122compiler    device     optimize    lit
123----------  ---------  ----------  -----
124GCC         Cortex-M3  none        33/33
125```
126
127The summary lists the amount of test cases executed and passed.
128
129## Analyse failing test cases
130
131In case of failing test cases, one can run a single test case with verbose output like this:
132
133```bash
134 ./CMSIS/Core/Test $ lit -D toolchain=GCC -D device=CM3 -D optimize=none -a src/apsr.c
135-- Testing: 1 tests, 1 workers --
136PASS: CMSIS-Core :: src/apsr.c (1 of 1)
137Script:
138--
139: 'RUN: at line 2';   arm-none-eabi-gcc -mcpu=cortex-m3 -mfloat-abi=soft -O1 -I ../Include -D CORE_HEADER=\"core_cm3.h\" -c -D __CM3_REV=0x0000U -D __MPU_PRESENT=1U -D __VTOR_PRESENT=1U -D __NVIC_PRIO_BITS=3U -D __Vendor_SysTickConfig=0U -o ./src/Output/apsr.c.o ./src/apsr.c; llvm-objdump --mcpu=cortex-m3 -d ./src/Output/apsr.c.o | FileCheck --allow-unused-prefixes --check-prefixes CHECK,CHECK-THUMB ./src/apsr.c
140--
141Exit Code: 0
142 :
143********************
144
145Testing Time: 0.10s
146  Passed: 1
147```
148
149The output reveales wich commands are chained and their error output if any.
150
151Failing FileCheck requires in detail analysis of the `// CHECK` annotations in the test source file
152against the `llvm-objdump` output of the test compilation.
153
154Investigating the disassembly can be done like
155
156```bash
157 ./CMSIS/Core/Test $ llvm-objdump --mcpu=cortex-m3 -d ./src/Output/apsr.c.o
158
159./src/Output/apsr.c.o:  file format elf32-littlearm
160
161Disassembly of section .text:
162
16300000000 <get_apsr>:
164       0: b082          sub     sp, #0x8
165       2: f3ef 8300     mrs     r3, apsr
166       6: 9301          str     r3, [sp, #0x4]
167       8: b002          add     sp, #0x8
168       a: 4770          bx      lr
169```
170
171This output is expected to match the test case
172
173```c
174    // CHECK: mrs {{r[0-9]+}}, apsr
175    volatile uint32_t result = __get_APSR();
176```
177
178I.e., the test case expects the `mrs {{r[0-9]+}}, apsr` instruction, additional whitespace is ignored.
179