README.md
1# Unit tests for CMSIS-NN
2
3Unit test CMSIS-NN functions on any [Arm Mbed OS](https://os.mbed.com/mbed-os/) supported HW or using a fixed virtual platform (FVP) based on [Arm Corstone-300 software](https://developer.arm.com/ip-products/subsystem/corstone/corstone-300).
4
5The [Unity test framework](http://www.throwtheswitch.org/unity) is used for running the actual unit tests.
6
7For a quick setup, it is reccomended to the helper script targetting the Arm Corstone-300 softwware. See the section " Using FVP based on Arm Corstone-300 software ".
8
9## Requirements
10
11The following apt packages are required. Replace python venv version with your python version.
12```
13sudo apt install ruby-full cmake python3.X-venv pip curl git git-lfs
14```
15
16Python3 is required.
17It has been tested with Python 3.10 and it has been tested on Ubuntu 22.04.
18
19Make sure to use the latest pip version before starting.
20If in a virtual environment just start by upgrading pip.
21
22```
23pip install --upgrade pip
24```
25
26After upgrading pip, the requirements file found in Tests/UnitTests can be installed. This contains all
27python modules required to run all of the scripts. This will install tensorflow and keras to allow the use of
28the generate_test_data.py script. If you have version specific requirements, it is reccomended to install this
29requirements.txt in a virtual environment.
30
31```
32pip install -r requirements.txt
33```
34
35#### Get flatc and schema
36
37Note this is only needed for generating SVDF unit tests.
38
39For flatc compiler clone this [repo](https://github.com/google/flatbuffers) and build:
40```
41cd flatbuffers
42cmake -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Release
43make
44```
45Remember to add the built flatc binary to the path.
46
47For schema file download [schema.fbs](https://raw.githubusercontent.com/tensorflow/tensorflow/master/tensorflow/lite/schema/schema.fbs).
48
49#### Interpreter for generating reference output
50Python package tensorflow is always needed however the script has the option to use other interpreters than the tensorflow default, which will generate the actual reference output.
51
52##### tflite_runtime
53Python package tflite_runtime can be installed with pip and it can also be built locally. Check this [link](https://www.tensorflow.org/lite/guide/build_cmake_pip) on how to do that.
54Use the -h flag to get more info on supported interpreters.
55
56##### tflite_micro
57Python package tflite_micro can be installed with pip and it can also be built locally. See this comment for more info: https://github.com/tensorflow/tflite-micro/issues/1484#issuecomment-1677842603. This interpreter is only partially supported, see *Tests depending on TFLM interpreter*.
58
59## Getting started
60
61### Using Arm Mbed OS supported hardware
62
63Connect any HW (e.g. NUCLEO_F746ZG) that is supported by Arm Mbed OS. Multiple boards are supported. Note that board must be mounted, which may or may not be done automatically depending on OS. If all requirements are satisfied you can just run:
64
65```
66./unittest_targets.py
67```
68
69Use the -h flag to get more info.
70
71### Using FVP based on Arm Corstone-300 software
72
73The easiest way to run the unit tests on Corstone-300 is to use the build_and_run_tests.sh script. This script will install required packages, build unit tests and run unit tests. This script has been designed for Linux hosts with both aarch64 and x86_64 architectures. For more help use the '-h' flag on the script.
74
75Sample usage:
76```
77./build_and_run_tests.sh -c cortex-m3,cortex-m7,cortex-m55 -o '-Ofast'
78```
79By default the script will download and target gcc. To use arm compiler ensure that arm compilers folder is located in path, export CC and use the -a option on the script.
80
81Downloaded dependencies including python venv can be found in Tests/UnitTests/downloads. Test elfs can be found in Tests/UnitTests/build-($cpu) directories.
82
83Otherwise, you can build it manually:
84
85The build for unit tests differs from the build of CMSIS-NN as a [standalone library](https://github.com/ARM-software/CMSIS-NN/blob/main/README.md#building-cmsis-nn-as-a-library) in that, there is a dependency to [CMSIS](https://github.com/ARM-software/CMSIS_5) project for the startup files from CMSIS-Core. This is specified by the mandatory CMSIS_PATH CMake argument.
86
87
88Here is an example for testing on Arm Cortex-M55:
89
90```
91cd </path/to/CMSIS_NN/Tests/Unittest>
92mkdir build
93cd build
94cmake .. -DCMAKE_TOOLCHAIN_FILE=</path/to/ethos-u-core-platform>/cmake/toolchain/arm-none-eabi-gcc.cmake -DTARGET_CPU=cortex-m55 -DCMSIS_PATH=</path/to/CMSIS>
95make
96```
97
98This will build all unit tests. You can also just build a specific unit test only, for example:
99
100```
101make test_arm_depthwise_conv_s8_opt
102```
103
104Then you need to download and install the FVP based Arm Corstone-300 software, for example:
105
106```
107mkdir -p /home/$user/FVP
108wget https://developer.arm.com/-/media/Arm%20Developer%20Community/Downloads/OSS/FVP/Corstone-300/FVP_Corstone_SSE-300_Ethos-U55_11.12_57.tgz
109tar -xvzf FVP_Corstone_SSE-300_Ethos-U55_11.12_57.tgz
110./FVP_Corstone_SSE-300_Ethos-U55.sh --i-agree-to-the-contained-eula --no-interactive -d /home/$user/FVP
111export PATH="/home/$user/FVP/models/Linux64_GCC-6.4:$PATH"
112```
113
114Finally you can run the unit tests. For example:
115
116```
117FVP_Corstone_SSE-300_Ethos-U55 --cpulimit 2 -C mps3_board.visualisation.disable-visualisation=1 -C mps3_board.telnetterminal0.start_telnet=0 -C mps3_board.uart0.out_file="-" -C mps3_board.uart0.unbuffered_output=1 ./TestCases/test_arm_depthwise_conv_s8_opt/test_arm_depthwise_conv_s8_opt.elf
118```
119
120## Generating new test data
121**NOTE:** The data generation scrips are being reworked, see *Refactoring of generate_test_data*
122
123Generating new test data is done with the following script. Use the -h flag to get more info.
124
125```
126./generate_test_data.py -h
127
128```
129
130The script use a concept of test data sets, i.e. it need a test set data name as input. It will then generate files with that name as prefix. Multiple header files of different test sets can then be included in the actual unit test files.
131When adding a new test data set, new c files should be added or existing c files should be updated to use the new data set. See overview of the folders on how/where to add new c files.
132
133The steps to add a new unit test are as follows. Add a new test test in the load_all_testdatasets() function. Run the generate script with that new test set as input. Add the new generated header files to an existing or new unit test.
134
135### Tests depending on TFLM interpreter
136
137If TFL and TFLM reference kernels differ, CMSIS-NN aims to be bit-exact to TFLM reference kernels. Hence those operators depends on tflite_micro interpreter.
138
139Operator bit-exactness compability:
140
141| Operator | TFL bit-exact | TFLM bit-exact | Notes
142| --- | --- | --- | ---
143| convolution | x | x |
144| fully_connected | x | x |
145| lstm | | x |
146| svdf | | x | Operator is only fully supported by TFLM.
147| softmax | x | x |
148| avgpool | x | x |
149| maxpool | x | x |
150| add | x | x |
151| mul | x | x |
152| batch matmul | x | x |
153| pad | x | x |
154| minimum | x | x |
155| maximum | x | x |
156| transpose | x | x |
157
158### Refactoring of generate_test_data.py
159Test data generation is in progress of incrementally moving over to the cleaned up scripts placed in `RefactoredTestGen`.
160
161To try out the new scripts, use
162```
163./RefactoredTestGen/generate_test_data.py --help
164```
165
166The previous generate_test_data will remain as the main data generator until all functionality is replicated with the new scripts.
167
168Current progress:
169
170| Operator | Old | New | Notes
171| --- | --- | --- | ---
172| convolution | x | x | New version only supports 16x8 and int4 packed weights
173| depthwise conv | x | |
174| fully_connected | x | x | New version supports int4 packed weights. Only new version supports per channels quantization for int8.
175| lstm | | x | Only new version supporting 16x8
176| svdf | x | |
177| softmax | x | |
178| avgpool | | x |
179| maxpool | | x |
180| add | x | |
181| mul | x | |
182| batch matmul | | x |
183| pad | | x |
184| minimum | | x |
185| maximum | | x |
186| transpose | | x |
187
188## Overview of the Folders
189
190- `Corstone-300` - These are dependencies, like linker files etc, needed when building binaries targeting the FVP based on Arm Corstone-300 software. This is mostly taken from Arm Ethos-U Core Platform project.
191- `Mbed` - These are the Arm Mbed OS settings that are used. See Mbed/README.md.
192- `Output` - This will be created when building.
193- `PregeneratedData` - Host local(Not part of GitHub) test data for model creation using Keras in unit tests. It can be used for debug purposes when
194 adding new operators or debugging existing ones.
195- `TestCases` - Here are the actual unit tests. For each function under test there is a folder under here.
196- `TestCases/<cmsis-nn function name>` - For each function under test there is a folder with the same name with test_ prepended to the name and it contains a c-file with the actual unit tests. For example for arm_convolve_s8() the file is called test_arm_convolve_s8.c
197- `TestCases/<cmsis-nn function name>/Unity` - This folder contains a Unity file that calls the actual unit tests. For example for arm_convolve_s8() the file is called unity_test_arm_convolve_s8.c.
198- `TestCases/<cmsis-nn function name>/Unity/TestRunner` - This folder will contain the autogenerated Unity test runner.
199- `TestCases/TestData` - This is auto generated test data in .h files that the unit tests are using. The data in PregenrateData folder has fp32 data of a network whereas here it is the quantized equivalent of the same. They are not the same. All data can regenerated or only parts of it (e.g. only bias data). Of course even the config can be regenerated. This partial/full regeneration is useful during debugging.
200- `TestCases/Common` - Common files used in test data generation is placed here.
201- `RefactoredTestGen` - Temporary location for new test generation scripts
202
203## Formatting
204
205The python test scripts should be formatted with yapf.
206
207```
208pip install --upgrade yapf
209```
210
211The following settings are used.
212
213```
214python -m yapf --in-place --style='{based_on_style:pep8,column_limit:120,indent_width:4}' *.py
215```
216