README.md
1# Test Apps
2
3This directory contains a set of ESP-IDF projects to be used as tests only, which aim to exercise various
4configuration of components to check completely arbitrary functionality should it be building only, executing under
5various conditions or combination with other components, including custom test frameworks.
6
7The test apps are not intended to demonstrate the ESP-IDF functionality in any way.
8
9# Test Apps projects
10
11Test applications are treated the same way as ESP-IDF examples, so each project directory shall contain
12* Build recipe in cmake and the main component with app sources
13* Configuration files, `sdkconfig.ci` and similar (see below)
14* Test executor in `ttfw_idf` format if the project is intended to also run tests (otherwise the example is build only)
15 - test file in the project dir must end with `_test.py`, by should be named `app_test.py`
16 - test cases shall be decorated with `@ttfw_idf.idf_custom_test(env_tag="...")`
17
18
19# CI Behavior
20
21## Configuration Files
22
23For each project in test_apps (and also examples):
24
25* If a file `sdkconfig.ci` exists then it's built as the `default` CI config.
26* If any additional files `sdkconfig.ci.<CONFIG>` exist then these are built as alternative configs, with the specified `<CONFIG>` name.
27
28The CI system expects to see at least a "default" config, so add `sdkconfig.ci` before adding any `sdkconfig.ci.CONFIG` files.
29
30* By default, every CI configurations is built for every target SoC (an `m * n` configuration matrix). However if any `sdkconfig.ci.*` file contains a line of the form `CONFIG_IDF_TARGET="targetname"` then that CI config is only built for that one target. This only works in `sdkconfig.ci.CONFIG`, not in the default `sdkconfig.ci`.
31* Each configuration is also built with the contents of any `sdkconfig.defaults` file or a file named `sdkconfig.defaults.<TARGET>` appended. (Same as a normal ESP-IDF project build.)
32
33## Test Execution
34
35If an example test or test app test supports more targets than just `ESP32`, then the `app_test.py` file needs to specify the list of supported targets in the test decorator. For example:
36
37```
38@ttfw_idf.idf_example_test(env_tag='Example_GENERIC', target=['esp32', 'esp32s2'])
39def test_app_xyz(env, extra_data):
40```
41
42If the app test supports multiple targets but you only want some of these targets to be run automatically in CI, the list can be further filtered down by adding the `ci_target` list:
43
44```
45@ttfw_idf.idf_example_test(env_tag='Example_GENERIC', target=['esp32', 'esp32s2'], ci_target=['esp32'])
46def test_app_xyz(env, extra_data):
47```
48
49(If no ci_target list is specified, all supported targets will be tested in CI.)
50
51# Test Apps layout
52
53The test apps should be grouped into subdirectories by category. Categories are:
54* `protocols` contains test of protocol interactions.
55* `network` contains system network tests
56* `system` contains tests on the internal chip features, debugging and development tools.
57* `security` contains tests on the chip security features.
58
59# Test Apps local execution
60All the following instructions are general. Part of them may be complemented by more particular instructions in the corresponding app's README.
61
62## Requirements
63The following requirements need to be satisfied in the IDF python virtual environment.
64
65* ttfw needs to be in the `PYTHONPATH`. Add it like this: `export PYTHONPATH=$PYTHONPATH:$IDF_PATH/tools/ci/python_packages`
66* Install all requirements from `tools/ci/python_packages/ttfw_idf/requirements.txt`: `pip install -r $IDF_PATH/tools/ci/python_packages/ttfw_idf/requirements.txt`
67
68You should also set the port via the environment variable ESPPORT to prevent the tools from looking and iterating over all serial ports. The latter causes much trouble, currently:
69```
70export ESPPORT=/dev/ttyUSB<X>
71```
72
73## Execution
74* Create an sdkconfig file from the relevant `sdkconfig.ci.<CONFIG>` and `sdkconfig.defaults`: `cat sdkconfig.defaults sdkconfig.ci.<CONFIG> > sdkconfig`
75* Run `idf.py menuconfig` to configure local project attributes
76* Run `idf.py build` to build the test app
77* Run `python app_test.py` to run the test locally
78
