1Unit Testing (Legacy GNU Make)
2==============================
3:link_to_translation:`zh_CN:[中文]`
4
5.. include:: ../gnu-make-legacy.rst
6
7ESP-IDF comes with a unit test application that is based on the Unity - unit test framework. Unit tests are integrated in the ESP-IDF repository and are placed in the ``test`` subdirectories of each component respectively.
8
9Normal Test Cases
10------------------
11
12Unit tests are located in the ``test`` subdirectory of a component.
13Tests are added in C files, a single C file can include multiple test cases.
14Test files start with the word "test".
15
16Each test file should include the ``unity.h`` header and the header for the C module to be tested.
17
18Tests are added in a function in the C file as follows:
19
20.. code-block:: c
21
22    TEST_CASE("test name", "[module name]"
23    {
24            // Add test here
25    }
26
27The first argument is a descriptive name for the test, the second argument is an identifier in square brackets.
28Identifiers are used to group related test, or tests with specific properties.
29
30.. note::
31    There is no need to add a main function with ``UNITY_BEGIN()`` and ``​UNITY_END()`` in each test case. ``unity_platform.c`` will run ``UNITY_BEGIN()`` autonomously, and run the test cases, then call ``​UNITY_END()``.
32
33Each ``test`` subdirectory needs to include a ``component.mk`` file with the following line of code::
34
35    COMPONENT_ADD_LDFLAGS = -Wl,--whole-archive -l$(COMPONENT_NAME) -Wl,--no-whole-archive
36
37See http://www.throwtheswitch.org/unity for more information about writing tests in Unity.
38
39
40Multi-device Test Cases
41------------------------
42
43The normal test cases will be executed on one DUT (Device Under Test). However, components that require some form of communication (e.g., GPIO, SPI) require another device to communicate with, thus cannot be tested normal test cases.
44Multi-device test cases involve writing multiple test functions, and running them on multiple DUTs.
45
46The following is an example of a Multi-device test case:
47
48.. code-block:: c
49
50    void gpio_master_test()
51    {
52        gpio_config_t slave_config = {
53                .pin_bit_mask = 1 << MASTER_GPIO_PIN,
54                .mode = GPIO_MODE_INPUT,
55        };
56        gpio_config(&slave_config);
57        unity_wait_for_signal("output high level");
58        TEST_ASSERT(gpio_get_level(MASTER_GPIO_PIN) == 1);
59    }
60
61    void gpio_slave_test()
62    {
63        gpio_config_t master_config = {
64                .pin_bit_mask = 1 << SLAVE_GPIO_PIN,
65                .mode = GPIO_MODE_OUTPUT,
66        };
67        gpio_config(&master_config);
68        gpio_set_level(SLAVE_GPIO_PIN, 1);
69        unity_send_signal("output high level");
70    }
71
72    TEST_CASE_MULTIPLE_DEVICES("gpio multiple devices test example", "[driver]", gpio_master_test, gpio_slave_test);
73
74
75The macro ``TEST_CASE_MULTIPLE_DEVICES`` is used to declare a multi-device test case.
76The first argument is test case name, the second argument is test case description.
77From the third argument, up to 5 test functions can be defined, each function will be the entry point of tests running on each DUT.
78
79Running test cases from different DUTs could require synchronizing between DUTs. We provide ``unity_wait_for_signal`` and ``unity_send_signal`` to support synchronizing with UART.
80As the scenario in the above example, the slave should get GPIO level after master set level. DUT UART console will prompt and requires user interaction:
81
82DUT1 (master) console::
83
84    Waiting for signal: [output high level]!
85    Please press "Enter" key once any board send this signal.
86
87DUT2 (slave) console::
88
89    Send signal: [output high level]!
90
91Once the signal is sent from DUT2, you need to press "Enter" on DUT1, then DUT1 unblocks from ``unity_wait_for_signal`` and starts to change GPIO level.
92
93Signals can also be used to pass parameters between multiple devices. For example, DUT1 want to know the MAC address of DUT2, so it can connect to DUT2.
94In this case, ``unity_wait_for_signal_param`` and ``unity_send_signal_param`` can be used:
95
96DUT1 console::
97
98    Waiting for signal: [dut2 mac address]!
99    Please input parameter value from any board send this signal and press "Enter" key.
100
101DUT2 console::
102
103    Send signal: [dut2 mac address][10:20:30:40:50:60]!
104
105Once the signal is sent from DUT2, you need to input ``10:20:30:40:50:60`` on DUT1 and press "Enter". Then DUT1 will get the MAC address string of DUT2 and unblock from ``unity_wait_for_signal_param``, then start to connect to DUT2.
106
107
108Multi-stage Test Cases
109-----------------------
110
111The normal test cases are expected to finish without reset (or only need to check if reset happens). Sometimes we expect to run some specific tests after certain kinds of reset.
112For example, we expect to test if reset reason is correct after a wakeup from deep sleep. We need to create a deep-sleep reset first and then check the reset reason.
113To support this, we can define multi-stage test cases, to group a set of test functions::
114
115    static void trigger_deepsleep(void)
116    {
117        esp_sleep_enable_timer_wakeup(2000);
118        esp_deep_sleep_start();
119    }
120
121    void check_deepsleep_reset_reason()
122    {
123        soc_reset_reason_t reason = esp_rom_get_reset_reason(0);
124        TEST_ASSERT(reason == RESET_REASON_CORE_DEEP_SLEEP);
125    }
126
127    TEST_CASE_MULTIPLE_STAGES("reset reason check for deepsleep", "[esp32]", trigger_deepsleep, check_deepsleep_reset_reason);
128
129Multi-stage test cases present a group of test functions to users. It need user interactions (select cases and select different stages) to run the case.
130
131
132Building Unit Test App
133----------------------
134
135Follow the setup instructions in the top-level esp-idf README.
136Make sure that ``IDF_PATH`` environment variable is set to point to the path of esp-idf top-level directory.
137
138Change into ``tools/unit-test-app`` directory to configure and build it:
139
140* ``make menuconfig`` - configure unit test app.
141
142* ``make TESTS_ALL=1`` - build unit test app with tests for each component having tests in the ``test`` subdirectory.
143* ``make TEST_COMPONENTS='xxx'`` - build unit test app with tests for specific components.
144* ``make TESTS_ALL=1 TEST_EXCLUDE_COMPONENTS='xxx'`` - build unit test app with all unit tests, except for unit tests of some components. (For instance: ``make TESTS_ALL=1 TEST_EXCLUDE_COMPONENTS='ulp mbedtls'`` - build all unit tests exludes ``ulp`` and ``mbedtls`` components).
145
146When the build finishes, it will print instructions for flashing the chip. You can simply run ``make flash`` to flash all build output.
147
148You can also run ``make flash TESTS_ALL=1`` or ``make TEST_COMPONENTS='xxx'`` to build and flash. Everything needed will be rebuilt automatically before flashing.
149
150Use menuconfig to set the serial port for flashing.
151
152Running Unit Tests
153------------------
154
155After flashing reset the ESP32 and it will boot the unit test app.
156
157When unit test app is idle, press "Enter" will make it print test menu with all available tests::
158
159    Here's the test menu, pick your combo:
160    (1)     "esp_ota_begin() verifies arguments" [ota]
161    (2)     "esp_ota_get_next_update_partition logic" [ota]
162    (3)     "Verify bootloader image in flash" [bootloader_support]
163    (4)     "Verify unit test app image" [bootloader_support]
164    (5)     "can use new and delete" [cxx]
165    (6)     "can call virtual functions" [cxx]
166    (7)     "can use static initializers for non-POD types" [cxx]
167    (8)     "can use std::vector" [cxx]
168    (9)     "static initialization guards work as expected" [cxx]
169    (10)    "global initializers run in the correct order" [cxx]
170    (11)    "before scheduler has started, static initializers work correctly" [cxx]
171    (12)    "adc2 work with wifi" [adc]
172    (13)    "gpio master/slave test example" [ignore][misc][test_env=UT_T2_1][multi_device]
173            (1)     "gpio_master_test"
174            (2)     "gpio_slave_test"
175    (14)    "SPI Master clockdiv calculation routines" [spi]
176    (15)    "SPI Master test" [spi][ignore]
177    (16)    "SPI Master test, interaction of multiple devs" [spi][ignore]
178    (17)    "SPI Master no response when switch from host1 (SPI2) to host2 (SPI3)" [spi]
179    (18)    "SPI Master DMA test, TX and RX in different regions" [spi]
180    (19)    "SPI Master DMA test: length, start, not aligned" [spi]
181    (20)    "reset reason check for deepsleep" [esp32][test_env=UT_T2_1][multi_stage]
182            (1)     "trigger_deepsleep"
183            (2)     "check_deepsleep_reset_reason"
184
185The normal case will print the case name and description. Master-slave cases will also print the sub-menu (the registered test function names).
186
187Test cases can be run by inputting one of the following:
188
189- Test case name in quotation marks (for example, ``"esp_ota_begin() verifies arguments"``) to run a single test case.
190
191- Test case index (for example, ``1``) to run a single test case.
192
193- Module name in square brackets (for example, ``[cxx]``) to run all test cases for a specific module.
194
195- An asterisk (``*``) to run all test cases
196
197``[multi_device]`` and ``[multi_stage]`` tags tell the test runner whether a test case is a multi-device or multi-stage test case.
198These tags are automatically added by ```TEST_CASE_MULTIPLE_STAGES`` and ``TEST_CASE_MULTIPLE_DEVICES`` macros.
199
200After you select a multi-device test case, it will print sub menu::
201
202    Running gpio master/slave test example...
203    gpio master/slave test example
204            (1)     "gpio_master_test"
205            (2)     "gpio_slave_test"
206
207You need to input a number to select the test running on the DUT.
208
209Similar to multi-device test cases, multi-stage test cases will also print sub-menu::
210
211    Running reset reason check for deepsleep...
212    reset reason check for deepsleep
213            (1)     "trigger_deepsleep"
214            (2)     "check_deepsleep_reset_reason"
215
216For the first time you execute this case, please input ``1`` to run the first stage (trigger deep-sleep).
217After DUT is rebooted and test cases are available to run, select this case again and input ``2`` to run the second stage.
218The case will only pass if the last stage passes and all previous stages trigger reset.
219