1# EDTT framework and BabbleSim 2(With focus on Zephyr BLE testing on a PC) 3 4## Introduction 5 6The purpose of this document is to provide a basic understanding of the EDTT Framework for developers writing Tests for the Framework. It is assumed that the reader is familiar and knows how to use [BabbleSim](http://babblesim.github.io/), and how to run [Zephyr](https://docs.zephyrproject.org/) code on a standard Linux PC thru the [nrf52_bsim board](https://docs.zephyrproject.org/latest/boards/posix/nrf52_bsim/doc/index.html). 7The EDTT Framework is made for testing BLE functionality in the Zephyr Bluetooth implementation. 8Here, the EDTT Framework is using the BabbleSim radio simulation, together with simulated Zephyr Bluetooth stacks, where the Zephyr code is running on a standard Linux PC rather than a dedicated embedded system. 9This setup provides the developer with a powerful environment, where tracing and debugging becomes a lot easier compared to an embedded system. 10Tests, and the EDTT framework are written in Python, and their execution is decoupled from the simulated Zephyr Bluetooth stacks, in the sense that tests are running in their own process communicating with the simulated Zephyr Bluetooth stack through Named Pipes. 11Each test relies on a specific Zephyr application running in the simulated devices. This application can be a standalone application reacting to stimuli received “over the air”, a dedicated application responding to specific commands sent from the EDTT or a combination of the two. 12One of these test applications responds to standard HCI commands sent from the test and returns standard Events to the test. 13 14## Framework 15The Framework consists of a test execution part and a set of simulated Zephyr Bluetooth stacks as shown in the Figure 1 (EDTT framework TODO) below. 16 17 18 19The two execution environments are connected thru Named Pipes. Since Bluetooth is all about communicating entities, a test needs access to more than a single DUT. 20 21The EDTT transport ensures that the simulation is halted when needed while the EDTT is processing. That is, the result of the simulation does not depend on the python execution speed. 22In principle any number of DUTs can be supported, but in reality only two DUTs are used for testing. The setup is illustrated in the next figure . 23 24 25 26In addition to the DUTs, the EDTT BabbleSim transport also has a low level device that connects directly to BabbleSim. This device can be used to send raw BLE packets without going through a Zephyr device (with the limitations that brings). 27 28--- 29### The transport 30 31The transport is conceptually a two-way FIFO for each DUT. 32 33Commands are atomic transactions that maps to calls on the DUT side. Cannot be interleaved (for simplicity). 34 35Utils are a set of commands shared between tests. 36 37Transport EDTT side API: 38 39* open(device_id), connect(device_id), close(device_id) 40* send(device_id, message), 41* recv(device_id, count, timeout) 42* wait(time), get_time() 43 44Transport DUT side API: 45 46* start(), stop() 47* read(u8_t *pbuffer, size_t count, int flags) 48* write(u8_t *pbuffer, size_t count, int flags) 49 50--- 51 52In BabbleSim, EDTT is just another program being executed in parallel with the programs running the individual DUTs. Each DUT is configured to run one of the EDTT Test APPs, implementing the Command executor. 53 54The EDTT transport must know how many DUTs it needs to connect to and the identity of each DUT. This information is passed as run-time parameters to the EDTT. A typical execution of BabbleSim and two DUTs could look as shown here: 55 56``` 57./bs_2G4_phy_v1 –s=Test –D=3 –sim_length=5e6 -dump_imm & 58 59./bs_nrf52_bsim_hci_test_app_bsim –s=Test –d=1 –v=3 & 60 61./bs_nrf52_bsim_hci_test_app_bsim –s=Test –d=2 –v=3 62 63./bs_edttool –s=Test –d=0 –t bsim -T hci_verification –C HCI/CIN/BV-04-C -l -D=2 -devs 1 2 –RxWait=2.5e3 & 64``` 65 66 67The radio simulation is configured with the simulation name “Test” (-s=Test). It is told to host three devices (-D=3). The simulation length is set to 5 seconds (-sim_length=5e6). And it is told to disable cached writes for the dump files (-dump_imm). 68 69The EDTT Test APP is started twice to simulate the two DUTs. The EDTT Test APP is handed the name of the simulation “Test” (-s=Test) and its own device identifier (-d=1 or –d=2). 70 71Last, the EDTT itself is started. It is handed the name of the simulation “Test” (-s=Test), its own device identifier (-d=0), the number of DUTs to service (-D=2) and the device identifiers for the two DUTs (-dev 1 2). 72 73To run with the low level device enabled, EDTT itself also needs to be assigned a BabbleSim device number. This is done via the --low-level-device-nbr argument. For example, expanding on the above: 74 75``` 76./bs_2G4_phy_v1 –s=Test –D=4 –sim_length=5e6 -dump_imm & 77 78./bs_nrf52_bsim_hci_test_app_bsim –s=Test –d=1 –v=3 & 79 80./bs_nrf52_bsim_hci_test_app_bsim –s=Test –d=2 –v=3 & 81 82./bs_edttool –s=Test –d=0 –t bsim -T hci_verification –C HCI/CIN/BV-04-C -l --low-level-device-nbr=3 -D=2 -devs 1 2 –RxWait=2.5e3 83``` 84 85Now the simulation environment is started and ready to receive HCI Command requests from a test. See Tests to see how the test execution is started. 86 87Details on the transport layer API overall can be found in docs/EDTT_trasnport.md 88 89More information about the bsim transport, both the embedded driver and python side can be found in docs/EDTT_transport_bsim.md 90 91 92## HCI Command API 93 94One of the EDTT test applications running in the simulated Zephyr Bluetooth environment implements an HCI Command API that enables tests to execute HCI commands and receive the results of these executions as well as the HCI events generated. The HCI Command API is implemented as a request, reply protocol. In order to use this API, tests must pack commands into a byte stream that is sent over to the EDTT test application. The EDTT test application will generate a reply that is sent back to the test. 95All requests and replies share a common syntax of the form: 96``` 97 <request> ::= <request_id> <request_size> [ <request_parameter_list> ] 98 <request_id> ::= 1 | 3 | 5 | … | 193 99 <request_size> ::= 0 | 1 | 2 | … | 255 100 <request_parameter_list> ::= <request_parameter> { <request_parameter> } 101 <request_parameter> ::= 0 | 1 | 2 | … | 255 102 <reply> ::= <reply_id> <reply_size> [ <reply_parameter_list> ] 103 <reply_id> ::= 2 | 4 | 6 | … | 194 104 <reply_size> ::= 0 | 1 | 2 | … | 255 105 <reply_parameter_list> ::= <reply_parameter> { <reply_parameter> } 106 <reply_parameter> ::= 0 | 1 | 2 | … | 255 107 108<request_id>, <request_size>, <reply_id> and <reply_size> are little-endian 16 bit unsigned numbers. <request_parameter> and <reply_parameter> are 8 bit numbers. 109``` 110 111## Packet inspection 112 113It is possible to inspect the raw packets sent by the upper and lower tester via BabbleSims dump files. Note that you will usuallly want to use the `-dump_imm` command line argument when using packet inspection; Otherwise the dump files will use cached writes and the latest packets will likely not show up in EDTT. 114 115The raw packets will get decoded into a `Packet` before being handed over to the test cases via the `Packets` class. The content of a `Packet` is: 116 117* `direction`: Either `'Tx'` or `'Rx'`. Note that the current implementation only reads the Tx part of the dump files, so this will currently always be `'Tx'` 118* `idx`: The BSim device id of the device that received or transmitted the package (see `direction` for whether it was a transmit or receive). Will correspond to lower or upper tester 119* `ts`: The start timestamp of the packet (in microseconds) 120* `aa`: The access address used 121* `channel_num`: The channel number used (note: Not the channel *index*, but the channel *number* - ie. advertising channels are 0, 12 and 39) 122* `phy`: The BLE PHY used - one of `'1M'` or `'2M'` 123* `data`: The raw data of the packet (excluding header) 124* `type`: The type of packet, for example `'ADV_IND'` or `'CONNECT_IND'` 125* `header`: The header of the packet - a named tuple containing `pdu_type`, `ch_sel`, `tx_add`, `rx_add` and `payload_len` 126* `payload`: The decoded payload of the packet. Content depends on the type of packet; It will generally be a named tuple containing the fields specified in the BT Core Spec. One notable exception is for the extended advertisement packets - for these the payload is a `dict` since the fields may or may not be present 127 128### Packets class interface 129 130There are a few different ways to inquire about packets sent on the phy from within a test case. Generally, the way to use the API is to filter for the packets you are interested - see the API describtion below. 131 132#### Packets.fetch(packet_filter=()) 133 134Returns an iterator to use in a `for` loop. Will iterate through all packets matching the provided filter in the order of oldest packet to newest. 135 136Arguments: 137 138* `packet_filter`: Type or list of types to match on - for instance `('AUX_CONNECT_RSP', 'AUX_CONNECT_REQ')` 139 140Simple example - to loop through all AUX_ADV_IND packets and print them: 141 142``` 143 for packet in packets.fetch(packet_filter=('AUX_ADV_IND')): 144 # Do something with packet 145 print(packet) 146``` 147 148#### Packets.find(packet_type=None) 149 150Returns the first (ie. oldest) `Packet` matching the provided type filter. 151 152Arguments: 153 154* `packet_type`: Type or list of types to match on - for instance `('AUX_CONNECT_RSP', 'AUX_CONNECT_REQ')` 155 156#### Packets.findLast(packet_filter=()) 157 158Returns the last (ie. newest) `Packet` matching the provided filter. 159 160Arguments: 161 162* `packet_filter`: Type or list of types to match on - for instance `('AUX_CONNECT_RSP', 'AUX_CONNECT_REQ')` 163 164#### Packets.flush() 165 166Flushes the current packets to start fresh. All packets currently known will be removed and can no longer be retrieved. 167Note that this is implicitly done between test cases, so calling this function is only needed if a flush inside a test case is wanted. 168 169## Tests 170 171Tests are written in Python and executed in their own execution environment. All the necessary support for handling any LE HCI command is implemented in the EDTT Test APP and the supporting Python module basic_commands.py. The Python module basic_commands.py provides a function interface for all supported LE HCI commands. The supporting Python module utils.py provides a set of common functions that can be utilized by tests. 172 173### Writing tests 174 175Tests are grouped in modules. 176A test module must implement the functions `get_tests_specs()` and `run_a_test()`: 177 178* `get_tests_specs()`: Must return a dictionary, containing one entry for each test provided by the module. Each entry of type TestSpec TestSpec is defined in the supporting Python module test_spec.py from components, which includes between others the test name, description, and the number_devices parameter, which specifies how many devices this test or tests requires. See the example below. 179* `run_a_test()`: Which will execute an actual test given a TestSpec. 180 181Python tests that wish to exploit the HCI command API should start by importing the two Python modules basic_commands.py and utils.py from components. 182 183``` 184from components.utils import *; 185from components.basic_commands import *; 186``` 187 188This provides access to all the LE HCI commands in basic_commands.py and the utility functions in utils.py. 189 190For the actual tests, call functions from basic_commands.py and handle the HCI events generated. It is essential that the generated HCI events are handled in the test, otherwise they will remain in the event queue and possibly be picked up by later tests, which will then fail. 191 192Please note that the call to get_event is blocking, it will wait for an event to arrive in the queue. If unsure whether an event is generated in a particular situation, use the has_event function to poll for events prior to calling get_event. 193 194In the example below, the test is calling the HCI function read_local_version_information with the parameters transport, 0 and 100. Transport is a reference to the transport layer, to be used for sending the HCI command to the DUT. The 0 is an identifier for the DUT to receive the command. The 100 is a timeout parameter (specified in milliseconds) specifying how long to wait for an answer to arrive. 195 196The read_local_version_information command will return a status and several other parameters. Here we only pick up the status and check that it is zero, indicating that the command succeeded. 197 198The read_local_version_information command will generate a COMMAND COMPLETE event, which is picked up. The test verifies that the event generated is a COMMAND COMPLETE event and displays details from the event using one of the utility functions from utils.py. 199 200Note that get_event returns an eventTime, which is derived from the system up-time in milliseconds. 201 202``` 203""" 204 HCI/CIN/BV-04-C [Read Local Version Information Command] 205""" 206def main(args, transport, trace): 207 trace.trace(3, \ 208 "HCI/CIN/BV-04-C [Read Local Version Information Command]"); 209 try: 210 status = read_local_version_information(transport,0,100)[0]; 211 trace.trace(1, "VIC returns status: 0x%02X" % status); 212 success = status == 0; 213 eventTime,event,subEvent,eventData = get_event(transport,0,100); 214 success = success and (event == Events.BT_HCI_EVT_CMD_COMPLETE); 215 showEvent(event, eventData); 216 except Exception as e: 217 trace.trace(3, "VIC test failed: %s" % str(e)); 218 success = False; 219 220 trace.trace(3, "VIC test " + ("PASSED" if success else "FAILED")); 221``` 222 223For a trivially simple test and module example check tests/echo_test.py 224 225### Executing tests 226 227A Python wrapper has been made to make test execution easy. The wrapper is edttool.py. The syntax is shown below. 228 229The `<test>` argument is the name of the Python file (without .py) holding the code for the test to execute. The `<trace-level>` argument is a number that can be used to limit the trace output from the test. 230 231When the bsim transport is selected: The `<sim_id>` is a string that must match the `<sim_id>` used for the execution of the EDTT APP in the simulator. The `<eddtool_dev_nbr>` is the device identifier assigned to the EDTT itself. 232 233``` 234edttool.py [-h] [-v trace-level] -t TRANSPORT -T TEST [-C CASE] [--shuffle] [-S] [--seed SEED] [-s sim_id] [-d eddtool_dev_nbr] [-l] [--low-level-device-nbr low_level_dev_nbr] 235 236 237Example: 238edttool.py –s=Test –d=0 –t bsim -T hci_verification –C HCI/CIN/BV-04-C 239``` 240 241## Repositories 242 243BabbleSim, EDTT Tool and EDTT applications all resides in different GIT repositories: 244 245* Babblesim resides in: https://github.com/BabbleSim 246 247* nrf52_bsim is part of Zephyr upstream repo: https://github.com/zephyrproject-rtos/zephyr 248 249* The Zephyr edtt_app is also part of upstream Zephyr 250 251* EDTTool resides in the repository: https://github.com/EDTTool/EDTT 252
