1.. TinyTestFW documentation master file, created by
2   sphinx-quickstart on Thu Sep 21 20:19:12 2017.
3   You can adapt this file completely to your liking, but it should at least
4   contain the root `toctree` directive.
5
6Welcome to TinyTestFW's documentation!
7======================================
8
9We have a lot of test which depends on interact with DUT via communication port.
10Usually we send command to the port and then check response to see if the test succeed.
11TinyTestFW is designed for such scenarios.
12It supports ESP-IDF applications and can be adapted to other applications by writing new bundles.
13
14Example
15-------
16
17Let's first check a simple example::
18
19    import re
20    import os
21    import sys
22    test_fw_path = os.getenv("TEST_FW_PATH")
23    if test_fw_path:
24        sys.path.insert(0, test_fw_path)
25
26    import TinyFW
27    from IDF import IDFApp, IDFDUT
28
29
30    @TinyFW.test_method(app=IDFApp.Example, dut=IDFDUT.IDFDUT, env_tag="Example_WIFI",
31                        chip="ESP32", module="examples", execution_time=1)
32    def test_examples_protocol_https_request(env, extra_data):
33        """
34        steps: |
35          1. join AP
36          2. connect to www.howsmyssl.com:443
37          3. send http request
38        """
39        dut1 = env.get_dut("https_request", "examples/protocols/https_request")
40        dut1.start_app()
41        dut1.expect("Connecting to www.howsmyssl.com:443", timeout=30)
42        dut1.expect("Performing the SSL/TLS handshake")
43        dut1.expect("Certificate verified.", timeout=15)
44        dut1.expect_all(re.compile(r"Cipher suite is TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256"),
45                        "Reading HTTP response",
46                        timeout=20)
47        dut1.expect(re.compile(r"Completed (\d) requests"))
48
49
50    if __name__ == '__main__':
51        TinyFW.set_default_config(env_config_file="EnvConfigTemplate.yml")
52        test_examples_protocol_https_request()
53
54
55SOP for adding test cases
56-------------------------
57
581. Import test framework:
59^^^^^^^^^^^^^^^^^^^^^^^^^
60
61* We assume ``TEST_FW_PATH`` is pre-defined before running the tests
62* Then we can import python packages and files from ``TEST_FW_PATH``
63
642. Define test case:
65^^^^^^^^^^^^^^^^^^^^
66
671. Define test case ``test_xxx(env, extra_data)``
68    * env: instance of test env, see :doc:`Test Env <Env>` for details
69    * extra_data: extra data passed from test case caller
702. Add decorator for test case
71    * add decorator ``TinyFW.test_method`` to test method
72    * define default case configs and filters in decorator, see :doc:`TinyFW.test_method <TinyFW>`
73
743. Execute test cases:
75^^^^^^^^^^^^^^^^^^^^^^
76
77* define in ``main`` section and execute from this file
78    1. set preset configs(optional). If the config is not define in case decorator, it will use the preset configs.
79    2. call test case method:
80        * if you don't pass any arguments, it will use default values
81        * you can pass ``extra_data`` to test case by adding ``extra_data=some_data`` as kwarg of test case method.
82          default value for extra_data is None.
83        * you can overwrite test case config by adding them as kwarg of test case method.
84          It will overwrite preset configs and case default configs.
85
86    Examples::
87
88        test_examples_protocol_https_request(extra_data=["data1", "data2"], dut=SomeOtherDUT, env_tag="OtherEnv")
89
90* or, use ``runner`` to execute. see :doc:`runner <Runner>` for details
91
92Test FW features
93----------------
94
951. Test Environment:
96    1. DUT: DUT class provides methods to interact with DUT
97        * read/write through port
98        * expect method which supports expect one or multiple string or RegEx
99        * tool methods provided by the tool bundle, like ``start_app``, ``reset``
100    2. App:
101        * provide some specific features to the test application of DUT, for example:
102            * SDK path
103            * SDK tools
104            * application information like partition table, download configs
105    3. Environment Configs:
106        * support get env configs from config file or auto-detect from current PC
107        * provide ``get_variable`` method to get variables
1082. Allow to customize components (DUT, App) to support different devices
1093. Integrate to CI:
110    * provide interfaces for Gitlab-CI
111    * provide ``search case`` and ``runner`` interfaces, able to integrate with other CI
112
113
114Class Diagram
115=============
116.. uml::
117
118   class BaseDUT {
119   {field} app
120   {method} expect
121   {method} expect_any
122   {method} expect_all
123   {method} read
124   {method} write
125   {method} start_receive
126   {method} stop_receive
127   {method} close
128   }
129   class SerialDUT {
130   {method} _port_read
131   {method} _port_write
132   {method} _port_open
133   {method} _port_close
134   }
135   class IDFDUT {
136   {method} reset
137   {method} start_app
138   }
139   class BaseApp {
140   {method} get_sdk_path
141   {method} get_log_folder
142   }
143   class IDFApp {
144   {field} flash_files
145   {field} flash_settings
146   {field} partition_table
147   }
148   class Example {
149   {method} get_binary_path
150   }
151   class EnvConfig {
152   {method} get_variable
153   }
154   class Env {
155   {field} config
156   {field} allocated_duts
157   {field} app_cls
158   {method} get_dut
159   {method} close_dut
160   {method} get_variable
161   {method} get_pc_nic_info
162   {method} close
163   }
164
165   SerialDUT --|> BaseDUT
166   IDFDUT --|> SerialDUT
167   IDFApp --|> BaseApp
168   Example --|> IDFApp
169   Env *-- EnvConfig
170   Env *-- BaseDUT
171   Env o-- BaseApp
172   BaseDUT o-- BaseApp
173
174
175.. toctree::
176   :maxdepth: 2
177   :caption: Contents:
178
179   modules
180
181Dependencies
182============
183
184Support for both Python2 and Python3 (tested on python 2.7.13 and 3.6.2).
185
186The following 3rd party lib is required:
187
188    * pyserial
189    * pyyaml
190    * junit_xml
191    * netifaces
192    * matplotlib (if use Utility.LineChart)
193
194These libraries can be installed by running ``pip install --user -r requirements.txt`` in tiny-test-fw directory.
195
196To build document, we need to install ``Sphinx``, ``plantweb`` and ``sphinx-rtd-theme`` (you may replace this with your own theme). ``plantweb`` requires internet access during building document.
197
198
199Indices and tables
200==================
201
202* :ref:`genindex`
203* :ref:`modindex`
204* :ref:`search`
205