1.. _twister_blackbox:
2
3Twister blackbox tests
4######################
5
6This guide aims to explain the structure of a test file so the reader will be able
7to understand existing files and create their own. All developers should fix any tests
8they break and create new ones when introducing new features, so this knowledge is
9important for any Twister developer.
10
11Basics
12******
13
14Twister blackbox tests are written in python, using the ``pytest`` library.
15Read up on it :ref:`here <integration_with_pytest>` .
16Auxiliary test data follows whichever format it was in originally.
17Tests and data are wholly contained in the :zephyr_file:`scripts/tests/twister_blackbox`
18directory and prepended with ``test_``.
19
20Blackbox tests should not be aware of the internal twister code. Instead, they should
21call twister as user would and check the results.
22
23Sample test file
24****************
25
26.. literalinclude:: ./sample_blackbox_test.py
27   :language: python
28   :linenos:
29
30Comparison with CLI
31*******************
32
33Test above runs the command
34
35.. code-block:: console
36
37    twister -i --outdir $OUTDIR -T $TEST_DATA/tests -y --level $LEVEL
38    --test-config $TEST_DATA/test_config.yaml -p qemu_x86 -p frdm_k64f
39
40It presumes a CLI with the ``zephyr-env.sh`` or ``zephyr-env.cmd`` already run.
41
42Such a test provides us with all the outputs we typically expect of a Twister run thanks to
43``importlib`` 's ``exec_module()`` [#f1]_ .
44We can easily set up all flags that we expect from a Twister call via ``args`` variable [#f2]_ .
45We can check the standard output or stderr in ``out`` and ``err`` variables.
46
47Beside the standard outputs, we can also investigate the file outputs, normally placed in
48``twister-out`` directories. Most of the time, we will use the ``out_path`` fixture in conjunction
49with ``--outdir`` flag (L52) to keep test-generated files in temporary directories.
50Typical files read in blackbox tests are ``testplan.json`` , ``twister.xml`` and ``twister.log`` .
51
52Other functionalities
53*********************
54
55Decorators
56==========
57
58* ``@pytest.mark.usefixtures('clear_log')``
59    - allows us to use ``clear_log`` fixture from ``conftest.py`` .
60      The fixture is to become ``autouse`` in the future.
61      After that, this decorator can be removed.
62* ``@pytest.mark.parametrize('level, expected_tests', TESTDATA_X, ids=['smoke', 'acceptance'])``
63    - this is an example of ``pytest`` 's test parametrization.
64      Read up on it `here <https://docs.pytest.org/en/7.1.x/example/parametrize.html#different-options-for-test-ids>`__.
65      TESTDATAs are most often declared as class fields.
66* ``@mock.patch.object(TestPlan, 'TESTSUITE_FILENAME', testsuite_filename_mock)``
67    - this decorator allows us to use only tests defined in the ``test_data`` and
68      ignore the Zephyr testcases in the ``tests`` directory. **Note that all ``test_data``
69      tests use** ``test_data.yaml`` **as a filename, not** ``testcase.yaml`` **!**
70      Read up on the ``mock`` library
71      `here <https://docs.python.org/3/library/unittest.mock.html>`__.
72
73Fixtures
74========
75
76Blackbox tests use ``pytest`` 's fixtures, further reading on which is available
77`here <https://docs.pytest.org/en/6.2.x/fixture.html>`__.
78
79If you would like to add your own fixtures,
80consider whether they will be used in just one test file, or in many.
81
82* If in many, create such a fixture in the
83  :zephyr_file:`scripts/tests/twister_blackbox/conftest.py` file.
84
85    - :zephyr_file:`scripts/tests/twister_blackbox/conftest.py` already contains some fixtures -
86      take a look there for an example.
87* If in just one, declare it in that file.
88
89    - Consider using class fields instead - look at TESTDATAs for an example.
90
91How do I...
92***********
93
94Call Twister multiple times in one test?
95========================================
96
97Sometimes we want to test something that requires prior Twister use. ``--test-only``
98flag would be a typical example, as it is to be coupled with previous ``--build-only``
99Twister call. How should we approach that?
100
101If we just call the ``importlib`` 's ``exec_module`` two times, we will experience log
102duplication. ``twister.log`` will duplicate every line (triplicate if we call it three times, etc.)
103instead of overwriting the log or appending to the end of it.
104
105It is caused by the use of logger module variables in the Twister files.
106Thus us executing the module again causes the loggers to have multiple handles.
107
108To overcome this, between the calls you ought to use
109
110.. code:: python
111
112    capfd.readouterr()   # To remove output from the buffer
113                         # Note that if you want output from all runs after each other,
114                         # skip this line.
115    clear_log_in_test()  # To remove log duplication
116
117
118------
119
120.. rubric:: Footnotes
121
122.. [#f1] Take note of the ``setup_class()`` class function, which allows us to run
123         ``twister`` python file as if it were called directly
124         (bypassing the ``__name__ == '__main__'`` check).
125
126.. [#f2] We advise you to keep the first section of ``args`` definition intact in almost all
127         of your tests, as it is used for the common test setup.
128