Lines Matching +full:pytest +full:- +full:twister +full:- +full:harness
3 Integration with pytest test framework
6 *Please mind that integration of twister with pytest is still work in progress. Not every platform
7 type is supported in pytest (yet). If you find any issue with the integration or have an idea for
13 Pytest is a python framework that *“makes it easy to write small, readable tests, and can scale to
14 support complex functional testing for applications and libraries”* (`<https://docs.pytest.org/en/7…
15 Python is known for its free libraries and ease of using it for scripting. In addition, pytest
17 A pytest plugin ``pytest-twister-harness`` was introduced to provide an integration between pytest
18 and twister, allowing Zephyr’s community to utilize pytest functionality with keeping twister as
21 Integration with twister
24 By default, there is nothing to be done to enable pytest support in twister. The plugin is
25 developed as a part of Zephyr’s tree. To enable install-less operation, twister first extends
26 ``PYTHONPATH`` with path to this plugin, and then during pytest call, it appends the command with
27 ``-p twister_harness.plugin`` argument. If one prefers to use the installed version of the plugin,
28 they must add ``--allow-installed-plugin`` flag to twister’s call.
30 Pytest-based test suites are discovered the same way as other twister tests, i.e., by a presence
31 of test/sample.yaml. Inside, a keyword ``harness`` tells twister how to handle a given test.
32 In the case of ``harness: pytest``, most of twister workflow (test suites discovery,
38 :figclass: align-center
41 If ``harness: pytest`` is used, twister delegates the test execution to pytest, by calling it as
43 through a CLI command. When pytest is done, twister looks for a pytest report (results.xml) and
46 How to create a pytest test
49 An example folder containing a pytest test, application source code and Twister configuration .yaml
52 .. code-block:: none
55 ├─── pytest/
63 An example of a pytest test is given at
64 :zephyr_file:`samples/subsys/testsuite/pytest/shell/pytest/test_shell.py`. Using the configuration
65 provided in the ``testcase.yaml`` file, Twister builds the application from ``src`` and then, if the
66 .yaml file contains a ``harness: pytest`` entry, it calls pytest in a separate subprocess. A sample
69 .. code-block:: yaml
73 harness: pytest
76 By default, pytest tries to look for tests in a ``pytest`` directory located next to a directory
80 Pytest scans the given locations looking for tests, following its default
81 `discovery rules <https://docs.pytest.org/en/7.1.x/explanation/goodpractices.html#conventions-for-p…
86 There are two ways for passing extra arguments to the called pytest subprocess:
88 #. From .yaml file, using ``pytest_args`` placed under ``harness_config`` section - more info
90 #. Through Twister command line interface as ``--pytest-args`` argument. This can be particularly
94 .. code-block:: console
96 $ ./scripts/twister --platform native_sim -T samples/subsys/testsuite/pytest/shell \
97 -s samples/subsys/testsuite/pytest/shell/sample.pytest.shell \
98 --pytest-args='-k test_shell_print_version'
110 the core of pytest harness plugin. It is required to launch DUT (initialize logging, flash device,
113 writing tests which are device-type-agnostic. Scope of this fixture is determined by the
118 .. code-block:: python
135 .. code-block:: python
145 Sample fixture to wrap ``mcumgr`` command-line tool used to manage remote devices. More information
154 .. code-block:: python
179 .. code-block:: python
225 Examples of pytest tests in the Zephyr project
228 * :zephyr:code-sample:`pytest_shell`
229 * MCUmgr tests - :zephyr_file:`tests/boot/with_mcumgr`
230 * LwM2M tests - :zephyr_file:`tests/net/lib/lwm2m/interop`
231 * GDB stub tests - :zephyr_file:`tests/subsys/debug/gdbstub`
237 How to flash/run application only once per pytest session?
244 .. code-block:: yaml
246 harness: pytest
258 .. code-block:: yaml
260 harness: pytest
263 - "pytest/test_shell.py::test_shell_print_help"
265 Particular tests can be also chosen by pytest ``-k`` option (more info about pytest keyword
267 …`here <https://docs.pytest.org/en/latest/example/markers.html#using-k-expr-to-select-tests-based-o…
268 ). It can be applied by adding ``-k`` filter in ``pytest_args`` in .yaml file:
270 .. code-block:: yaml
272 harness: pytest
275 - "-k test_shell_print_help"
277 or by adding it to Twister command overriding parameters from the .yaml file:
279 .. code-block:: console
281 $ ./scripts/twister ... --pytest-args='-k test_shell_print_help'
288 .. code-block:: python
296 How to rerun locally pytest tests without rebuilding application by Twister?
299 This can be achieved by running Twister once again with ``--test-only`` argument added to Twister
300 command. Another way is running Twister with highest verbosity level (``-vv``) and then
301 copy-pasting from logs command dedicated for spawning pytest (log started by ``Running pytest
304 Is this possible to run pytest tests in parallel?
307 Basically ``pytest-harness-plugin`` wasn't written with intention of running pytest tests in
309 of tests is made by Twister, and it is responsible for managing available sources (jobs and
311 `pytest-xdist plugin <https://pytest-xdist.readthedocs.io/en/stable/>`_) they do so at their own