Lines Matching +full:pytest +full:- +full:twister +full:- +full:harness
3 Test Runner (Twister)
6 Twister scans for the set of test applications in the git repository
14 Because of the limited test execution coverage, twister
20 When using (at least) one ``-v`` option, twister's console output
24 of a test is likewise reported in the ``twister.json`` and other report files.
25 There are a few reasons why twister only builds a test and doesn't run it:
27 - The test is marked as ``build_only: true`` in its ``.yaml``
29 - The test configuration has defined a ``harness`` but you don't have
31 - The target device is not connected and not available for flashing
32 - You or some higher level automation invoked twister with
33 ``--build-only``.
39 .. group-tab:: Linux
41 .. code-block:: bash
43 $ source zephyr-env.sh
44 $ ./scripts/twister
46 .. group-tab:: Windows
48 .. code-block:: bat
50 zephyr-env.cmd
51 python .\scripts\twister
59 .. group-tab:: Linux
61 .. code-block:: bash
63 $ ./scripts/twister --all --enable-slow
65 .. group-tab:: Windows
67 .. code-block:: bat
69 python .\scripts\twister --all --enable-slow
75 the ``--platform`` option, it is a platform filter for testing, with this
78 you can use ``--platform board@revision`` to test on a specific revision.
80 The list of command line options supported by twister can be viewed using:
84 .. group-tab:: Linux
86 .. code-block:: bash
88 $ ./scripts/twister --help
90 .. group-tab:: Windows
92 .. code-block:: bat
94 python .\scripts\twister --help
110 .. code-block:: yaml
113 name: NXP FRDM-K64F
117 - zephyr
118 - gnuarmemb
119 - xtools
121 - arduino_gpio
122 - arduino_i2c
123 - netif:eth
124 - adc
125 - i2c
126 - nvs
127 - spi
128 - gpio
129 - usb_device
130 - watchdog
131 - can
132 - pwm
142 .. code-block:: console
145 west build -b reel_board
147 cmake -DBOARD=reel_board ..
156 .. code-block:: yaml
159 - name: qemu
160 - name: armfvp
162 - name: custom
166 simulation can be selected with ``--simulation <simulation_name>``.
186 .. code-block:: yaml
189 - pci
194 .. code-block:: yaml
197 - netif:eth
198 - sensor:bmi16
213 covered when invoking the simplified twister without any additional
226 power-efficient but slow CPU or simulation platform which can perform instruction accurate
230 A list of environment variables. Twister will check if all these environment variables are set,
233 presence to twister using these environment variables.
248 :alt: Twister and a Test application project.
249 :figclass: align-center
251 Twister and a Test application project.
298 The ``--no-detailed-test-id`` command line option modifies the above rules in this way:
301 Its Application Project path can be found in ``twister.json`` report as ``path:`` property.
304 must be unique for the Twister execution scope.
314 .. code-block:: yaml
333 .. code-block:: yaml
368 Don't run this test scenario unless ``--enable-slow`` or ``--enable-slow-only`` was
369 passed in on the command line. Intended for time-consuming test scenarios that
380 .. code-block:: yaml
389 - arch:x86:CONFIG_ADC_ASYNC=y
390 - platform:qemu_x86:CONFIG_DEBUG=y
391 - platform:mimxrt1060_evk:SHIELD=rk043fn66hs_ctg
392 - simulation:qemu:CONFIG_MPU=y
398 .. code-block:: yaml
407 - CONFIG_ADC_ASYNC=y
412 .. code-block:: yaml
421 - arch:x86:CONFIG_ADC_ASYNC=y
422 - platform:qemu_x86:CONFIG_DEBUG=y
426 If true, twister will not try to run the test even if the test is runnable
448 test will be selectable using the command line option ``--level <level name>``
488 This option limits the scope to the listed platforms when twister is
489 invoked with the ``--integration`` option. Use this instead of
497 When computing sizes, twister will report errors if it finds
505 the images onto the target. The ``--erase`` option of west flash is
509 harness: <string>
510 A harness keyword in the ``testcase.yaml`` file identifies a Twister
511 harness needed to run a test successfully. A harness is a feature of
512 Twister and implemented by Twister, some harnesses are defined as
515 A harness can be seen as the handler that needs to be implemented in
516 Twister to be able to evaluate if a test passes criteria. For example, a
517 keyboard harness is set on tests that require keyboard interaction to reach
518 verdict on whether a test has passed or failed, however, Twister lack this
519 harness implementation at the moment.
523 - ztest
524 - test
525 - console
526 - pytest
527 - gtest
528 - robot
533 harness if you've already got tests written in the gTest framework and do
534 not wish to update them to zTest. The ``console`` harness tells Twister to
536 The ``robot`` harness is used to execute Robot Framework test suites
541 - keyboard
542 - net
543 - bluetooth
545 Harness ``bsim`` is implemented in limited way - it helps only to copy the
564 .. code-block:: yaml
567 - arch
568 - simulation
575 harness_config: <harness configuration options>
576 Extra harness configuration options to be used to select a board and/or
599 of the test suite object in ``twister.json``.
604 .. code-block:: yaml
612 into ``twister.json`` as nested ``recording`` object properties.
613 The corresponding ``recording.csv`` columns will contain strings as-is.
621 .. code-block:: yaml
629 .. code-block:: none
633 will be reported in ``twister.json`` as:
635 .. code-block:: json
660 pytest_root: <list of pytest testpaths> (default pytest)
661 Specify a list of pytest directories, files or subtests that need to be
662 executed when a test scenario begins to run. The default pytest directory is
663 ``pytest``. After the pytest run is finished, Twister will check if
664 the test scenario passed or failed according to the pytest report.
665 As an example, a list of valid pytest roots is presented below:
667 .. code-block:: yaml
671 - "pytest/test_shell_help.py"
672 - "../shell/pytest/test_shell.py"
673 - "/tmp/test_shell.py"
674 - "~/tmp/test_shell.py"
675 - "$ZEPHYR_BASE/samples/subsys/testsuite/pytest/shell/pytest/test_shell.py"
676 - "pytest/test_shell_help.py::test_shell2_sample" # select pytest subtest
677 … - "pytest/test_shell_help.py::test_shell2_sample[param_a]" # select pytest parametrized subtest
682 Specify a list of additional arguments to pass to ``pytest`` e.g.:
683 ``pytest_args: [‘-k=test_method’, ‘--log-level=DEBUG’]``. Note that
684 ``--pytest-args`` can be passed multiple times to pass several arguments
685 to the pytest.
690 The scope for which ``dut`` and ``shell`` pytest fixtures are shared.
707 .. code-block:: yaml
713 harness: console
718 - "Temperature:(.*)C"
719 - "Relative Humidity:(.*)%"
726 The following is an example yaml file with pytest harness_config options,
727 default pytest_root name "pytest" will be used if pytest_root not specified.
728 please refer the examples in samples/subsys/testsuite/pytest/.
730 .. code-block:: yaml
733 harness: pytest
735 pytest.example.directories:
738 - pytest_dir1
739 - $ENV_VAR/samples/test/pytest_dir2
740 pytest.example.files_and_subtests:
743 - pytest/test_file_1.py
744 - test_file_2.py::test_A
745 - test_file_2.py::test_B[param_a]
749 .. code-block:: yaml
753 harness: robot
759 .. code-block:: yaml
763 harness: robot
766 - [robot file path 1]
767 - [robot file path 2]
768 - [robot file path n]
772 .. code-block:: yaml
776 harness: robot
780 - --exclude tag
781 - --stop-on-error
787 .. code-block:: none
795 …Twister will first evaluate the expression to find if a "limited" cmake call, i.e. using package_h…
803 .. code-block:: antlr
827 if the symbol is defined to a non-empty string.
834 * all comparison operators (non-associative)
839 .. code-block:: none
845 .. code-block:: none
853 .. code-block:: none
860 :ref:`Snippets <snippets>` are supported in twister for test scenarios that
861 require them. As with normal applications, twister supports using the base
869 .. code-block:: yaml
874 - cdc-acm-console
875 - user-snippet-example
879 running with ``-v`` or examining the discard report
885 To update this, pass the ``--all --release`` options.
896 One of the key features of Twister is its ability to decide on which platforms a given
897 test scenario should run. This behavior has its roots in Twister being developed as
902 Twister always prepares an initial list of platforms in scope for a given test,
906 Using ``--force-platform`` allows to override filtering caused by ``platform_allow``,
912 * ``-p/--platform <platform_name>`` (can be used multiple times): only platforms
914 * ``-l/--all``: all available platforms;
915 * ``-G/--integration``: all platforms from an ``integration_platforms`` list in
920 *"Scope presumption"*: A list of Twister's :ref:`default platforms <twister_default_testing_board>`
933 * ``--timeout-multiplier`` twister option which can be used to adjust timeouts in exact twister run.
945 be activated using the ``--integration`` option of twister and narrows down
954 Apart from the already supported QEMU and other simulated environments, Twister
955 supports running any out-of-tree custom emulator defined in the board's :file:`board.cmake`.
959 .. code-block:: yaml
962 - name: custom
965 This tells Twister that the board is using a custom emulator called ``<name_of_emu_binary>``,
970 .. code-block:: cmake
977 .. code-block:: cmake
992 twister supports running most of the tests on real devices and produces
999 To use this feature on a single connected device, run twister with
1004 .. group-tab:: Linux
1006 .. code-block:: bash
1008 scripts/twister --device-testing --device-serial /dev/ttyACM0 \
1009 --device-serial-baud 115200 -p frdm_k64f -T tests/kernel
1011 .. group-tab:: Windows
1013 .. code-block:: bat
1015 python .\scripts\twister --device-testing --device-serial COM1 \
1016 --device-serial-baud 115200 -p frdm_k64f -T tests/kernel
1018 The ``--device-serial`` option denotes the serial device the board is connected to.
1019 This needs to be accessible by the user running twister. You can run this on
1020 only one board at a time, specified using the ``--platform`` option.
1022 The ``--device-serial-baud`` option is only needed if your device does not run at
1025 To support devices without a physical serial port, use the ``--device-serial-pty``
1027 In this case you can run twister with the following options:
1031 .. group-tab:: Linux
1033 .. code-block:: bash
1035 scripts/twister --device-testing --device-serial-pty "script.py" \
1036 -p intel_adsp/cavs25 -T tests/kernel
1038 .. group-tab:: Windows
1044 The script is user-defined and handles delivering the messages which can be
1045 used by twister to determine the test execution status.
1047 The ``--device-flash-timeout`` option allows to set explicit timeout on the
1051 The ``--device-flash-with-test`` option indicates that on the platform
1065 .. group-tab:: Linux
1067 .. code-block:: bash
1069 ./scripts/twister --generate-hardware-map map.yml
1071 .. group-tab:: Windows
1073 .. code-block:: bat
1075 python .\scripts\twister --generate-hardware-map map.yml
1082 .. group-tab:: Linux
1084 .. code-block:: yaml
1086 - connected: true
1089 product: DAPLink CMSIS-DAP
1092 - connected: true
1095 product: J-Link
1099 .. group-tab:: Windows
1101 .. code-block:: yaml
1103 - connected: true
1109 - connected: true
1124 .. group-tab:: Linux
1126 .. code-block:: yaml
1128 - connected: true
1131 product: DAPLink CMSIS-DAP
1135 - connected: true
1138 product: J-Link
1143 .. group-tab:: Windows
1145 .. code-block:: yaml
1147 - connected: true
1150 product: DAPLink CMSIS-DAP
1154 - connected: true
1157 product: J-Link
1172 .. group-tab:: Linux
1174 .. code-block:: bash
1176 ./scripts/twister --device-testing --hardware-map map.yml -T samples/hello_world/
1178 .. group-tab:: Windows
1180 .. code-block:: bat
1182 python .\scripts\twister --device-testing --hardware-map map.yml -T samples\hello_world
1184 The above command will result in twister building tests for the platforms
1194 Hardware map allows to set ``--device-flash-timeout`` and ``--device-flash-with-test``
1195 command line options as ``flash-timeout`` and ``flash-with-test`` fields respectively.
1198 Serial PTY support using ``--device-serial-pty`` can also be used in the
1201 .. code-block:: yaml
1203 - connected: true
1210 - --remote-host=remote_host_ip_addr
1211 - --key=/path/to/key.pem
1216 work. It is equivalent to following west and twister commands.
1220 .. group-tab:: Linux
1222 .. code-block:: bash
1224 west flash --remote-host remote_host_ip_addr --key /path/to/key.pem
1226 twister -p intel_adsp/cavs25 --device-testing --device-serial-pty script.py
1227 --west-flash="--remote-host=remote_host_ip_addr,--key=/path/to/key.pem"
1229 .. group-tab:: Windows
1237 For serial PTY, the "--generate-hardware-map" option cannot scan it out
1253 .. code-block:: yaml
1255 - connected: true
1257 - gpio_loopback
1260 product: DAPLink CMSIS-DAP
1264 When running ``twister`` with ``--device-testing``, the configured fixture
1269 :figclass: align-center
1271 Fixtures can also be provided via twister command option ``--fixture``, this option
1274 current twister command can run those test scenarios which request the same fixtures.
1287 .. code-block:: yaml
1289 - connected: false
1291 - gpio_loopback
1299 product: J-Link
1306 When (re-)generated the hardware map file will contain an ``id`` keyword
1307 that serves as the argument to ``--board-id`` when flashing. In some
1309 using an external J-Link probe. The ``probe_id`` keyword overrides the
1312 .. code-block:: yaml
1314 - connected: false
1318 product: DAPLink CMSIS-DAP
1328 re-configuring the hardware map file for each variant. For example:
1330 .. code-block:: yaml
1332 - connected: true
1335 - nrf5340dk/nrf5340/cpuapp
1336 - nrf5340dk/nrf5340/cpuapp/ns
1337 product: J-Link
1344 Twister allows user to provide configuration files defining a list of tests or
1351 ``--quarantine-list <PATH_TO_QUARANTINE_YAML>`` to a twister call.
1354 ``--quarantine-verify`` to the above argument. This will make twister skip all tests
1370 .. code-block:: yaml
1372 - scenarios:
1373 - sample.basic.helloworld
1374 comment: "Link to the issue: https://github.com/zephyrproject-rtos/zephyr/pull/33287"
1376 - scenarios:
1377 - kernel.common
1378 - kernel.common.(misra|tls)
1379 - kernel.common.nano64
1381 - .*_cortex_.*
1382 - native_sim
1386 .. code-block:: yaml
1388 - platforms:
1389 - qemu_x86
1397 A test configuration can be used to customize various aspects of twister
1404 twister it is then possible to select a level and just execute the tests
1419 - Ability to ignore default platforms as defined in board definitions
1422 - Option to specify your own list of default platforms overriding what
1424 - Ability to override ``build_on_all`` options used in some test scenarios.
1427 - Ignore some logic in twister to expand platform coverage in cases where
1434 The following options control platform filtering in twister:
1436 - ``override_default_platforms``: override default key a platform sets in board
1440 - ``increased_platform_scope``: This option is set to True by default, when
1441 disabled, twister will not increase platform coverage automatically and will
1443 - ``default_platforms``: A list of additional default platforms to add. This list
1449 .. code-block:: yaml
1455 - qemu_x86
1471 .. code-block:: yaml
1474 - name: my-test-level
1478 - kernel.threads.*
1479 - kernel.timer.behavior
1480 - arch.interrupt
1481 - boards.*
1491 .. code-block:: yaml
1496 - frdm_k64f
1498 - name: smoke
1501 - name: unit
1504 - name: integration
1507 - name: acceptance
1510 - name: system
1513 - name: regression
1523 .. group-tab:: Linux
1525 .. code-block:: bash
1527 scripts/twister --test-config=<path to>/test_config.yaml
1528 -T tests --level="smoke"
1537 the seed to the random number generator by providing ``-seed=value`` as an
1538 argument to twister. See :ref:`Shuffling Test Sequence <ztest_shuffle>` for more
1545 Robot files allow you to express interactive test scenarios in human-readable text format and execu…
1548 To execute a Robot test suite with twister, run the following command:
1552 .. group-tab:: Linux
1554 .. code-block:: bash
1556 …$ ./scripts/twister --platform hifive1 --test samples/subsys/shell/shell_module/sample.shell.shell…
1558 .. group-tab:: Windows
1560 .. code-block:: bat
1562 …python .\scripts\twister --platform hifive1 --test samples/subsys/shell/shell_module/sample.shell.…
1572 …t test suite files, as an external Python library or, like Renode does it, dynamically via XML-RPC.
1573 …ramework.org/robotframework/latest/RobotFrameworkUserGuide.html#extending-robot-framework>`_ secti…
1580 .. code-block:: bash
1582 $ twister -p qemu_riscv32 -s tests/kernel/interrupt/arch.shared_interrupt