Lines Matching +full:twister +full:- +full:tests
3 Test Runner (Twister)
6 Twister scans for the set of test applications in the git repository
14 Because of the limited test execution coverage, twister
17 tests for different boards and different configurations to help keep the
20 When using (at least) one ``-v`` option, twister's console output
24 of a test is likewise reported in the ``twister.json`` and other report files.
25 There are a few reasons why twister only builds a test and doesn't run it:
27 - The test is marked as ``build_only: true`` in its ``.yaml``
29 - The test configuration has defined a ``harness`` but you don't have
31 - The target device is not connected and not available for flashing
32 - You or some higher level automation invoked twister with
33 ``--build-only``.
39 .. group-tab:: Linux
41 .. code-block:: bash
43 $ source zephyr-env.sh
44 $ ./scripts/twister
46 .. group-tab:: Windows
48 .. code-block:: bat
50 zephyr-env.cmd
51 python .\scripts\twister
55 you can build and run all possible tests using the following options:
59 .. group-tab:: Linux
61 .. code-block:: bash
63 $ ./scripts/twister --all --enable-slow
65 .. group-tab:: Windows
67 .. code-block:: bat
69 python .\scripts\twister --all --enable-slow
71 This will build for all available boards and run all applicable tests in
74 If you want to run tests on one or more specific platforms, you can use
75 the ``--platform`` option, it is a platform filter for testing, with this
78 you can use ``--platform board@revision`` to test on a specific revision.
80 The list of command line options supported by twister can be viewed using:
84 .. group-tab:: Linux
86 .. code-block:: bash
88 $ ./scripts/twister --help
90 .. group-tab:: Windows
92 .. code-block:: bat
94 python .\scripts\twister --help
100 To build tests for a specific board and to execute some of the tests on real
110 .. code-block:: yaml
113 name: NXP FRDM-K64F
117 - zephyr
118 - gnuarmemb
120 - arduino_gpio
121 - arduino_i2c
122 - netif:eth
123 - adc
124 - i2c
125 - nvs
126 - spi
127 - gpio
128 - usb_device
129 - watchdog
130 - can
131 - pwm
141 .. code-block:: console
144 west build -b reel_board
146 cmake -DBOARD=reel_board ..
155 .. code-block:: yaml
158 - name: qemu
159 - name: armfvp
161 - name: custom
164 By default, tests will be executed using the first entry in the simulation array. Another
165 simulation can be selected with ``--simulation <simulation_name>``.
167 tests will be built only.
168 If it is not set and the required simulator is not available the tests will fail to run.
185 .. code-block:: yaml
188 - pci
193 .. code-block:: yaml
196 - netif:eth
197 - sensor:bmi16
212 covered when invoking the simplified twister without any additional
215 Do not attempt to build (and therefore run) tests marked with this list of
218 Only execute tests with this list of tags on a specific platform.
225 power-efficient but slow CPU or simulation platform which can perform instruction accurate
229 A list of environment variables. Twister will check if all these environment variables are set,
232 presence to twister using these environment variables.
236 Tests section in Test Runner (Twister)
239 Tests are detected by the presence of a ``testcase.yaml`` or a ``sample.yaml``
241 configuration file may contain one or more entries in the ``tests:`` section each
247 :alt: Twister and a Test application project.
248 :figclass: align-center
250 Twister and a Test application project.
277 * ``kernel.semaphore``: For general semaphore tests
286 * **Ztest tests**:
291 * **Standalone tests and samples**:
297 The ``--no-detailed-test-id`` command line option modifies the above rules in this way:
300 Its Application Project path can be found in ``twister.json`` report as ``path:`` property.
303 must be unique for the Twister execution scope.
313 .. code-block:: yaml
315 tests:
329 A sample with tests will have the same structure with additional information
332 .. code-block:: yaml
337 tests:
340 tags: tests
346 tags: tests
349 A Test Scenario entry in the ``tests:`` YAML dictionary has its Test Scenario
360 of this script can filter the set of tests to run based on tag.
363 skip test scenario unconditionally. This can be used for broken tests for
367 Don't run this test scenario unless ``--enable-slow`` or ``--enable-slow-only`` was
368 passed in on the command line. Intended for time-consuming test scenarios that
379 .. code-block:: yaml
383 tests:
388 - arch:x86:CONFIG_ADC_ASYNC=y
389 - platform:qemu_x86:CONFIG_DEBUG=y
390 - platform:mimxrt1060_evk:SHIELD=rk043fn66hs_ctg
391 - simulation:qemu:CONFIG_MPU=y
397 .. code-block:: yaml
401 tests:
406 - CONFIG_ADC_ASYNC=y
411 .. code-block:: yaml
415 tests:
420 - arch:x86:CONFIG_ADC_ASYNC=y
421 - platform:qemu_x86:CONFIG_DEBUG=y
425 If true, twister will not try to run the test even if the test is runnable
428 This keyword is reserved for tests that are used to test if some code
439 used in CI for increased coverage. Do not use this flag in new tests.
447 test will be selectable using the command line option ``--level <level name>``
487 This option limits the scope to the listed platforms when twister is
488 invoked with the ``--integration`` option. Use this instead of
497 test scenario -> platforms1 -> toolchain1
498 test scenario -> platforms2 -> toolchain1
502 twister run, it is possible to expand the test configurations to include
503 additional tests for each toolchain. For example, if a platform supports
507 .. code-block:: yaml
510 - toolchain1
511 - toolchain2
515 test scenario -> platforms1 -> toolchain1
516 test scenario -> platforms1 -> toolchain2
517 test scenario -> platforms2 -> toolchain1
518 test scenario -> platforms2 -> toolchain2
524 ``--integration`` option.
530 When computing sizes, twister will report errors if it finds
536 generated devicetree and Kconfig will be used for filtering tests.
538 the images onto the target. The ``--erase`` option of west flash is
540 in tests requiring sysbuild support being skipped.
543 A harness keyword in the ``testcase.yaml`` file identifies a Twister
545 Twister and implemented by Twister, some harnesses are defined as
549 Twister to be able to evaluate if a test passes criteria. For example, a
550 keyboard harness is set on tests that require keyboard interaction to reach
551 verdict on whether a test has passed or failed, however, Twister lack this
556 - ztest
557 - test
558 - console
559 - pytest
560 - gtest
561 - robot
562 - ctest
563 - shell
571 tests and code as passing. The platform_key attribute enables doing just
577 .. code-block:: yaml
580 - arch
581 - simulation
586 building and running SPI tests once for each unique IP block.
604 be unique across all tests in the test suite.
609 .. code-block:: yaml
611 tests:
619 .. code-block:: yaml
621 tests:
626 - [robot file path 1]
627 - [robot file path 2]
628 - [robot file path n]
632 .. code-block:: yaml
634 tests:
640 - --exclude tag
641 - --stop-on-error
647 .. code-block:: none
655 …Twister will first evaluate the expression to find if a "limited" cmake call, i.e. using package_h…
663 .. code-block:: antlr
687 if the symbol is defined to a non-empty string.
694 * all comparison operators (non-associative)
699 .. code-block:: none
705 .. code-block:: none
713 .. code-block:: none
720 :ref:`Snippets <snippets>` are supported in twister for test scenarios that
721 require them. As with normal applications, twister supports using the base
723 snippets. Listed snippets will filter supported tests for boards (snippets
729 .. code-block:: yaml
731 tests:
734 - cdc-acm-console
735 - user-snippet-example
739 running with ``-v`` or examining the discard report
745 To update this, pass the ``--all --release`` options.
764 - keyboard
765 - net
766 - bluetooth
770 .. code-block:: yaml
781 - "Temperature:(.*)C"
782 - "Relative Humidity:(.*)%"
784 tests:
794 ``ctest_args: [‘--repeat until-pass:5’]``. Note that
795 ``--ctest-args`` can be passed multiple times to pass several arguments
802 Use ``gtest`` harness if you've already got tests written in the gTest
816 ``pytest``. After the pytest run is finished, Twister will check if
820 .. code-block:: yaml
824 - "pytest/test_shell_help.py"
825 - "../shell/pytest/test_shell.py"
826 - "/tmp/test_shell.py"
827 - "~/tmp/test_shell.py"
828 - "$ZEPHYR_BASE/samples/subsys/testsuite/pytest/shell/pytest/test_shell.py"
829 - "pytest/test_shell_help.py::test_shell2_sample" # select pytest subtest
830 … - "pytest/test_shell_help.py::test_shell2_sample[param_a]" # select pytest parametrized subtest
836 ``pytest_args: [‘-k=test_method’, ‘--log-level=DEBUG’]``. Note that
837 ``--pytest-args`` can be passed multiple times to pass several arguments
852 .. code-block:: yaml
856 tests:
860 - pytest_dir1
861 - $ENV_VAR/samples/test/pytest_dir2
865 - pytest/test_file_1.py
866 - test_file_2.py::test_A
867 - test_file_2.py::test_B[param_a]
872 The ``console`` harness tells Twister to parse a test's text output for a
893 of the test suite object in ``twister.json``.
906 .. code-block:: yaml
910 - "(?P<metric>.*):(?P<cycles>.*) cycles, (?P<nanoseconds>.*) ns"
922 into ``twister.json`` as nested ``recording`` object properties.
924 as-is.
932 .. code-block:: yaml
940 .. code-block:: none
944 will be reported in ``twister.json`` as:
946 .. code-block:: json
973 Harness ``bsim`` is implemented in limited way - it helps only to copy the
977 This action is useful to allow BabbleSim's tests to directly run after.
992 framework and the pytest harness of twister.
1000 .. code-block:: yaml
1004 - command: "kernel cycles"
1006 - command: "kernel version"
1008 - command: "kernel sleep 100"
1018 .. code-block:: yaml
1020 - command: "mpu mtest 1"
1022 - command: "mpu mtest 2"
1033 One of the key features of Twister is its ability to decide on which platforms a given
1034 test scenario should run. This behavior has its roots in Twister being developed as
1036 tests, the testing tools should be able to adapt the scope and select/filter out what
1039 Twister always prepares an initial list of platforms in scope for a given test,
1043 Using ``--force-platform`` allows to override filtering caused by ``platform_allow``,
1049 * ``-p/--platform <platform_name>`` (can be used multiple times): only platforms
1051 * ``-l/--all``: all available platforms;
1052 * ``-G/--integration``: all platforms from an ``integration_platforms`` list in
1057 *"Scope presumption"*: A list of Twister's :ref:`default platforms <twister_default_testing_board>`
1061 Managing tests timeouts
1064 There are several parameters which control tests timeouts on various levels:
1070 * ``--timeout-multiplier`` twister option which can be used to adjust timeouts in exact twister run.
1082 be activated using the ``--integration`` option of twister and narrows down
1083 the scope of builds and tests if applicable to platforms defined under the
1088 Running tests on custom emulator
1091 Apart from the already supported QEMU and other simulated environments, Twister
1092 supports running any out-of-tree custom emulator defined in the board's :file:`board.cmake`.
1096 .. code-block:: yaml
1099 - name: custom
1102 This tells Twister that the board is using a custom emulator called ``<name_of_emu_binary>``,
1107 .. code-block:: cmake
1114 .. code-block:: cmake
1125 Running Tests on Hardware
1128 Beside being able to run tests in QEMU and other simulated environments,
1129 twister supports running most of the tests on real devices and produces
1133 Executing tests on a single device
1136 To use this feature on a single connected device, run twister with
1141 .. group-tab:: Linux
1143 .. code-block:: bash
1145 scripts/twister --device-testing --device-serial /dev/ttyACM0 \
1146 --device-serial-baud 115200 -p frdm_k64f -T tests/kernel
1148 .. group-tab:: Windows
1150 .. code-block:: bat
1152 python .\scripts\twister --device-testing --device-serial COM1 \
1153 --device-serial-baud 115200 -p frdm_k64f -T tests/kernel
1155 The ``--device-serial`` option denotes the serial device the board is connected to.
1156 This needs to be accessible by the user running twister. You can run this on
1157 only one board at a time, specified using the ``--platform`` option.
1159 The ``--device-serial-baud`` option is only needed if your device does not run at
1162 To support devices without a physical serial port, use the ``--device-serial-pty``
1164 In this case you can run twister with the following options:
1168 .. group-tab:: Linux
1170 .. code-block:: bash
1172 scripts/twister --device-testing --device-serial-pty "script.py" \
1173 -p intel_adsp/cavs25 -T tests/kernel
1175 .. group-tab:: Windows
1181 The script is user-defined and handles delivering the messages which can be
1182 used by twister to determine the test execution status.
1184 The ``--device-flash-timeout`` option allows to set explicit timeout on the
1188 The ``--device-flash-with-test`` option indicates that on the platform
1192 Executing tests on multiple devices
1195 To build and execute tests on multiple devices connected to the host PC, a
1202 .. group-tab:: Linux
1204 .. code-block:: bash
1206 ./scripts/twister --generate-hardware-map map.yml
1208 .. group-tab:: Windows
1210 .. code-block:: bat
1212 python .\scripts\twister --generate-hardware-map map.yml
1219 .. group-tab:: Linux
1221 .. code-block:: yaml
1223 - connected: true
1226 product: DAPLink CMSIS-DAP
1229 - connected: true
1232 product: J-Link
1236 .. group-tab:: Windows
1238 .. code-block:: yaml
1240 - connected: true
1246 - connected: true
1261 .. group-tab:: Linux
1263 .. code-block:: yaml
1265 - connected: true
1268 product: DAPLink CMSIS-DAP
1272 - connected: true
1275 product: J-Link
1280 .. group-tab:: Windows
1282 .. code-block:: yaml
1284 - connected: true
1287 product: DAPLink CMSIS-DAP
1291 - connected: true
1294 product: J-Link
1305 With the hardware map ready, you can run any tests by pointing to the map
1309 .. group-tab:: Linux
1311 .. code-block:: bash
1313 ./scripts/twister --device-testing --hardware-map map.yml -T samples/hello_world/
1315 .. group-tab:: Windows
1317 .. code-block:: bat
1319 python .\scripts\twister --device-testing --hardware-map map.yml -T samples\hello_world
1321 The above command will result in twister building tests for the platforms
1322 defined in the hardware map and subsequently flashing and running the tests
1331 Hardware map allows to set ``--device-flash-timeout`` and ``--device-flash-with-test``
1332 command line options as ``flash-timeout`` and ``flash-with-test`` fields respectively.
1335 Serial PTY support using ``--device-serial-pty`` can also be used in the
1338 .. code-block:: yaml
1340 - connected: true
1347 - --remote-host=remote_host_ip_addr
1348 - --key=/path/to/key.pem
1353 work. It is equivalent to following west and twister commands.
1357 .. group-tab:: Linux
1359 .. code-block:: bash
1361 west flash --remote-host remote_host_ip_addr --key /path/to/key.pem
1363 twister -p intel_adsp/cavs25 --device-testing --device-serial-pty script.py
1364 --west-flash="--remote-host=remote_host_ip_addr,--key=/path/to/key.pem"
1366 .. group-tab:: Windows
1374 For serial PTY, the "--generate-hardware-map" option cannot scan it out
1382 Some tests require additional setup or special wiring specific to the test.
1383 Running the tests without this setup or test fixture may fail. A test scenario can
1390 .. code-block:: yaml
1392 - connected: true
1394 - gpio_loopback
1397 product: DAPLink CMSIS-DAP
1401 When running ``twister`` with ``--device-testing``, the configured fixture
1403 and these tests will be executed on the boards that provide this fixture.
1406 :figclass: align-center
1408 Fixtures can also be provided via twister command option ``--fixture``, this option
1411 current twister command can run those test scenarios which request the same fixtures.
1424 .. code-block:: yaml
1426 - connected: false
1428 - gpio_loopback
1436 product: J-Link
1443 When (re-)generated the hardware map file will contain an ``id`` keyword
1444 that serves as the argument to ``--board-id`` when flashing. In some
1446 using an external J-Link probe. The ``probe_id`` keyword overrides the
1449 .. code-block:: yaml
1451 - connected: false
1455 product: DAPLink CMSIS-DAP
1463 with names separated by spaces. This allows to run tests for
1465 re-configuring the hardware map file for each variant. For example:
1467 .. code-block:: yaml
1469 - connected: true
1472 - nrf5340dk/nrf5340/cpuapp
1473 - nrf5340dk/nrf5340/cpuapp/ns
1474 product: J-Link
1481 Twister allows user to provide configuration files defining a list of tests or
1482 platforms to be put under quarantine. Such tests will be skipped and marked
1485 of other tests (e.g. putting the physical board in a corrupted state).
1488 ``--quarantine-list <PATH_TO_QUARANTINE_YAML>`` to a twister call.
1490 The current status of tests on the quarantine list can also be verified by adding
1491 ``--quarantine-verify`` to the above argument. This will make twister skip all tests
1500 When quarantining a class of tests or many scenarios in a single testsuite or
1503 all kernel tests.
1507 .. code-block:: yaml
1509 - scenarios:
1510 - sample.basic.helloworld
1511 comment: "Link to the issue: https://github.com/zephyrproject-rtos/zephyr/pull/33287"
1513 - scenarios:
1514 - kernel.common
1515 - kernel.common.(misra|tls)
1516 - kernel.common.nano64
1518 - .*_cortex_.*
1519 - native_sim
1523 .. code-block:: yaml
1525 - platforms:
1526 - qemu_x86
1529 Additionally you can quarantine entire architectures or a specific simulator for executing tests.
1534 A test configuration can be used to customize various aspects of twister
1541 twister it is then possible to select a level and just execute the tests
1545 dependencies and additional inclusion of tests into a specific level if
1556 - Ability to ignore default platforms as defined in board definitions
1557 (Those are mostly emulation platforms used to run tests in upstream
1559 - Option to specify your own list of default platforms overriding what
1561 - Ability to override ``build_on_all`` options used in some test scenarios.
1562 This will treat tests or sample as any other just build for default
1564 - Ignore some logic in twister to expand platform coverage in cases where
1571 The following options control platform filtering in twister:
1573 - ``override_default_platforms``: override default key a platform sets in board
1577 - ``increased_platform_scope``: This option is set to True by default, when
1578 disabled, twister will not increase platform coverage automatically and will
1579 only build and run tests on the specified platforms.
1580 - ``default_platforms``: A list of additional default platforms to add. This list
1586 .. code-block:: yaml
1592 - qemu_x86
1599 additional inclusion of tests into a specific test level if the test itself
1608 .. code-block:: yaml
1611 - name: my-test-level
1615 - kernel.threads.*
1616 - kernel.timer.behavior
1617 - arch.interrupt
1618 - boards.*
1628 .. code-block:: yaml
1633 - frdm_k64f
1635 - name: smoke
1638 - name: unit
1641 - name: integration
1644 - name: acceptance
1647 - name: system
1650 - name: regression
1660 .. group-tab:: Linux
1662 .. code-block:: bash
1664 scripts/twister --test-config=<path to>/test_config.yaml
1665 -T tests --level="smoke"
1669 Running in Tests in Random Order
1672 run your tests in random order. This can be beneficial for identifying
1674 the seed to the random number generator by providing ``-seed=value`` as an
1675 argument to twister. See :ref:`Shuffling Test Sequence <ztest_shuffle>` for more
1678 Robot Framework Tests
1682 Robot files allow you to express interactive test scenarios in human-readable text format and execu…
1683 At this moment Zephyr integration supports running Robot tests in the `Renode <https://renode.io/>`…
1685 To execute a Robot test suite with twister, run the following command:
1689 .. group-tab:: Linux
1691 .. code-block:: bash
1693 …$ ./scripts/twister --platform hifive1 --test samples/subsys/shell/shell_module/sample.shell.shell…
1695 .. group-tab:: Windows
1697 .. code-block:: bat
1699 …python .\scripts\twister --platform hifive1 --test samples/subsys/shell/shell_module/sample.shell.…
1701 Writing Robot tests
1706 Information on writing and running Robot Framework tests in Renode can be found in `the testing sec…
1709 …t test suite files, as an external Python library or, like Renode does it, dynamically via XML-RPC.
1710 …ramework.org/robotframework/latest/RobotFrameworkUserGuide.html#extending-robot-framework>`_ secti…
1717 .. code-block:: bash
1719 $ twister -p qemu_riscv32 -s tests/kernel/interrupt/arch.shared_interrupt