Lines Matching +full:twister +full:- +full:out
3 Test Runner (Twister)
6 Twister scans for the set of test applications in the git repository
14 Because of the limited test execution coverage, twister
20 When using (at least) one ``-v`` option, twister's console output
24 of a test is likewise reported in the ``twister.json`` and other report files.
25 There are a few reasons why twister only builds a test and doesn't run it:
27 - The test is marked as ``build_only: true`` in its ``.yaml``
29 - The test configuration has defined a ``harness`` but you don't have
31 - The target device is not connected and not available for flashing
32 - You or some higher level automation invoked twister with
33 ``--build-only``.
39 .. group-tab:: Linux
41 .. code-block:: bash
43 $ source zephyr-env.sh
44 $ ./scripts/twister
46 .. group-tab:: Windows
48 .. code-block:: bat
50 zephyr-env.cmd
51 python .\scripts\twister
59 .. group-tab:: Linux
61 .. code-block:: bash
63 $ ./scripts/twister --all --enable-slow
65 .. group-tab:: Windows
67 .. code-block:: bat
69 python .\scripts\twister --all --enable-slow
75 the ``--platform`` option, it is a platform filter for testing, with this
78 you can use ``--platform board@revision`` to test on a specific revision.
80 The list of command line options supported by twister can be viewed using:
84 .. group-tab:: Linux
86 .. code-block:: bash
88 $ ./scripts/twister --help
90 .. group-tab:: Windows
92 .. code-block:: bat
94 python .\scripts\twister --help
110 .. code-block:: yaml
113 name: NXP FRDM-K64F
117 - zephyr
118 - gnuarmemb
120 - arduino_gpio
121 - arduino_i2c
122 - netif:eth
123 - adc
124 - i2c
125 - nvs
126 - spi
127 - gpio
128 - usb_device
129 - watchdog
130 - can
131 - pwm
141 .. code-block:: console
144 west build -b reel_board
146 cmake -DBOARD=reel_board ..
155 .. code-block:: yaml
158 - name: qemu
159 - name: armfvp
161 - name: custom
165 simulation can be selected with ``--simulation <simulation_name>``.
185 .. code-block:: yaml
188 - pci
193 .. code-block:: yaml
196 - netif:eth
197 - sensor:bmi16
212 covered when invoking the simplified twister without any additional
225 power-efficient but slow CPU or simulation platform which can perform instruction accurate
229 A list of environment variables. Twister will check if all these environment variables are set,
232 presence to twister using these environment variables.
247 :alt: Twister and a Test application project.
248 :figclass: align-center
250 Twister and a Test application project.
297 The ``--no-detailed-test-id`` command line option modifies the above rules in this way:
300 Its Application Project path can be found in ``twister.json`` report as ``path:`` property.
303 must be unique for the Twister execution scope.
313 .. code-block:: yaml
332 .. code-block:: yaml
367 Don't run this test scenario unless ``--enable-slow`` or ``--enable-slow-only`` was
368 passed in on the command line. Intended for time-consuming test scenarios that
379 .. code-block:: yaml
388 - arch:x86:CONFIG_ADC_ASYNC=y
389 - platform:qemu_x86:CONFIG_DEBUG=y
390 - platform:mimxrt1060_evk:SHIELD=rk043fn66hs_ctg
391 - simulation:qemu:CONFIG_MPU=y
397 .. code-block:: yaml
406 - CONFIG_ADC_ASYNC=y
411 .. code-block:: yaml
420 - arch:x86:CONFIG_ADC_ASYNC=y
421 - platform:qemu_x86:CONFIG_DEBUG=y
425 If true, twister will not try to run the test even if the test is runnable
447 test will be selectable using the command line option ``--level <level name>``
487 This option limits the scope to the listed platforms when twister is
488 invoked with the ``--integration`` option. Use this instead of
497 test scenario -> platforms1 -> toolchain1
498 test scenario -> platforms2 -> toolchain1
502 twister run, it is possible to expand the test configurations to include
507 .. code-block:: yaml
510 - toolchain1
511 - toolchain2
515 test scenario -> platforms1 -> toolchain1
516 test scenario -> platforms1 -> toolchain2
517 test scenario -> platforms2 -> toolchain1
518 test scenario -> platforms2 -> toolchain2
524 ``--integration`` option.
530 When computing sizes, twister will report errors if it finds
538 the images onto the target. The ``--erase`` option of west flash is
543 A harness keyword in the ``testcase.yaml`` file identifies a Twister
545 Twister and implemented by Twister, some harnesses are defined as
549 Twister to be able to evaluate if a test passes criteria. For example, a
551 verdict on whether a test has passed or failed, however, Twister lack this
556 - ztest
557 - test
558 - console
559 - pytest
560 - gtest
561 - robot
562 - ctest
563 - shell
577 .. code-block:: yaml
580 - arch
581 - simulation
599 board(s) out of multiple boards that fulfill the dependency in an
609 .. code-block:: yaml
619 .. code-block:: yaml
626 - [robot file path 1]
627 - [robot file path 2]
628 - [robot file path n]
632 .. code-block:: yaml
640 - --exclude tag
641 - --stop-on-error
647 .. code-block:: none
655 Twister will first evaluate the expression to find if a "limited" cmake call, i.e. using package_helper cmake script,
663 .. code-block:: antlr
687 if the symbol is defined to a non-empty string.
694 * all comparison operators (non-associative)
699 .. code-block:: none
705 .. code-block:: none
713 .. code-block:: none
720 :ref:`Snippets <snippets>` are supported in twister for test scenarios that
721 require them. As with normal applications, twister supports using the base
729 .. code-block:: yaml
734 - cdc-acm-console
735 - user-snippet-example
739 running with ``-v`` or examining the discard report
745 To update this, pass the ``--all --release`` options.
764 - keyboard
765 - net
766 - bluetooth
770 .. code-block:: yaml
781 - "Temperature:(.*)C"
782 - "Relative Humidity:(.*)%"
794 ``ctest_args: [‘--repeat until-pass:5’]``. Note that
795 ``--ctest-args`` can be passed multiple times to pass several arguments
816 ``pytest``. After the pytest run is finished, Twister will check if
820 .. code-block:: yaml
824 - "pytest/test_shell_help.py"
825 - "../shell/pytest/test_shell.py"
826 - "/tmp/test_shell.py"
827 - "~/tmp/test_shell.py"
828 - "$ZEPHYR_BASE/samples/subsys/testsuite/pytest/shell/pytest/test_shell.py"
829 - "pytest/test_shell_help.py::test_shell2_sample" # select pytest subtest
830 - "pytest/test_shell_help.py::test_shell2_sample[param_a]" # select pytest parametrized subtest
836 ``pytest_args: [‘-k=test_method’, ‘--log-level=DEBUG’]``. Note that
837 ``--pytest-args`` can be passed multiple times to pass several arguments
852 .. code-block:: yaml
860 - pytest_dir1
861 - $ENV_VAR/samples/test/pytest_dir2
865 - pytest/test_file_1.py
866 - test_file_2.py::test_A
867 - test_file_2.py::test_B[param_a]
872 The ``console`` harness tells Twister to parse a test's text output for a
893 of the test suite object in ``twister.json``.
906 .. code-block:: yaml
910 - "(?P<metric>.*):(?P<cycles>.*) cycles, (?P<nanoseconds>.*) ns"
922 into ``twister.json`` as nested ``recording`` object properties.
924 as-is.
932 .. code-block:: yaml
940 .. code-block:: none
944 will be reported in ``twister.json`` as:
946 .. code-block:: json
973 Harness ``bsim`` is implemented in limited way - it helps only to copy the
992 framework and the pytest harness of twister.
1000 .. code-block:: yaml
1004 - command: "kernel cycles"
1006 - command: "kernel version"
1008 - command: "kernel sleep 100"
1018 .. code-block:: yaml
1020 - command: "mpu mtest 1"
1022 - command: "mpu mtest 2"
1033 One of the key features of Twister is its ability to decide on which platforms a given
1034 test scenario should run. This behavior has its roots in Twister being developed as
1036 tests, the testing tools should be able to adapt the scope and select/filter out what
1039 Twister always prepares an initial list of platforms in scope for a given test,
1042 (e.g. minimum ram) are filtered out from the scope.
1043 Using ``--force-platform`` allows to override filtering caused by ``platform_allow``,
1049 * ``-p/--platform <platform_name>`` (can be used multiple times): only platforms
1051 * ``-l/--all``: all available platforms;
1052 * ``-G/--integration``: all platforms from an ``integration_platforms`` list in
1057 *"Scope presumption"*: A list of Twister's :ref:`default platforms <twister_default_testing_board>`
1070 * ``--timeout-multiplier`` twister option which can be used to adjust timeouts in exact twister run.
1082 be activated using the ``--integration`` option of twister and narrows down
1091 Apart from the already supported QEMU and other simulated environments, Twister
1092 supports running any out-of-tree custom emulator defined in the board's :file:`board.cmake`.
1096 .. code-block:: yaml
1099 - name: custom
1102 This tells Twister that the board is using a custom emulator called ``<name_of_emu_binary>``,
1107 .. code-block:: cmake
1114 .. code-block:: cmake
1129 twister supports running most of the tests on real devices and produces
1136 To use this feature on a single connected device, run twister with
1141 .. group-tab:: Linux
1143 .. code-block:: bash
1145 scripts/twister --device-testing --device-serial /dev/ttyACM0 \
1146 --device-serial-baud 115200 -p frdm_k64f -T tests/kernel
1148 .. group-tab:: Windows
1150 .. code-block:: bat
1152 python .\scripts\twister --device-testing --device-serial COM1 \
1153 --device-serial-baud 115200 -p frdm_k64f -T tests/kernel
1155 The ``--device-serial`` option denotes the serial device the board is connected to.
1156 This needs to be accessible by the user running twister. You can run this on
1157 only one board at a time, specified using the ``--platform`` option.
1159 The ``--device-serial-baud`` option is only needed if your device does not run at
1162 To support devices without a physical serial port, use the ``--device-serial-pty``
1164 In this case you can run twister with the following options:
1168 .. group-tab:: Linux
1170 .. code-block:: bash
1172 scripts/twister --device-testing --device-serial-pty "script.py" \
1173 -p intel_adsp/cavs25 -T tests/kernel
1175 .. group-tab:: Windows
1181 The script is user-defined and handles delivering the messages which can be
1182 used by twister to determine the test execution status.
1184 The ``--device-flash-timeout`` option allows to set explicit timeout on the
1188 The ``--device-flash-with-test`` option indicates that on the platform
1202 .. group-tab:: Linux
1204 .. code-block:: bash
1206 ./scripts/twister --generate-hardware-map map.yml
1208 .. group-tab:: Windows
1210 .. code-block:: bat
1212 python .\scripts\twister --generate-hardware-map map.yml
1219 .. group-tab:: Linux
1221 .. code-block:: yaml
1223 - connected: true
1226 product: DAPLink CMSIS-DAP
1229 - connected: true
1232 product: J-Link
1236 .. group-tab:: Windows
1238 .. code-block:: yaml
1240 - connected: true
1246 - connected: true
1261 .. group-tab:: Linux
1263 .. code-block:: yaml
1265 - connected: true
1268 product: DAPLink CMSIS-DAP
1272 - connected: true
1275 product: J-Link
1280 .. group-tab:: Windows
1282 .. code-block:: yaml
1284 - connected: true
1287 product: DAPLink CMSIS-DAP
1291 - connected: true
1294 product: J-Link
1309 .. group-tab:: Linux
1311 .. code-block:: bash
1313 ./scripts/twister --device-testing --hardware-map map.yml -T samples/hello_world/
1315 .. group-tab:: Windows
1317 .. code-block:: bat
1319 python .\scripts\twister --device-testing --hardware-map map.yml -T samples\hello_world
1321 The above command will result in twister building tests for the platforms
1331 Hardware map allows to set ``--device-flash-timeout`` and ``--device-flash-with-test``
1332 command line options as ``flash-timeout`` and ``flash-with-test`` fields respectively.
1335 Serial PTY support using ``--device-serial-pty`` can also be used in the
1338 .. code-block:: yaml
1340 - connected: true
1347 - --remote-host=remote_host_ip_addr
1348 - --key=/path/to/key.pem
1353 work. It is equivalent to following west and twister commands.
1357 .. group-tab:: Linux
1359 .. code-block:: bash
1361 west flash --remote-host remote_host_ip_addr --key /path/to/key.pem
1363 twister -p intel_adsp/cavs25 --device-testing --device-serial-pty script.py
1364 --west-flash="--remote-host=remote_host_ip_addr,--key=/path/to/key.pem"
1366 .. group-tab:: Windows
1374 For serial PTY, the "--generate-hardware-map" option cannot scan it out
1390 .. code-block:: yaml
1392 - connected: true
1394 - gpio_loopback
1397 product: DAPLink CMSIS-DAP
1401 When running ``twister`` with ``--device-testing``, the configured fixture
1406 :figclass: align-center
1408 Fixtures can also be provided via twister command option ``--fixture``, this option
1411 current twister command can run those test scenarios which request the same fixtures.
1424 .. code-block:: yaml
1426 - connected: false
1428 - gpio_loopback
1436 product: J-Link
1443 When (re-)generated the hardware map file will contain an ``id`` keyword
1444 that serves as the argument to ``--board-id`` when flashing. In some
1446 using an external J-Link probe. The ``probe_id`` keyword overrides the
1449 .. code-block:: yaml
1451 - connected: false
1455 product: DAPLink CMSIS-DAP
1465 re-configuring the hardware map file for each variant. For example:
1467 .. code-block:: yaml
1469 - connected: true
1472 - nrf5340dk/nrf5340/cpuapp
1473 - nrf5340dk/nrf5340/cpuapp/ns
1474 product: J-Link
1481 Twister allows user to provide configuration files defining a list of tests or
1488 ``--quarantine-list <PATH_TO_QUARANTINE_YAML>`` to a twister call.
1491 ``--quarantine-verify`` to the above argument. This will make twister skip all tests
1507 .. code-block:: yaml
1509 - scenarios:
1510 - sample.basic.helloworld
1511 comment: "Link to the issue: https://github.com/zephyrproject-rtos/zephyr/pull/33287"
1513 - scenarios:
1514 - kernel.common
1515 - kernel.common.(misra|tls)
1516 - kernel.common.nano64
1518 - .*_cortex_.*
1519 - native_sim
1523 .. code-block:: yaml
1525 - platforms:
1526 - qemu_x86
1534 A test configuration can be used to customize various aspects of twister
1541 twister it is then possible to select a level and just execute the tests
1556 - Ability to ignore default platforms as defined in board definitions
1559 - Option to specify your own list of default platforms overriding what
1561 - Ability to override ``build_on_all`` options used in some test scenarios.
1564 - Ignore some logic in twister to expand platform coverage in cases where
1571 The following options control platform filtering in twister:
1573 - ``override_default_platforms``: override default key a platform sets in board
1577 - ``increased_platform_scope``: This option is set to True by default, when
1578 disabled, twister will not increase platform coverage automatically and will
1580 - ``default_platforms``: A list of additional default platforms to add. This list
1586 .. code-block:: yaml
1592 - qemu_x86
1608 .. code-block:: yaml
1611 - name: my-test-level
1615 - kernel.threads.*
1616 - kernel.timer.behavior
1617 - arch.interrupt
1618 - boards.*
1628 .. code-block:: yaml
1633 - frdm_k64f
1635 - name: smoke
1638 - name: unit
1641 - name: integration
1644 - name: acceptance
1647 - name: system
1650 - name: regression
1660 .. group-tab:: Linux
1662 .. code-block:: bash
1664 scripts/twister --test-config=<path to>/test_config.yaml
1665 -T tests --level="smoke"
1674 the seed to the random number generator by providing ``-seed=value`` as an
1675 argument to twister. See :ref:`Shuffling Test Sequence <ztest_shuffle>` for more
1682 Robot files allow you to express interactive test scenarios in human-readable text format and execute them in simulation or against hardware.
1685 To execute a Robot test suite with twister, run the following command:
1689 .. group-tab:: Linux
1691 .. code-block:: bash
1693 $ ./scripts/twister --platform hifive1 --test samples/subsys/shell/shell_module/sample.shell.shell_module.robot
1695 .. group-tab:: Windows
1697 .. code-block:: bat
1699 python .\scripts\twister --platform hifive1 --test samples/subsys/shell/shell_module/sample.shell.shell_module.robot
1709 It's possible to extend the framework by adding new keywords expressed directly in Robot test suite files, as an external Python library or, like Renode does it, dynamically via XML-RPC.
1710 For details see the `extending Robot Framework <https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#extending-robot-framework>`_ section in the official Robot documentation.
1717 .. code-block:: bash
1719 $ twister -p qemu_riscv32 -s tests/kernel/interrupt/arch.shared_interrupt