Lines Matching +full:directory +full:- +full:to +full:- +full:scan
7 and attempts to execute them. By default, it tries to build each test
17 tests for different boards and different configurations to help keep the
20 When using (at least) one ``-v`` option, twister's console output
27 - The test is marked as ``build_only: true`` in its ``.yaml``
29 - The test configuration has defined a ``harness`` but you don't have
31 - The target device is not connected and not available for flashing
32 - You or some higher level automation invoked twister with
33 ``--build-only``.
35 To run the script in the local tree, follow the steps below:
39 .. group-tab:: Linux
41 .. code-block:: bash
43 $ source zephyr-env.sh
46 .. group-tab:: Windows
48 .. code-block:: bat
50 zephyr-env.cmd
59 .. group-tab:: Linux
61 .. code-block:: bash
63 $ ./scripts/twister --all --enable-slow
65 .. group-tab:: Windows
67 .. code-block:: bat
69 python .\scripts\twister --all --enable-slow
74 If you want to run tests on one or more specific platforms, you can use
75 the ``--platform`` option, it is a platform filter for testing, with this
78 you can use ``--platform board@revision`` to test on a specific revision.
84 .. group-tab:: Linux
86 .. code-block:: bash
88 $ ./scripts/twister --help
90 .. group-tab:: Windows
92 .. code-block:: bat
94 python .\scripts\twister --help
100 To build tests for a specific board and to execute some of the tests on real
102 is required which is generic enough to be used for other tasks that require a
106 The board metadata file is located in the board directory and is structured
110 .. code-block:: yaml
113 name: NXP FRDM-K64F
117 - zephyr
118 - gnuarmemb
120 - arduino_gpio
121 - arduino_i2c
122 - netif:eth
123 - adc
124 - i2c
125 - nvs
126 - spi
127 - gpio
128 - usb_device
129 - watchdog
130 - can
131 - pwm
141 .. code-block:: console
144 west build -b reel_board
146 cmake -DBOARD=reel_board ..
153 Simulator(s) used to simulate the platform, e.g. qemu.
155 .. code-block:: yaml
158 - name: qemu
159 - name: armfvp
161 - name: custom
165 simulation can be selected with ``--simulation <simulation_name>``.
168 If it is not set and the required simulator is not available the tests will fail to run.
176 Available RAM on the board (specified in KB). This is used to match test scenario
177 requirements. If not specified we default to 128KB.
179 Available FLASH on the board (specified in KB). This is used to match test scenario
180 requirements. If not specified we default to 512KB.
185 .. code-block:: yaml
188 - pci
193 .. code-block:: yaml
196 - netif:eth
197 - sensor:bmi16
199 A test scenario can depend on 'eth' to only test ethernet or on 'netif' to run
203 testing relating keywords to provide best coverage for the features of this
209 A list of custom binaries to be kept for device testing.
215 Do not attempt to build (and therefore run) tests marked with this list of
223 Multiply each test scenario timeout by specified ratio. This option allows to tune timeouts only
225 power-efficient but slow CPU or simulation platform which can perform instruction accurate
230 and otherwise skip this platform. This allows the user to define a platform which should be
231 used, for example, only if some required software or hardware is present, and to signal that
232 presence to twister using these environment variables.
240 files in the application's project directory. This test application
248 :figclass: align-center
259 A Test Suite is a collection of Test Cases which are intended to be used to test
260 a software program to ensure it meets certain requirements. The Test Cases in a
261 Test Suite are either related or meant to be executed together.
263 Test Scenario, Test Suite, and Test Case names must follow to these basic rules:
274 need to be unique.
297 The ``--no-detailed-test-id`` command line option modifies the above rules in this way:
306 Its parent Test Suite name equals to the corresponding Test Scenario identifier.
313 .. code-block:: yaml
330 related to the sample and what is being demonstrated:
332 .. code-block:: yaml
358 A set of string tags for the test scenario. Usually pertains to
360 of this script can filter the set of tests to run based on tag.
367 Don't run this test scenario unless ``--enable-slow`` or ``--enable-slow-only`` was
368 passed in on the command line. Intended for time-consuming test scenarios that
373 Extra arguments to pass to build tool when building or running the
376 Using namespacing, it is possible to apply extra_args only to some
379 .. code-block:: yaml
388 - arch:x86:CONFIG_ADC_ASYNC=y
389 - platform:qemu_x86:CONFIG_DEBUG=y
390 - platform:mimxrt1060_evk:SHIELD=rk043fn66hs_ctg
391 - simulation:qemu:CONFIG_MPU=y
394 Extra configuration options to be merged with a main prj.conf
397 .. code-block:: yaml
406 - CONFIG_ADC_ASYNC=y
408 Using namespacing, it is possible to apply a configuration only to some
411 .. code-block:: yaml
420 - arch:x86:CONFIG_ADC_ASYNC=y
421 - platform:qemu_x86:CONFIG_DEBUG=y
425 If true, twister will not try to run the test even if the test is runnable
428 This keyword is reserved for tests that are used to test if some code
429 actually builds. A ``build_only`` test is not designed to be run in any
433 This option is often used to test drivers and the fact that they are correctly
435 test shall not be used to verify the functionality of the driver.
438 If true, attempt to build test scenario on all available platforms. This is mostly
447 test will be selectable using the command line option ``--level <level name>``
450 estimated minimum amount of RAM in KB needed for this test to build and run. This is
454 estimated minimum amount of ROM in KB needed for this test to build and run. This is
460 Length of time to run test before automatically killing it.
461 Default to 60 seconds.
482 this option to limit testing or building in CI due to time or resource
487 This option limits the scope to the listed platforms when twister is
488 invoked with the ``--integration`` option. Use this instead of
489 platform_allow if the goal is to limit scope due to timing or
493 This option expands the scope to all the listed toolchains variants and
497 test scenario -> platforms1 -> toolchain1
498 test scenario -> platforms2 -> toolchain1
502 twister run, it is possible to expand the test configurations to include
507 .. code-block:: yaml
510 - toolchain1
511 - toolchain2
515 test scenario -> platforms1 -> toolchain1
516 test scenario -> platforms1 -> toolchain2
517 test scenario -> platforms2 -> toolchain1
518 test scenario -> platforms2 -> toolchain2
523 This functionality is evaluated always and is not limited to the
524 ``--integration`` option.
537 on device testing must use the hardware map, or west flash to load
538 the images onto the target. The ``--erase`` option of west flash is
544 harness needed to run a test successfully. A harness is a feature of
548 A harness can be seen as the handler that needs to be implemented in
549 Twister to be able to evaluate if a test passes criteria. For example, a
550 keyboard harness is set on tests that require keyboard interaction to reach
556 - ztest
557 - test
558 - console
559 - pytest
560 - gtest
561 - robot
562 - ctest
563 - shell
568 Often a test needs to only be built and run once to qualify as passing.
570 passing the test on a single platform for each arch is enough to qualify the
574 For example to key on (arch, simulation) to ensure a test is run once
577 .. code-block:: yaml
580 - arch
581 - simulation
583 Adding platform (board) attributes to include things such as soc name,
589 Extra harness configuration options to be used to select a board and/or
591 what features it supports. This option will enable the test to run on
598 specific test setup and board selection logic to pick the particular
603 Only one fixture can be defined per test scenario and the fixture name has to
609 .. code-block:: yaml
619 .. code-block:: yaml
626 - [robot file path 1]
627 - [robot file path 2]
628 - [robot file path n]
630 One or more options can be passed to robotframework.
632 .. code-block:: yaml
640 - --exclude tag
641 - --stop-on-error
647 .. code-block:: none
655 …Twister will first evaluate the expression to find if a "limited" cmake call, i.e. using package_h…
663 .. code-block:: antlr
686 For the case where ``expression ::= symbol``, it evaluates to ``true``
687 if the symbol is defined to a non-empty string.
689 Operator precedence, starting from lowest to highest:
694 * all comparison operators (non-associative)
699 .. code-block:: none
705 .. code-block:: none
713 .. code-block:: none
722 zephyr snippet directory and test application directory for finding
724 must be compatible with a board for the test to run on them, they are not
729 .. code-block:: yaml
734 - cdc-acm-console
735 - user-snippet-example
739 running with ``-v`` or examining the discard report
745 To update this, pass the ``--all --release`` options.
747 To load arguments from a file, add ``+`` before the file name, e.g.,
764 - keyboard
765 - net
766 - bluetooth
770 .. code-block:: yaml
781 - "Temperature:(.*)C"
782 - "Relative Humidity:(.*)%"
793 Specify a list of additional arguments to pass to ``ctest`` e.g.:
794 ``ctest_args: [‘--repeat until-pass:5’]``. Note that
795 ``--ctest-args`` can be passed multiple times to pass several arguments
796 to the ctest.
803 framework and do not wish to update them to zTest.
808 The :ref:`pytest harness <integration_with_pytest>` is used to execute pytest
809 test suites in the Zephyr test. The following options apply to the pytest harness:
814 Specify a list of pytest directories, files or subtests that need to be
815 executed when a test scenario begins to run. The default pytest directory is
817 the test scenario passed or failed according to the pytest report.
820 .. code-block:: yaml
824 - "pytest/test_shell_help.py"
825 - "../shell/pytest/test_shell.py"
826 - "/tmp/test_shell.py"
827 - "~/tmp/test_shell.py"
828 - "$ZEPHYR_BASE/samples/subsys/testsuite/pytest/shell/pytest/test_shell.py"
829 - "pytest/test_shell_help.py::test_shell2_sample" # select pytest subtest
830 … - "pytest/test_shell_help.py::test_shell2_sample[param_a]" # select pytest parametrized subtest
835 Specify a list of additional arguments to pass to ``pytest`` e.g.:
836 ``pytest_args: [‘-k=test_method’, ‘--log-level=DEBUG’]``. Note that
837 ``--pytest-args`` can be passed multiple times to pass several arguments
838 to the pytest.
844 If the scope is set to ``function``, DUT is launched for every test case
852 .. code-block:: yaml
860 - pytest_dir1
861 - $ENV_VAR/samples/test/pytest_dir2
865 - pytest/test_file_1.py
866 - test_file_2.py::test_A
867 - test_file_2.py::test_B[param_a]
872 The ``console`` harness tells Twister to parse a test's text output for a
878 Depends on the regex string to be matched
881 Strings with regular expressions to match with the test's output
882 to confirm the test runs as expected.
889 Regular expressions with named subgroups to match data fields found
892 directory ``recording.csv`` file as well as ``recording`` property
896 to each output line producing either several different records from
903 For example, to extract three data fields ``metric``, ``cycles``,
906 .. code-block:: yaml
910 - "(?P<metric>.*):(?P<cycles>.*) cycles, (?P<nanoseconds>.*) ns"
913 Allows to keep only one record in a test instance with all the data
916 It is possible for such multi value fields to have different number
924 as-is.
932 .. code-block:: yaml
938 when matched to a test log string:
940 .. code-block:: none
946 .. code-block:: json
961 The ``robot`` harness is used to execute Robot Framework test suites
965 Specify one or more paths to a file containing a Robot Framework test suite to be run.
968 One or more options to be send to robotframework.
973 Harness ``bsim`` is implemented in limited way - it helps only to copy the
974 final executable (``zephyr.exe``) from build directory to BabbleSim's
975 ``bin`` directory (``${BSIM_OUT_PATH}/bin``).
977 This action is useful to allow BabbleSim's tests to directly run after.
984 If provided, the executable filename when copying to BabbleSim's bin
985 directory, will be ``bs_<platform_name>_<bsim_exe_name>`` instead of the
991 The shell harness is used to execute shell commands and parse the output and utilizes the pytest
994 The following options apply to the shell harness:
997 Specify a list of shell commands to be executed and their expected output.
1000 .. code-block:: yaml
1004 - command: "kernel cycles"
1006 - command: "kernel version"
1008 - command: "kernel sleep 100"
1015 Specify a file containing test parameters to be used in the test.
1018 .. code-block:: yaml
1020 - command: "mpu mtest 1"
1022 - command: "mpu mtest 2"
1027 ``test_shell.yml`` in the test directory.
1033 One of the key features of Twister is its ability to decide on which platforms a given
1036 tests, the testing tools should be able to adapt the scope and select/filter out what
1043 Using ``--force-platform`` allows to override filtering caused by ``platform_allow``,
1049 * ``-p/--platform <platform_name>`` (can be used multiple times): only platforms
1051 * ``-l/--all``: all available platforms;
1052 * ``-G/--integration``: all platforms from an ``integration_platforms`` list in
1070 * ``--timeout-multiplier`` twister option which can be used to adjust timeouts in exact twister run.
1081 environments used to give developers fast feedback on changes. The mode can
1082 be activated using the ``--integration`` option of twister and narrows down
1083 the scope of builds and tests if applicable to platforms defined under the
1092 supports running any out-of-tree custom emulator defined in the board's :file:`board.cmake`.
1093 To use this type of simulation, add the following properties to
1096 .. code-block:: yaml
1099 - name: custom
1105 Then, in :file:`custom_board/board.cmake`, set the supported emulation platforms to ``custom``:
1107 .. code-block:: cmake
1114 .. code-block:: cmake
1118 <name_of_emu_binary to invoke during 'run'>
1119 … <any args to be passed to the command, i.e. ${BOARD}, ${APPLICATION_BINARY_DIR}/zephyr/zephyr.elf>
1128 Beside being able to run tests in QEMU and other simulated environments,
1136 To use this feature on a single connected device, run twister with
1141 .. group-tab:: Linux
1143 .. code-block:: bash
1145 scripts/twister --device-testing --device-serial /dev/ttyACM0 \
1146 --device-serial-baud 115200 -p frdm_k64f -T tests/kernel
1148 .. group-tab:: Windows
1150 .. code-block:: bat
1152 python .\scripts\twister --device-testing --device-serial COM1 \
1153 --device-serial-baud 115200 -p frdm_k64f -T tests/kernel
1155 The ``--device-serial`` option denotes the serial device the board is connected to.
1156 This needs to be accessible by the user running twister. You can run this on
1157 only one board at a time, specified using the ``--platform`` option.
1159 The ``--device-serial-baud`` option is only needed if your device does not run at
1162 To support devices without a physical serial port, use the ``--device-serial-pty``
1168 .. group-tab:: Linux
1170 .. code-block:: bash
1172 scripts/twister --device-testing --device-serial-pty "script.py" \
1173 -p intel_adsp/cavs25 -T tests/kernel
1175 .. group-tab:: Windows
1181 The script is user-defined and handles delivering the messages which can be
1182 used by twister to determine the test execution status.
1184 The ``--device-flash-timeout`` option allows to set explicit timeout on the
1188 The ``--device-flash-with-test`` option indicates that on the platform
1195 To build and execute tests on multiple devices connected to the host PC, a
1196 hardware map needs to be created with all connected devices and their
1198 Run the following command to produce the hardware map:
1202 .. group-tab:: Linux
1204 .. code-block:: bash
1206 ./scripts/twister --generate-hardware-map map.yml
1208 .. group-tab:: Windows
1210 .. code-block:: bat
1212 python .\scripts\twister --generate-hardware-map map.yml
1219 .. group-tab:: Linux
1221 .. code-block:: yaml
1223 - connected: true
1226 product: DAPLink CMSIS-DAP
1229 - connected: true
1232 product: J-Link
1236 .. group-tab:: Windows
1238 .. code-block:: yaml
1240 - connected: true
1246 - connected: true
1254 Any options marked as ``unknown`` need to be changed and set with the correct
1256 to be replaced with the correct values corresponding to the connected hardware.
1261 .. group-tab:: Linux
1263 .. code-block:: yaml
1265 - connected: true
1268 product: DAPLink CMSIS-DAP
1272 - connected: true
1275 product: J-Link
1280 .. group-tab:: Windows
1282 .. code-block:: yaml
1284 - connected: true
1287 product: DAPLink CMSIS-DAP
1291 - connected: true
1294 product: J-Link
1303 it for every run to get the correct serial devices and status of the devices.
1305 With the hardware map ready, you can run any tests by pointing to the map
1309 .. group-tab:: Linux
1311 .. code-block:: bash
1313 ./scripts/twister --device-testing --hardware-map map.yml -T samples/hello_world/
1315 .. group-tab:: Windows
1317 .. code-block:: bat
1319 python .\scripts\twister --device-testing --hardware-map map.yml -T samples\hello_world
1328 are supported with the hardware map features. Boards that require other runners to flash the
1331 Hardware map allows to set ``--device-flash-timeout`` and ``--device-flash-with-test``
1332 command line options as ``flash-timeout`` and ``flash-with-test`` fields respectively.
1335 Serial PTY support using ``--device-serial-pty`` can also be used in the
1338 .. code-block:: yaml
1340 - connected: true
1345 serial_pty: path/to/script.py
1347 - --remote-host=remote_host_ip_addr
1348 - --key=/path/to/key.pem
1351 The runner_params field indicates the parameters you want to pass to the
1352 west runner. For some boards the west runner needs some extra parameters to
1353 work. It is equivalent to following west and twister commands.
1357 .. group-tab:: Linux
1359 .. code-block:: bash
1361 west flash --remote-host remote_host_ip_addr --key /path/to/key.pem
1363 twister -p intel_adsp/cavs25 --device-testing --device-serial-pty script.py
1364 --west-flash="--remote-host=remote_host_ip_addr,--key=/path/to/key.pem"
1366 .. group-tab:: Windows
1374 For serial PTY, the "--generate-hardware-map" option cannot scan it out
1375 and generate a correct hardware map automatically. You have to edit it
1376 manually according to above example. This is because the serial port
1382 Some tests require additional setup or special wiring specific to the test.
1390 .. code-block:: yaml
1392 - connected: true
1394 - gpio_loopback
1397 product: DAPLink CMSIS-DAP
1401 When running ``twister`` with ``--device-testing``, the configured fixture
1402 in the hardware map file will be matched to test scenarios requesting the same fixtures
1406 :figclass: align-center
1408 Fixtures can also be provided via twister command option ``--fixture``, this option
1410 given fixtures will be assigned to all boards, this means that all boards set by
1413 Some fixtures allow for configuration strings to be appended, separated from the
1420 It may be useful to annotate board descriptions in the hardware map file
1421 with additional information. Use the ``notes`` keyword to do this. For
1424 .. code-block:: yaml
1426 - connected: false
1428 - gpio_loopback
1433 this file you will need to update serial to reference the third port, and platform
1434 to nrf5340dk/nrf5340/cpuapp or another supported board target.
1436 product: J-Link
1443 When (re-)generated the hardware map file will contain an ``id`` keyword
1444 that serves as the argument to ``--board-id`` when flashing. In some
1445 cases the detected ID is not the correct one to use, for example when
1446 using an external J-Link probe. The ``probe_id`` keyword overrides the
1449 .. code-block:: yaml
1451 - connected: false
1455 product: DAPLink CMSIS-DAP
1463 with names separated by spaces. This allows to run tests for
1465 re-configuring the hardware map file for each variant. For example:
1467 .. code-block:: yaml
1469 - connected: true
1472 - nrf5340dk/nrf5340/cpuapp
1473 - nrf5340dk/nrf5340/cpuapp/ns
1474 product: J-Link
1481 Twister allows user to provide configuration files defining a list of tests or
1482 platforms to be put under quarantine. Such tests will be skipped and marked
1487 To use the quarantine feature one has to add the argument
1488 ``--quarantine-list <PATH_TO_QUARANTINE_YAML>`` to a twister call.
1491 ``--quarantine-verify`` to the above argument. This will make twister skip all tests
1494 A quarantine yaml has to be a sequence of dictionaries. Each dictionary has to have
1496 to put under quarantine. In addition, an optional entry ``comment`` can be used, where
1497 some more details can be given (e.g. link to a reported issue). These comments will also
1498 be added to the output reports.
1501 when dealing with multiple issues within a subsystem, it is possible to use
1507 .. code-block:: yaml
1509 - scenarios:
1510 - sample.basic.helloworld
1511 comment: "Link to the issue: https://github.com/zephyrproject-rtos/zephyr/pull/33287"
1513 - scenarios:
1514 - kernel.common
1515 - kernel.common.(misra|tls)
1516 - kernel.common.nano64
1518 - .*_cortex_.*
1519 - native_sim
1521 To exclude a platform, use the following syntax:
1523 .. code-block:: yaml
1525 - platforms:
1526 - qemu_x86
1534 A test configuration can be used to customize various aspects of twister
1536 capabilities depending on the environment and makes it possible to adapt and
1539 The test configuration also adds support for test levels and the ability to
1540 assign a specific test to one or more levels. Using command line options of
1541 twister it is then possible to select a level and just execute the tests
1549 regular expressions and you can specify which test level to import from
1552 To help with testing outside of upstream CI infrastructure, additional
1556 - Ability to ignore default platforms as defined in board definitions
1557 (Those are mostly emulation platforms used to run tests in upstream
1559 - Option to specify your own list of default platforms overriding what
1561 - Ability to override ``build_on_all`` options used in some test scenarios.
1564 - Ignore some logic in twister to expand platform coverage in cases where
1573 - ``override_default_platforms``: override default key a platform sets in board
1575 configuration file as the list of default platforms. This option is set to
1577 - ``increased_platform_scope``: This option is set to True by default, when
1580 - ``default_platforms``: A list of additional default platforms to add. This list
1581 can either be used to replace the existing default platforms or can extend it
1586 .. code-block:: yaml
1592 - qemu_x86
1603 regular expressions and you can specify which test level to import from
1608 .. code-block:: yaml
1611 - name: my-test-level
1615 - kernel.threads.*
1616 - kernel.timer.behavior
1617 - arch.interrupt
1618 - boards.*
1624 To mix the Platform and level configuration, you can take an example as below:
1628 .. code-block:: yaml
1633 - frdm_k64f
1635 - name: smoke
1637 A plan to be used verifying basic zephyr features.
1638 - name: unit
1640 A plan to be used verifying unit test.
1641 - name: integration
1643 A plan to be used verifying integration.
1644 - name: acceptance
1646 A plan to be used verifying acceptance.
1647 - name: system
1649 A plan to be used verifying system.
1650 - name: regression
1652 A plan to be used verifying regression.
1655 To run with above test_config.yaml file, only default_platforms with given test level
1660 .. group-tab:: Linux
1662 .. code-block:: bash
1664 scripts/twister --test-config=<path to>/test_config.yaml
1665 -T tests --level="smoke"
1671 Enable ZTEST framework's :kconfig:option:`CONFIG_ZTEST_SHUFFLE` config option to
1674 the seed to the random number generator by providing ``-seed=value`` as an
1675 argument to twister. See :ref:`Shuffling Test Sequence <ztest_shuffle>` for more
1682 Robot files allow you to express interactive test scenarios in human-readable text format and execu…
1685 To execute a Robot test suite with twister, run the following command:
1689 .. group-tab:: Linux
1691 .. code-block:: bash
1693 …$ ./scripts/twister --platform hifive1 --test samples/subsys/shell/shell_module/sample.shell.shell…
1695 .. group-tab:: Windows
1697 .. code-block:: bat
1699 …python .\scripts\twister --platform hifive1 --test samples/subsys/shell/shell_module/sample.shell.…
1704 For the list of keywords provided by the Robot Framework itself, refer to `the official Robot docum…
1707 It provides a list of the most commonly used keywords together with links to the source code where …
1709 …to extend the framework by adding new keywords expressed directly in Robot test suite files, as an…
1710 …ramework.org/robotframework/latest/RobotFrameworkUserGuide.html#extending-robot-framework>`_ secti…
1715 To run a single testsuite instead of a whole group of test you can run:
1717 .. code-block:: bash
1719 $ twister -p qemu_riscv32 -s tests/kernel/interrupt/arch.shared_interrupt