Lines Matching +full:bsim +full:- +full:test +full:- +full:results
3 Test Runner (Twister)
6 Twister scans for the set of test applications in the git repository
7 and attempts to execute them. By default, it tries to build each test
10 The default options will build the majority of the test applications on a
14 Because of the limited test execution coverage, twister
20 When using (at least) one ``-v`` option, twister's console output
21 shows for every test application how the test is run (qemu, native_sim, etc.) or
24 of a test is likewise reported in the ``twister.json`` and other report files.
25 There are a few reasons why twister only builds a test and doesn't run it:
27 - The test is marked as ``build_only: true`` in its ``.yaml``
29 - The test configuration has defined a ``harness`` but you don't have
31 - The target device is not connected and not available for flashing
32 - You or some higher level automation invoked twister with
33 ``--build-only``.
39 .. group-tab:: Linux
41 .. code-block:: bash
43 $ source zephyr-env.sh
46 .. group-tab:: Windows
48 .. code-block:: bat
50 zephyr-env.cmd
59 .. group-tab:: Linux
61 .. code-block:: bash
63 $ ./scripts/twister --all --enable-slow
65 .. group-tab:: Windows
67 .. code-block:: bat
69 python .\scripts\twister --all --enable-slow
75 the ``--platform`` option, it is a platform filter for testing, with this
76 option, test suites will only be built/run on the platforms specified.
78 you can use ``--platform board@revision`` to test on a specific revision.
84 .. group-tab:: Linux
86 .. code-block:: bash
88 $ ./scripts/twister --help
90 .. group-tab:: Windows
92 .. code-block:: bat
94 python .\scripts\twister --help
108 required for best test coverage for this specific board:
110 .. code-block:: yaml
113 name: NXP FRDM-K64F
117 - zephyr
118 - gnuarmemb
119 - xtools
121 - arduino_gpio
122 - arduino_i2c
123 - netif:eth
124 - adc
125 - i2c
126 - nvs
127 - spi
128 - gpio
129 - usb_device
130 - watchdog
131 - can
132 - pwm
142 .. code-block:: console
145 west build -b reel_board
147 cmake -DBOARD=reel_board ..
156 .. code-block:: yaml
159 - name: qemu
160 - name: armfvp
162 - name: custom
166 simulation can be selected with ``--simulation <simulation_name>``.
177 Available RAM on the board (specified in KB). This is used to match test scenario
180 Available FLASH on the board (specified in KB). This is used to match test scenario
186 .. code-block:: yaml
189 - pci
191 This indicates the board does support PCI. You can make a test scenario build or
194 .. code-block:: yaml
197 - netif:eth
198 - sensor:bmi16
200 A test scenario can depend on 'eth' to only test ethernet or on 'netif' to run
224 Multiply each test scenario timeout by specified ratio. This option allows to tune timeouts only
226 power-efficient but slow CPU or simulation platform which can perform instruction accurate
241 files in the application's project directory. This test application
243 identifying a Test Scenario.
248 :alt: Twister and a Test application project.
249 :figclass: align-center
251 Twister and a Test application project.
254 Test application configurations are written using the YAML syntax and share the
257 A Test Scenario is a set of conditions and variables defined in a Test Scenario
258 entry, under which a set of Test Suites will be built and executed.
260 A Test Suite is a collection of Test Cases which are intended to be used to test
261 a software program to ensure it meets certain requirements. The Test Cases in a
262 Test Suite are either related or meant to be executed together.
264 Test Scenario, Test Suite, and Test Case names must follow to these basic rules:
266 #. The format of the Test Scenario identifier shall be a string without any spaces or
270 #. Each Test Scenario identifier shall start with a section name followed by a
271 subsection names delimited with a dot (``.``). For example, a test scenario
274 #. All Test Scenario identifiers within a Test Configuration (``testcase.yaml`` file)
281 #. The full canonical name of a Test Suite is:
282 ``<Test Application Project path>/<Test Scenario identifier>``
284 #. Depending on the Test Suite implementation, its Test Case identifiers consist
288 a Test Scenario identifier from the corresponding ``testcase.yaml`` file,
289 a Ztest suite name, and a Ztest test name:
290 ``<Test Scenario identifier>.<Ztest suite name>.<Ztest test name>``
293 a Test Scenario identifier from the corresponding ``testcase.yaml`` (or
295 Test Case name, for example: ``debug.coredump.logging_backend``.
298 The ``--no-detailed-test-id`` command line option modifies the above rules in this way:
300 #. A Test Suite name has only ``<Test Scenario identifier>`` component.
303 #. With short Test Suite names in this mode, all corresponding Test Scenario names
306 #. **Ztest** Test Case names have only Ztest components ``<Ztest suite name>.<Ztest test name>``.
307 Its parent Test Suite name equals to the corresponding Test Scenario identifier.
310 The following is an example test configuration with a few options that are
314 .. code-block:: yaml
333 .. code-block:: yaml
350 A Test Scenario entry in the ``tests:`` YAML dictionary has its Test Scenario
353 Each Test Scenario entry in the Test Application configuration can define the
359 A set of string tags for the test scenario. Usually pertains to
364 skip test scenario unconditionally. This can be used for broken tests for
368 Don't run this test scenario unless ``--enable-slow`` or ``--enable-slow-only`` was
369 passed in on the command line. Intended for time-consuming test scenarios that
370 are only run under certain circumstances, like daily builds. These test
375 test scenario.
380 .. code-block:: yaml
385 test:
389 - arch:x86:CONFIG_ADC_ASYNC=y
390 - platform:qemu_x86:CONFIG_DEBUG=y
391 - platform:mimxrt1060_evk:SHIELD=rk043fn66hs_ctg
392 - simulation:qemu:CONFIG_MPU=y
396 when building or running the test scenario. For example:
398 .. code-block:: yaml
403 test:
407 - CONFIG_ADC_ASYNC=y
412 .. code-block:: yaml
417 test:
421 - arch:x86:CONFIG_ADC_ASYNC=y
422 - platform:qemu_x86:CONFIG_DEBUG=y
426 If true, twister will not try to run the test even if the test is runnable
429 This keyword is reserved for tests that are used to test if some code
430 actually builds. A ``build_only`` test is not designed to be run in any
434 This option is often used to test drivers and the fact that they are correctly
436 test shall not be used to verify the functionality of the driver.
439 If true, attempt to build test scenario on all available platforms. This is mostly
444 will enable the test only those platforms that provide this feature.
447 Test levels this test should be part of. If a level is present, this
448 test will be selectable using the command line option ``--level <level name>``
451 estimated minimum amount of RAM in KB needed for this test to build and run. This is
455 estimated minimum amount of ROM in KB needed for this test to build and run. This is
461 Length of time to run test before automatically killing it.
465 Set of architectures that this test scenario should only be run for.
468 Set of architectures that this test scenario should not run on.
471 Set of platform vendors that this test scenario should only be run for. The
477 Set of platform vendors that this test scenario should not run on.
482 Set of platforms that this test scenario should only be run for. Do not use
484 constraints, this option should only be used if the test or sample can
489 invoked with the ``--integration`` option. Use this instead of
494 Set of platforms that this test scenario should not run on.
505 the images onto the target. The ``--erase`` option of west flash is
511 harness needed to run a test successfully. A harness is a feature of
516 Twister to be able to evaluate if a test passes criteria. For example, a
518 verdict on whether a test has passed or failed, however, Twister lack this
523 - ztest
524 - test
525 - console
526 - pytest
527 - gtest
528 - robot
535 parse a test's text output for a regex defined in the test's YAML file.
536 The ``robot`` harness is used to execute Robot Framework test suites
541 - keyboard
542 - net
543 - bluetooth
545 Harness ``bsim`` is implemented in limited way - it helps only to copy the
555 Often a test needs to only be built and run once to qualify as passing.
557 passing the test on a single platform for each arch is enough to qualify the
561 For example to key on (arch, simulation) to ensure a test is run once
564 .. code-block:: yaml
567 - arch
568 - simulation
578 what features it supports. This option will enable the test to run on
587 Strings with regular expressions to match with the test's output
588 to confirm the test runs as expected.
596 at the test's output lines where the test provides some custom data
599 of the test suite object in ``twister.json``.
604 .. code-block:: yaml
613 The corresponding ``recording.csv`` columns will contain strings as-is.
615 Using this option, a test log can convey layered data structures
616 passed from the test image for further analysis with summary results,
621 .. code-block:: yaml
627 when matched to a test log string:
629 .. code-block:: none
635 .. code-block:: json
648 Specify a test scenario dependency on an external device(e.g., sensor),
650 specific test setup and board selection logic to pick the particular
655 Only one fixture can be defined per test scenario and the fixture name has to
656 be unique across all tests in the test suite.
662 executed when a test scenario begins to run. The default pytest directory is
664 the test scenario passed or failed according to the pytest report.
667 .. code-block:: yaml
671 - "pytest/test_shell_help.py"
672 - "../shell/pytest/test_shell.py"
673 - "/tmp/test_shell.py"
674 - "~/tmp/test_shell.py"
675 - "$ZEPHYR_BASE/samples/subsys/testsuite/pytest/shell/pytest/test_shell.py"
676 - "pytest/test_shell_help.py::test_shell2_sample" # select pytest subtest
677 … - "pytest/test_shell_help.py::test_shell2_sample[param_a]" # select pytest parametrized subtest
683 ``pytest_args: [‘-k=test_method’, ‘--log-level=DEBUG’]``. Note that
684 ``--pytest-args`` can be passed multiple times to pass several arguments
691 If the scope is set to ``function``, DUT is launched for every test case
695 Specify one or more paths to a file containing a Robot Framework test suite to be run.
703 default based on the test path and scenario name.
707 .. code-block:: yaml
718 - "Temperature:(.*)C"
719 - "Relative Humidity:(.*)%"
722 test:
730 .. code-block:: yaml
738 - pytest_dir1
739 - $ENV_VAR/samples/test/pytest_dir2
743 - pytest/test_file_1.py
744 - test_file_2.py::test_A
745 - test_file_2.py::test_B[param_a]
749 .. code-block:: yaml
757 It can be more than one test suite using a list.
759 .. code-block:: yaml
766 - [robot file path 1]
767 - [robot file path 2]
768 - [robot file path n]
772 .. code-block:: yaml
780 - --exclude tag
781 - --stop-on-error
784 Filter whether the test scenario should be run by evaluating an expression
787 .. code-block:: none
791 <all CONFIG_* key/value pairs in the test's generated defconfig>,
803 .. code-block:: antlr
827 if the symbol is defined to a non-empty string.
834 * all comparison operators (non-associative)
839 .. code-block:: none
845 .. code-block:: none
853 .. code-block:: none
860 :ref:`Snippets <snippets>` are supported in twister for test scenarios that
862 zephyr snippet directory and test application directory for finding
864 must be compatible with a board for the test to run on them, they are not
869 .. code-block:: yaml
874 - cdc-acm-console
875 - user-snippet-example
877 The set of test scenarios that actually run depends on directives in the test scenario
879 running with ``-v`` or examining the discard report
880 (:file:`twister_discard.csv`) can help show why particular test scenarios were
885 To update this, pass the ``--all --release`` options.
897 test scenario should run. This behavior has its roots in Twister being developed as
898 a test runner for Zephyr's CI. With hundreds of available platforms and thousands of
902 Twister always prepares an initial list of platforms in scope for a given test,
903 based on command line arguments and the :ref:`test's configuration <test_config_args>`. Then,
906 Using ``--force-platform`` allows to override filtering caused by ``platform_allow``,
907 ``platform_exclude``, ``arch_allow`` and ``arch_exclude`` keys in test configuration
912 * ``-p/--platform <platform_name>`` (can be used multiple times): only platforms
914 * ``-l/--all``: all available platforms;
915 * ``-G/--integration``: all platforms from an ``integration_platforms`` list in
916 a given test configuration file. If a test has no ``integration_platforms``
929 * ``timeout`` option in each test scenario. See :ref:`here <twister_test_case_timeout>` for more
933 * ``--timeout-multiplier`` twister option which can be used to adjust timeouts in exact twister run.
938 Overall test scenario timeout is a multiplication of these three parameters.
945 be activated using the ``--integration`` option of twister and narrows down
947 integration keyword in the test configuration file (``testcase.yaml`` and
955 supports running any out-of-tree custom emulator defined in the board's :file:`board.cmake`.
959 .. code-block:: yaml
962 - name: custom
970 .. code-block:: cmake
977 .. code-block:: cmake
993 reports for each run with detailed FAIL/PASS results.
1004 .. group-tab:: Linux
1006 .. code-block:: bash
1008 scripts/twister --device-testing --device-serial /dev/ttyACM0 \
1009 --device-serial-baud 115200 -p frdm_k64f -T tests/kernel
1011 .. group-tab:: Windows
1013 .. code-block:: bat
1015 python .\scripts\twister --device-testing --device-serial COM1 \
1016 --device-serial-baud 115200 -p frdm_k64f -T tests/kernel
1018 The ``--device-serial`` option denotes the serial device the board is connected to.
1020 only one board at a time, specified using the ``--platform`` option.
1022 The ``--device-serial-baud`` option is only needed if your device does not run at
1025 To support devices without a physical serial port, use the ``--device-serial-pty``
1031 .. group-tab:: Linux
1033 .. code-block:: bash
1035 scripts/twister --device-testing --device-serial-pty "script.py" \
1036 -p intel_adsp/cavs25 -T tests/kernel
1038 .. group-tab:: Windows
1044 The script is user-defined and handles delivering the messages which can be
1045 used by twister to determine the test execution status.
1047 The ``--device-flash-timeout`` option allows to set explicit timeout on the
1051 The ``--device-flash-with-test`` option indicates that on the platform
1052 the flash operation also executes a test scenario, so the flash timeout is
1053 increased by a test scenario timeout.
1065 .. group-tab:: Linux
1067 .. code-block:: bash
1069 ./scripts/twister --generate-hardware-map map.yml
1071 .. group-tab:: Windows
1073 .. code-block:: bat
1075 python .\scripts\twister --generate-hardware-map map.yml
1082 .. group-tab:: Linux
1084 .. code-block:: yaml
1086 - connected: true
1089 product: DAPLink CMSIS-DAP
1092 - connected: true
1095 product: J-Link
1099 .. group-tab:: Windows
1101 .. code-block:: yaml
1103 - connected: true
1109 - connected: true
1124 .. group-tab:: Linux
1126 .. code-block:: yaml
1128 - connected: true
1131 product: DAPLink CMSIS-DAP
1135 - connected: true
1138 product: J-Link
1143 .. group-tab:: Windows
1145 .. code-block:: yaml
1147 - connected: true
1150 product: DAPLink CMSIS-DAP
1154 - connected: true
1157 product: J-Link
1172 .. group-tab:: Linux
1174 .. code-block:: bash
1176 ./scripts/twister --device-testing --hardware-map map.yml -T samples/hello_world/
1178 .. group-tab:: Windows
1180 .. code-block:: bat
1182 python .\scripts\twister --device-testing --hardware-map map.yml -T samples\hello_world
1194 Hardware map allows to set ``--device-flash-timeout`` and ``--device-flash-with-test``
1195 command line options as ``flash-timeout`` and ``flash-with-test`` fields respectively.
1198 Serial PTY support using ``--device-serial-pty`` can also be used in the
1201 .. code-block:: yaml
1203 - connected: true
1210 - --remote-host=remote_host_ip_addr
1211 - --key=/path/to/key.pem
1220 .. group-tab:: Linux
1222 .. code-block:: bash
1224 west flash --remote-host remote_host_ip_addr --key /path/to/key.pem
1226 twister -p intel_adsp/cavs25 --device-testing --device-serial-pty script.py
1227 --west-flash="--remote-host=remote_host_ip_addr,--key=/path/to/key.pem"
1229 .. group-tab:: Windows
1237 For serial PTY, the "--generate-hardware-map" option cannot scan it out
1245 Some tests require additional setup or special wiring specific to the test.
1246 Running the tests without this setup or test fixture may fail. A test scenario can
1253 .. code-block:: yaml
1255 - connected: true
1257 - gpio_loopback
1260 product: DAPLink CMSIS-DAP
1264 When running ``twister`` with ``--device-testing``, the configured fixture
1265 in the hardware map file will be matched to test scenarios requesting the same fixtures
1269 :figclass: align-center
1271 Fixtures can also be provided via twister command option ``--fixture``, this option
1274 current twister command can run those test scenarios which request the same fixtures.
1278 requested by test scenarios.
1287 .. code-block:: yaml
1289 - connected: false
1291 - gpio_loopback
1299 product: J-Link
1306 When (re-)generated the hardware map file will contain an ``id`` keyword
1307 that serves as the argument to ``--board-id`` when flashing. In some
1309 using an external J-Link probe. The ``probe_id`` keyword overrides the
1312 .. code-block:: yaml
1314 - connected: false
1318 product: DAPLink CMSIS-DAP
1328 re-configuring the hardware map file for each variant. For example:
1330 .. code-block:: yaml
1332 - connected: true
1335 - nrf5340dk/nrf5340/cpuapp
1336 - nrf5340dk/nrf5340/cpuapp/ns
1337 product: J-Link
1347 running larger test suits, where a failure of one test can affect the execution
1351 ``--quarantine-list <PATH_TO_QUARANTINE_YAML>`` to a twister call.
1354 ``--quarantine-verify`` to the above argument. This will make twister skip all tests
1370 .. code-block:: yaml
1372 - scenarios:
1373 - sample.basic.helloworld
1374 comment: "Link to the issue: https://github.com/zephyrproject-rtos/zephyr/pull/33287"
1376 - scenarios:
1377 - kernel.common
1378 - kernel.common.(misra|tls)
1379 - kernel.common.nano64
1381 - .*_cortex_.*
1382 - native_sim
1386 .. code-block:: yaml
1388 - platforms:
1389 - qemu_x86
1394 Test Configuration
1397 A test configuration can be used to customize various aspects of twister
1402 The test configuration also adds support for test levels and the ability to
1403 assign a specific test to one or more levels. Using command line options of
1407 Additionally, the test configuration allows defining level
1409 the test itself does not have this information already.
1412 regular expressions and you can specify which test level to import from
1419 - Ability to ignore default platforms as defined in board definitions
1422 - Option to specify your own list of default platforms overriding what
1424 - Ability to override ``build_on_all`` options used in some test scenarios.
1427 - Ignore some logic in twister to expand platform coverage in cases where
1436 - ``override_default_platforms``: override default key a platform sets in board
1440 - ``increased_platform_scope``: This option is set to True by default, when
1443 - ``default_platforms``: A list of additional default platforms to add. This list
1449 .. code-block:: yaml
1455 - qemu_x86
1458 Test Level Configuration
1461 The test configuration allows defining test levels, level dependencies and
1462 additional inclusion of tests into a specific test level if the test itself
1466 regular expressions and you can specify which test level to import from
1469 And example test level configuration:
1471 .. code-block:: yaml
1474 - name: my-test-level
1476 my custom test level
1478 - kernel.threads.*
1479 - kernel.timer.behavior
1480 - arch.interrupt
1481 - boards.*
1491 .. code-block:: yaml
1496 - frdm_k64f
1498 - name: smoke
1501 - name: unit
1503 A plan to be used verifying unit test.
1504 - name: integration
1507 - name: acceptance
1510 - name: system
1513 - name: regression
1518 To run with above test_config.yaml file, only default_platforms with given test level
1519 test scenarios will run.
1523 .. group-tab:: Linux
1525 .. code-block:: bash
1527 scripts/twister --test-config=<path to>/test_config.yaml
1528 -T tests --level="smoke"
1536 dependencies between test cases. For native_sim platforms, you can provide
1537 the seed to the random number generator by providing ``-seed=value`` as an
1538 argument to twister. See :ref:`Shuffling Test Sequence <ztest_shuffle>` for more
1545 Robot files allow you to express interactive test scenarios in human-readable text format and execu…
1548 To execute a Robot test suite with twister, run the following command:
1552 .. group-tab:: Linux
1554 .. code-block:: bash
1556 …$ ./scripts/twister --platform hifive1 --test samples/subsys/shell/shell_module/sample.shell.shell…
1558 .. group-tab:: Windows
1560 .. code-block:: bat
1562 …python .\scripts\twister --platform hifive1 --test samples/subsys/shell/shell_module/sample.shell.…
1572 …s expressed directly in Robot test suite files, as an external Python library or, like Renode does…
1573 …ramework.org/robotframework/latest/RobotFrameworkUserGuide.html#extending-robot-framework>`_ secti…
1578 To run a single testsuite instead of a whole group of test you can run:
1580 .. code-block:: bash
1582 $ twister -p qemu_riscv32 -s tests/kernel/interrupt/arch.shared_interrupt