1.. _twister_statuses:
2
3Twister Status
4##############
5
6What is a Twister Status?
7=========================
8
9Twister Status formulates the current state of
10
11- ``Harness``
12- ``TestCase``
13- ``TestSuite``
14- ``TestInstance``
15
16in a comprehensive and easy-to understand way.
17In practice, most users will be interested in Statuses
18of Instances and Cases after the conclusion of their Twister runs.
19
20.. tip::
21
22   Nomenclature reminder:
23
24   .. tabs::
25
26      .. tab:: ``Harness``
27
28         ``Harness`` is a Python class inside Twister that allows us
29         to capture and analyse output from a program external to Twister.
30         It has been excised from this page for clarity,
31         as it does not appear in final reports.
32
33      .. tab:: ``TestCase``
34
35         ``TestCase``, also called Case, is a piece of code that aims to verify some assertion.
36         It is the smallest subdivision of testing in Zephyr.
37
38      .. tab:: ``TestSuite``
39
40         ``TestSuite``, also called Suite, is a grouping of Cases.
41         One can modify Twister's behaviour on a per-Suite basis via ``testcase.yaml`` files.
42         Such grouped Cases should have enough in common
43         for it to make sense to treat them all the same by Twister.
44
45      .. tab:: ``TestInstance``
46
47         ``TestInstance``, also called Instance, is a Suite on some platform.
48         Twister typically reports its results for Instances,
49         despite them being called "Suites" there.
50         If a status is marked as applicable for Suites, it is also applicable for Instances.
51         As the distinction between them is not useful in this section,
52         whenever you read "Suite", assume the same for Instances.
53
54   More detailed explanation can be found :ref:`here <twister_tests_long_version>`.
55
56Possible Twister Statuses
57=========================
58
59.. list-table:: Twister Statuses
60   :widths: 10 10 66 7 7
61   :header-rows: 1
62
63   * - In-code
64     - In-text
65     - Description
66     - Suite
67     - Case
68   * - FILTER
69     - filtered
70     - A static or runtime filter has eliminated the test from the list of tests to use.
71     - ✓
72     - ✓
73   * - NOTRUN
74     - not run
75     - The test wasn't run because it was not runnable in current configuration.
76       It was, however, built correctly.
77     - ✓
78     - ✓
79   * - BLOCK
80     - blocked
81     - The test was not run because of an error or crash in the test suite.
82     - ✕
83     - ✓
84   * - SKIP
85     - skipped
86     - Test was skipped by some other reason than previously delineated.
87     - ✓
88     - ✓
89   * - ERROR
90     - error
91     - The test produced an error in running the test itself.
92     - ✓
93     - ✓
94   * - FAIL
95     - failed
96     - The test produced results different than expected.
97     - ✓
98     - ✓
99   * - PASS
100     - passed
101     - The test produced results as expected.
102     - ✓
103     - ✓
104
105**In-code** and **In-text** are the naming contexts of a given status:
106the former is rather internal for Twister and appears in logs,
107whereas the latter is used in the JSON reports.
108
109.. note::
110
111   There are two more Statuses that are internal to Twister.
112   ``NONE`` (A starting status for Cases and Suites) and
113   ``STARTED`` (Indicating an in-progress Case).
114   Those should not appear in the final Twister report.
115   Appearance of those non-terminal Statuses in one's report files indicates a problem with Twister.
116
117
118Case and Suite Status combinations
119============================================
120
121.. list-table:: Case and Suite Status combinations
122   :widths: 22 13 13 13 13 13 13
123   :align: center
124   :header-rows: 1
125   :stub-columns: 1
126
127   * - ↓ Case\\Suite →
128     - FILTER
129     - ERROR
130     - FAIL
131     - PASS
132     - NOTRUN
133     - SKIP
134   * - FILTER
135     - ✓
136     - ✕
137     - ✕
138     - ✕
139     - ✕
140     - ✕
141   * - ERROR
142     - ✕
143     - ✓
144     - ✕
145     - ✕
146     - ✕
147     - ✕
148   * - BLOCK
149     - ✕
150     - ✓
151     - ✓
152     - ✕
153     - ✕
154     - ✕
155   * - FAIL
156     - ✕
157     - ✓
158     - ✓
159     - ✕
160     - ✕
161     - ✕
162   * - PASS
163     - ✕
164     - ✓
165     - ✓
166     - ✓
167     - ✕
168     - ✕
169   * - NOTRUN
170     - ✕
171     - ✕
172     - ✕
173     - ✕
174     - ✓
175     - ✕
176   * - SKIP
177     - ✕
178     - ✓
179     - ✓
180     - ✓
181     - ✕
182     - ✓
183
184✕ indicates that such a combination should not happen in a proper Twister run. In other words,
185no Suite of a status indicated by the table column should contain any Cases of a status indicated
186by the table row.
187
188✓ indicates a proper combination.
189
190Detailed explanation, per Suite Status
191-------------------------------------------
192
193``FILTER``:
194  This status indicates that the whole Suite has been statically filtered
195  out of a given Twister run. Thus, any Case within it should also have such a status.
196
197``ERROR``:
198  Suite encountered a problem when running the test. It requires at least one case with
199  ``ERROR`` or ``BLOCK`` status. As this takes precedence over all other Case statuses, all valid
200  terminal Case statuses can be within such a Suite.
201
202``FAIL``:
203  Suite has at least one Case that did not meet its assertions. This takes precedence over
204  all other Case statuses, given that the conditions for an ERROR status have not been met.
205
206``PASS``:
207  Suite has passed properly. It cannot contain any Cases with ``BLOCK``, ``ERROR``, or ``FAIL``
208  statuses, as those indicate a problem when running the Suite.
209
210``NOTRUN``:
211  Whole suite was not run, but only built. It requires than all Cases within were not run.
212  As runnability is decided on a per-Suite basis, only ``NOTRUN`` is applicable for its Cases.
213
214``SKIP``:
215  Whole Suite has been skipped at runtime. All Cases need to have ``SKIP`` status as well.
216