1# Test a timer implementations variance and long term drift
2
3Records and calculates statistical values against a timer validating that.
4
51. Timer variance and standard deviation is below defined acceptable values.
62. Periodic timers do not drift in either direction from expected total time.
7
8Timers are meant to be precise and accurate. This test validates an implementation is both.
9
10--------------------------------------------------------------------------------
11
12External tool testing
13
14It's also possible to use an external tool, such as a logic analyzer, to
15collect samples. The "kernel.timer.timer_behavior_external" test will toggle a
16GPIO pin on every cycle - first cycle will be a rising edge. This test expects
17a Python module to interface with the external tool, which will provide the
18necessary statistics that the test will use to assert the test status.
19
20GPIO pin
21
22To enable external tool testing on a board, it must provide the compatible
23"test-kernel-timer-behavior-external", with property "timeout-gpios" being the
24GPIO pin that will be toggled each period.
25
26External tool interface
27
28In order to get data from the external tool, the test expects a Python module,
29named on testcase.yaml, with the following interface:
30
31    run(seconds: float, options: str) -> {}
32
33The `seconds` parameter defines for how long the data collection is expected
34to run; `options` is a string defined on testcase.yaml with options known to
35the external tool helper module. It should return a dictionary with the
36following statistics, in seconds:
37
38    'mean':         Mean time of each period
39    'stddev':       Standard deviation from the mean time
40    'var':          Variance from the mean time
41    'min':          Minimum period registered
42    'max':          Maximum period registered
43    'total_time':   Total time, between first and last period.
44
45Note that the collection may need to go a bit after the "seconds" parameter,
46to account for expected drift in the test and between the DUT and the external
47tool.
48
49One can check `pytest/saleae_logic2.py` file as a sample of external tool
50helper module.
51
52Twister
53
54For Twister execute the external tool testing, the fixture "gpio_timerout"
55must be available on the device. Also, testcase.yaml
56"harness_config/pytest_args" from "kernel.timer.timer_behavior_external" must
57have parameters "tool", with the name of a loadable Python module and
58"tool-options", a string with options passed to the Python module helper.
59
60Check testcase.yaml for an example using the saleae_logic2 module.
61
62Configuration options
63
64At its heart, the external testing is actually comparing two clocks: one on
65the board under test and one at the external tool (Zephyr implementation of
66the timer also plays a role, and it's the real target of the test, but it
67depends on the board's clock). Different clocks run at different speeds, so
68they tend to drift. To be able to account for this drift, the external test
69doesn't reuse CONFIG_TIMER_TEST_MAX_DRIFT and
70CONFIG_TIMER_TEST_PERIOD_MAX_DRIFT_PERCENT, instead introducing
71CONFIG_TIMER_EXTERNAL_TEST_MAX_DRIFT_PPM and
72CONFIG_TIMER_EXTERNAL_TEST_PERIOD_MAX_DRIFT_PPM, that can be more finely tuned
73for the hardware in question.
74
75Also, CONFIG_TIMER_EXTERNAL_TEST_SYNC_DELAY is used to set a delay before a
76test cycle starts, so that the external tool can be set up.
77
78Dependencies
79
80The external tool interface may have its own dependencies. For instance,
81the saleae_logic2 module ones are listed at pytest/requirements-saleae.txt.
82One can install them with pip (possibly in some virtual environment):
83
84    pip install -r pytest/requirements-saleae.txt
85