1# Test a timer implementations variance and long term drift
2
3Records and calculates statistical values against a timer validating that.
4
51. Timer variance and standard deviation is below defined acceptable values.
62. Periodic timers do not drift in either direction from expected total time.
7
8Timers are meant to be precise and accurate. This test validates an implementation is both.
9
10--------------------------------------------------------------------------------
11
12External tool testing
13
14It's also possible to use an external tool, such as a logic analyzer, to
15collect samples. The "kernel.timer.timer_behavior_external" test will toggle a
16GPIO pin on every cycle - first cycle will be a rising edge. This test expects
17a Python module to interface with the external tool, which will provide the
18necessary statistics that the test will use to assert the test status.
19
20GPIO pin
21
22To enable external tool testing on a board, it must provide the compatible
23"test-kernel-timer-behavior-external", with property "timeout-gpios" being the
24GPIO pin that will be toggled each period.
25
26External tool interface
27
28In order to get data from the external tool, the test expects a Python module,
29named on testcase.yaml, with the following interface:
30
31    run(seconds: float, options: str) -> {}, int
32
33The `seconds` parameter defines for how long the data collection is expected
34to run; `options` is a string defined on testcase.yaml with options known to
35the external tool helper module. It should return a tuple with a dictionary
36for the following time statistics, in seconds:
37
38    'mean':         Mean time of each period
39    'stddev':       Standard deviation from the mean time
40    'var':          Variance from the mean time
41    'min':          Minimum period registered
42    'max':          Maximum period registered
43    'total_time':   Total time, between first and last period.
44
45and an integer value of how many data values are collected for the statistics.
46
47Note that the collection may need to go a bit after the "seconds" parameter,
48to account for expected drift in the test and between the DUT and the external
49tool.
50
51One can check `pytest/saleae_logic2.py` file as a sample of external tool
52helper module.
53
54Twister
55
56For Twister execute the external tool testing, the fixture "gpio_timerout"
57must be available on the device. Also, testcase.yaml
58"harness_config/pytest_args" from "kernel.timer.timer_behavior_external" must
59have parameters "tool", with the name of a loadable Python module and
60"tool-options", a string with options passed to the Python module helper.
61
62Check testcase.yaml for an example using the saleae_logic2 module.
63
64Configuration options
65
66At its heart, the external testing is actually comparing two clocks: one on
67the board under test and one at the external tool (Zephyr implementation of
68the timer also plays a role, and it's the real target of the test, but it
69depends on the board's clock). Different clocks run at different speeds, so
70they tend to drift. To be able to account for this drift, the external test
71doesn't reuse CONFIG_TIMER_TEST_MAX_DRIFT and
72CONFIG_TIMER_TEST_PERIOD_MAX_DRIFT_PERCENT, instead introducing
73CONFIG_TIMER_EXTERNAL_TEST_MAX_DRIFT_PPM and
74CONFIG_TIMER_EXTERNAL_TEST_PERIOD_MAX_DRIFT_PPM, that can be more finely tuned
75for the hardware in question.
76
77Also, CONFIG_TIMER_EXTERNAL_TEST_SYNC_DELAY is used to set a delay before a
78test cycle starts, so that the external tool can be set up.
79
80Dependencies
81
82The external tool interface may have its own dependencies. For instance,
83the saleae_logic2 module ones are listed at pytest/requirements-saleae.txt.
84One can install them with pip (possibly in some virtual environment):
85
86    pip install -r pytest/requirements-saleae.txt
87