1# ElasticSearch index map examples
2
3This directory contains [ElasticSearch data store](https://github.com/elastic/elasticsearch)
4index map examples for use by [upload_test_results_es.py](../upload_test_results_es.py)
5script for [Twister](https://docs.zephyrproject.org/latest/develop/test/twister.html)
6JSON reports upload.
7
8An index map file allows to create destination index files on ElasticSearch server
9with explicit scheme having all required Twister JSON report objects associated
10to proper data types, eventually to store the expected document structure.
11Besides, it allows to track changes in Twister JSON report scheme
12and the corresponding data scheme in the same source code repository.
13An alternative way is to have index files created at the ElasticSearch server
14by other tools, or rely on default data mapping with potential type misalignments.
15
16The command examples below are simplified with parameters essential for data transformations.
17For other command line options and more details run the upload script with `--help`,
18or read its source code with inline comments there.
19
20Use these examples as a reference for your setup. Always pay attention to possible changes
21in the current Twister report format and in the upload script itself.
22Tune resulting data scheme and size depending on your particular needs.
23
24It is recommended to try the upload script with `--dry-run` option first
25to check the resulting data without its actual upload.
26
27You should set environment variables `ELASTICSEARCH_SERVER` and `ELASTICSEARCH_KEY`
28to connect with your server running the upload script.
29
30
31## Index creation with a map file
32
33Execute the upload script once for the first time to create your index at the ElasticSearch
34server with the appropriate map file, for example:
35```bash
36python3 ./scripts/ci/upload_test_results_es.py --create-index \
37  --index YOUR_INDEX_NAME \
38  --map-file YOUR_INDEX_MAP.json
39```
40
41
42## Upload transformations
43
44The upload script has several command line options to change `twister.json` data:
45exclude excess data, extract substrings by regular expressions, change data structure
46making it suitable for Kibana default visual components
47(see `--help` for more details):
48
49* `--exclude` removes excess testsuite properties not needed to store them
50  at Elasticsearch index.
51
52* `--transform` applies regexp group parsing rules to string properties extracting
53  derived object properties.
54
55* `--flatten` changes testsuite data structure in regard of one of its list components:
56  either `testcases` or `recording`, so each item there becomes an independent data record
57  inheriting all other testsuite properties, whereas the children object's properties
58  are renamed with the parent object's name as a prefix:
59  `testcases_` or `recording_` respectively.
60  Only one testsuite property object can be flattened this way per index upload.
61  All other testsuite objects will be treated according to the index structure.
62
63* Other command line options: `--flatten-dict-name`, `--flatten-list-names`,
64`--flatten-separator`, `--transpose-separator`, `--escape-separator`.
65
66
67## Index map examples and use cases
68
69
70### Twister test results
71
72Create the index:
73```bash
74python3 ./scripts/ci/upload_test_results_es.py --create-index \
75    --index zephyr-test-example \
76    --map-file zephyr_twister_index.json
77```
78
79Upload Twister test results 'as-is', without additional transformations:
80```bash
81python3 ./scripts/ci/upload_test_results_es.py \
82    --index zephyr-test-example \
83    ./twister-out/**/twister.json
84```
85
86
87### Twister tests with recording
88
89Store test results with `recording` data entries, for example from
90[Kernel Timer Behavior](https://github.com/zephyrproject-rtos/zephyr/tree/main/tests/kernel/timer/timer_behavior)
91test suite.
92
93Create the index:
94```bash
95python3 ./scripts/ci/upload_test_results_es.py --create-index \
96    --index zephyr-test-recording-example-1 \
97    --map-file zephyr_twister_flat_recording_metrics_index.json
98```
99
100Upload data with 'flattened' test suites creating documents for each `recording` data entry.
101```bash
102python3 ./scripts/ci/upload_test_results_es.py \
103    --index zephyr-test-recording-example-1 \
104    --exclude path run_id \
105    --flatten recording \
106    ./twister-out/**/twister.json
107```
108
109
110### Twister test with recording and extracting more data
111
112Store test results with `recording` data entries which are also scanned by regular expressions
113to extract embedded values, for example
114[Kernel Latency Benchmarks](https://github.com/zephyrproject-rtos/zephyr/tree/main/tests/benchmarks/latency_measure)
115test suite.
116
117Create the index:
118```bash
119python3 ./scripts/ci/upload_test_results_es.py --create-index \
120    --index zephyr-test-recording-example-2 \
121    --map-file zephyr_twister_flat_recording_index.json
122```
123
124Upload data with 'flattened' test suites creating documents for each `record` data entry
125with 3 additional data properties extracted by regular expressions.
126```bash
127python3 ./scripts/ci/upload_test_results_es.py \
128    --index zephyr-test-recording-example-2 \
129    --exclude path run_id \
130    --flatten recording \
131    --transform "{ 'recording_metric': '(?P<recording_metric_object>[^\.]+)\.(?P<recording_metric_action>[^\.]+)\.(?P<recording_metric_details>[^ -]+)' }" \
132    ./twister-out/**/twister.json
133```
134
135
136### Memory Footprint results
137
138To store Memory Footprint data reported in `twister-footprint.json` (see Twister `--footprint-report`).
139
140Create the index:
141```bash
142python3 ./scripts/ci/upload_test_results_es.py --create-index \
143    --index zephyr-memory-footprint-example \
144    --map-file zephyr_twister_flat_footprint_index.json
145```
146
147Upload data with 'flattened' footprint entries creating documents for each of them.
148```bash
149python3 ./scripts/ci/upload_test_results_es.py  \
150    --index zephyr-memory-footprint-example \
151    --exclude path run_id \
152    --flatten footprint \
153    --flatten-list-names "{'children':'name'}" \
154    --transform "{ 'footprint_name': '^(?P<footprint_area>([^\/]+\/){0,2})(?P<footprint_path>([^\/]*\/)*)(?P<footprint_symbol>[^\/]*)$' }" \
155    ../footprint_data/**/twister_footprint.json
156```
157