Lines Matching +full:- +full:a
7 Here is a quick summary of how to get started with the framework the first time the repository is c…
11 You can also look at the artifact for the commit : it is containing a CMSIS Pack for this commit.
28 `python preprocess.py -f desc.txt`
30 `python preprocess.py -f desc_f16.txt -o Output_f16.pickle`
32 `python processTests.py -e`
34 `python processTests.py -e -f Output_f16.pickle`
38 `python processTests.py -e BasicTestsF32`
44 `python processTests.py -f Output_f16.pickle -e BasicTestsF16`
67 `cpackget add -f test_packlist.txt`
75 …_v5\ARM\VHT\VHT_Corstone_SSE-300_Ethos-U55.exe -f configs\ARM_VHT_Corstone_300_config.txt Objects\…
81 `python ../processResult.py -f ../Output.pickle -e -r results.txt`
89 #### R1 : The tests shall be described in a file
90 We need a source of truth which is describing all the tests and can be used
93 #### R2 : The test description should support a hierarchy
94 We have lots of tests. We need to be able to organize them in a
97 #### R3 : A test shall be uniquely identified
98 We need a way to identify in an unique way each test to ensure traceability and enable to create
112 #### R7 : It shall be possible to specify a subset of parameters (which could be empty) to compute …
113 For instance, if our test is dependent on a vector size, we may want to compute a linear regression
120 #### R8 : The concept of a test suite shall be supported.
121 A test suite is a set of tests packaged with some data.
125 …a device under tests (DUT) as the place where the function to test is executed. But the test itsel…
129 A test should start (as far as possible) in a clean state. There should not be interferences betwee…
137 #### R13 : Control of the tests could run on the DUT but could also be run on a host
147 The design is a consequence of all the requirements.
151 A test description file is defined with a specific syntax to support R1 to R8.
164 The tests are organized in a hierarchy. For each node of the hierarchy, a C++ class is specified.
166 For the test suite, the script is generating a partial implementation since a test suite is contain…
168 The patterns, output of tests, parameters are also following a hierarchical structure. But they do …
169 to be organized in exactly the same way. So, the folder property of a node is optional.
171 A folder can be reused for different nodes. For instance, you may have a suite for testing and one …
173 A test suite is more complex than a group since it contains the description of the tests and relate…
188 A function is described with some text and followed by the name of the function in the C++ class.
193 A test is requiring input patterns, reference patterns and outputs (to be compared to the reference…
212 A pattern or output description is an ID (to be used in the code) followed by a filename.
220 A benchmark will often have to be run with different lengths for the input.
221 So we need a way to communicate arguments to a function.
224 In the benchmark results, we may want to generate a CSV (or any other format) with different column…
226 And we may want to compute a regression formula using only a subset of those arguments.
228 So, we have the possibility in the suite section to add a parameter section to describe all of this.
235 A,B,C
236 Summary A,B
237 Names "Param A", "Param B"
238 Formula "A*B"
244 In above example we declare that the functions of the suite are using 3 parameters named A,B and C.
245 We declare that a regression formula will use only A and B. So for each C value, we will get a diff…
249 We define a regression formula using R syntax. (We do not write "cycles ~ A*B" but only "A*B")
251 Once parameters have been described, we need a way to feed parameter values to a test.
253 … way is a parameter file. Problem of a parameter file when it has to be included in the test (C ar…
255 Those parameters values, when specified with a file, are described with:
262 When the parameter is specified with a generator then the syntax is :
265 A = [1,3,5]
272 To use parameters with a function the syntax is:
275 Test A:testA -> PARAM3_ID
276 } -> PARAM1_ID
314 First line must be a multiple of the number of parameters. In our above example we have 3 parameter…
315 So, the number of possible run must be a multiple of 3 since we need to specify values for all para…
321 A disabled group/suite/test is not executed (and its code not generated for group/suite).
322 Using disabled for tests is allowing to disable a test without changing the test ID of following te…
327 …defined by virtual class Memory. An implementation ArrayMemory is provided which is using a buffer.
330 A memory manager can provide new buffer, free all the already allocated buffers and give a generati…
334 It is implemented with a Runner class. The only implementation provided is IORunner,
336 A Runner is just an implementation of the visitor pattern. A runner is applied to the tree of tests.
337 In case of the IO runner, an IO mechanism and a memory manager must be provided.
339 The runner is running a test and for benchmark measuring the cycles.
341 Generally, there is a calibration at beginning of the Runner to estimate the overhead of
345 According to R12 and R15, tests do not know how to access patterns. It is a responsiblity implement…
350 FPGA is when you need to run the tests in a constrained environment where you only have stdout. The…
353 They will return NULL when a pattern is still referencing a generation of memory older than the cur…
355 PatternMgr is the link between IO and Memory and knows how to load a pattern and save it into memor…
359 When instantiating a runner, you can specify the running mode with an enum. For instance Testing::k…
367 For R14, we have a python script which will process the result of tests and format it into several …
407 python preprocess.py -f desc.txt
409 This will create a file `Output.pickle` which is containing a Python object representing
410 the parsed data structure. It is done because parsing a big test description file is quite slow.
416 python processTests.py -f Output.pickle
422 …-e option (for embedded). It will include all the patterns (for the selected tests) into a C array…
423 always use -e option.
425 python processTests.py -e
427 You can pass a C++ class to specifiy that you want to generate tests only for a specific group or s…
429 python processTests.py -e BasicTests
431 You can add a test ID to specify that you wan to run only a specific test in the suite:
433 python processTests.py -e BasicTests 4
439 …iltering and will include some source files even if they are not needed for a given test suite. So…
443 `python preprocess.py -f desc.txt`
445 `python preprocess.py -f desc_f16.txt -o Output_f16.pickle`
447 `python processTests.py -e`
449 `python processTests.py -e -f Output_f16.pickle`
455 You can use the [CMSIS build tools](https://github.com/Open-CMSIS-Pack/devtools) to build.
464 `C:\Keil_v5\ARM\FVP\MPS2_Cortex-M\FVP_MPS2_Cortex-M55_MDK.exe ^
465 -f configs/ARM_VHT_MPS2_M55_config.txt ^
470 The results generated in previous step can be processed with a Python script.
472 The `-f` option should be used to tell the script where to find the `Output.pickle` file if the scr…
474 python processResult.py -f Output.pickle -e -r result.txt
476 -e option is needed if the mode -e was used with processTests because the output has a different
477 format with or without -e option.
480 …re displayed with the test status (passed or failed). **Don't trust** those cycles for a benchmark.
490 python summaryBench.py -r build\result.txt
496 The Output.pickle file is used by default. It can be changed with -f option.
501 …1,"DSP:ComplexMaths",1,"225.3749999999999 + A * 0.7083333333333606 + B * 0.7083333333333641 + A*B …
504 The MAX column is the max of cycles computed for all values of A and B which were used for this ben…
511 python convertToOld.py -e AGroup
513 Output.pickle is used by default. It can be changed with -f option.
515 To add a to sqlite3 databse:
517 python addToDB.py -e AGroup
519 Output.pickle is used by default. It can be changed with -f option.
524 The script processTests and processResult must be used with additional option -e for the FPGA (embe…
561 and of course, the test must contain a line to dump the outputs.
565 So, if a test is in the middle of useful code, some part of the code will not execute.
574 * Allocate a memory manager
576 * Instantiate a pattern manager (linking IO and memory)
577 * Choose a test Runner (IORunner)
585 For a test suite MyClass, the scripts are generating an include file MyClass_decl.h
638 // benchmark. So a benchmark should not contain tests.
639 ASSERT_NEAR_EQ(ref,output,(float)1e-6);
644 Warning : in case of a benchmark the xxx.ptr() function calls should be done in the setup function …
688 To add a to sqlite3 database:
692 Output.pickle is used by default. It can be changed with -f option.
694 AGroup should be the class name of a Group in the desc.txt
698 For instance, we have a BasicBenchmarks group is desc.txt
707 Output.pickle is used by default. It can be changed with -f option.
709 A table BasicBenchmarks will be create and the benchmarks result for F32, Q31, Q15 and Q7 will be a…
715 The a table BasicMathsBenchmarksF32 will be created which is probably not what you want since the t…
717 The script addToRegDB.py is working on the same principle but using the regression csv to fill a re…
735 But then it means that to display the tables in a readable format by the user, some joins are neede…
739 examples.sql : how to do simple queries and join with the configuration columns to get a readable f…
741 …iff.sql : How to compute a performance ratio (max cycle and regression) based on a reference core …
747 …Arm AC5 compiler \_\_fp16 type (float16_t in CMSIS-DSP) can't be used as argument or return value …
751 In CMSIS-DSP, we want to keep the possibility of having float16_t as an argument.
756 * The corresponding float16_t tests are put in a different test file desc_f16.txt