Lines Matching +full:data +full:- +full:valid +full:- +full:time
5 Zephyr Memory Storage is a new key-value storage system that is designed to work with all types
6 of non-volatile storage technologies. It supports classical on-chip NOR flash as well as new
8 data on these types of devices can be overwritten directly at any time.
12 ZMS divides the memory space into sectors (minimum 2), and each sector is filled with key-value
15 The key-value pair is divided into two parts:
17 - The key part is written in an ATE (Allocation Table Entry) called "ID-ATE" which is stored
19 - The value part is defined as "DATA" and is stored raw starting from the top of the sector
21 Additionally, for each sector we store at the last positions Header-ATEs which are ATEs that
25 collect the N+2 sector (where N is the current sector number) by moving the valid ATEs to the
37 .. list-table::
39 :header-rows: 1
41 * - Sector 0 (closed)
42 - Sector 1 (open)
43 - Sector 2 (empty)
44 * - Data_a0
45 - Data_b0
46 - Data_c0
47 * - Data_a1
48 - Data_b1
49 - Data_c1
50 * - Data_a2
51 - Data_b2
52 - Data_c2
53 * - GC_done
54 - .
55 - .
56 * - .
57 - .
58 - .
59 * - .
60 - .
61 - .
62 * - .
63 - ATE_b2
64 - ATE_c2
65 * - ATE_a2
66 - ATE_b1
67 - ATE_c1
68 * - ATE_a1
69 - ATE_b0
70 - ATE_c0
71 * - ATE_a0
72 - GC_done
73 - GC_done
74 * - Close (cyc=1)
75 - Close (cyc=1)
76 - Close (cyc=1)
77 * - Empty (cyc=1)
78 - Empty (cyc=2)
79 - Empty (cyc=2)
91 ``ID-ATE:`` are entries that contain a 32 bits Key and describe where the data is stored, its
94 ``Data:`` is the actual value associated to the ID-ATE
108 .. code-block:: c
115 * erase-blocks if the device has erase capabilities
128 As ZMS has a fast-forward write mechanism, we must find the last sector and the last pointer of
129 the entry where it stopped the last time.
134 ZMS ID-Data write
137 To avoid rewriting the same data with the same ID again, it must look in all the sectors if the
138 same ID exist then compares its data, if the data is identical no write is performed.
139 If we must perform a write, then an ATE and Data (if not a delete) are written in the sector.
140 If the sector is full (cannot hold the current data + ATE) we have to move to the next sector,
142 Data size that is smaller or equal to 8 bytes are written within the ATE.
144 ZMS ID/data read (with history)
147 By default it looks for the last data with the same ID by browsing through all stored ATEs from
148 the most recent ones to the oldest ones. If it finds a valid ATE with a matching ID it retrieves
149 its data and returns the number of bytes that were read.
150 If history count is provided that is different than 0, older data with same ID is retrieved.
156 However, this operation is very time consuming and needs to browse all valid ATEs in all sectors
157 of the partition and for each valid ATE try to find if an older one exist.
158 It is not recommended for application to use this function often, as it is time consuming and
167 To become valid, an ATE must have the same cycle counter as the one stored in the empty ATE.
168 Each time an ATE is moved from a sector to another it must get the cycle counter of the
179 When closing a sector, all the remaining space that has not been used is filled with garbage data
180 to avoid having old ATEs with a valid cycle counter.
188 This operation is time consuming and it will cause some applications to not meet their real time
200 .. code-block:: c
204 uint8_t cycle_cnt; /* cycle counter for non-erasable devices */
205 uint16_t len; /* data len within sector */
206 uint32_t id; /* data id */
208 uint8_t data[8]; /* used to store small size data */
210 uint32_t offset; /* data offset within sector */
212 uint32_t data_crc; /* crc for data */
221 .. note:: The CRC of the data is checked only when the whole the element is read.
222 The CRC of the data is not checked for a partial read, as it is computed for the whole element.
225 will make all existing data invalid.
227 .. _free-space:
229 Available space for user data (key-value pairs)
235 Key-value pairs and keep one sector empty to be able to launch GC.
238 .. note:: The maximum single data length that could be written at once in a sector is 64K
241 Small data values
244 Values smaller than 8 bytes will be stored within the entry (ATE) itself, without writing data
247 store data is computed in this scenario as :
251 \small\frac{(NUM\_SECTORS - 1) \times (SECTOR\_SIZE - (5 \times ATE\_SIZE))}{2}
263 For example for 4 sectors of 1024 bytes, free space for data is :math:`\frac{3 \times 944}{2} = 141…
265 Large data values
268 Large data values ( > 8 bytes) are stored separately at the top of the sector.
270 the data. But we can take into account that for N bytes of data (N > 8 bytes) an additional
275 For a partition that has 4 sectors of 1024 bytes and for data size of 64 bytes.
277 Each Key-value pair needs an extra 16 bytes for ATE which makes it possible to store 11 pairs
279 Total data that could be stored in this partition for this case is :math:`11 \times 3 \times 64 = 2…
281 .. _wear-leveling:
287 Using storage systems that rely on an erase-value (NVS as an example) will need to emulate the
293 As an example, to erase a 4096 bytes sector on a non-erasable device using NVS, 256 flash writes
294 must be performed (supposing that write-block-size=16 bytes), while using ZMS only 1 write of
301 data (including extra headers) that could be written in the storage.
303 See :ref:`free-space`.
310 Flash devices are erased one page at a time as part of their functional behavior (otherwise
311 memory cells cannot be overwritten) and for non-erasable storage devices memory cells can be
318 As we have 944 bytes available for ATEs for each sector, and because ZMS is a fast-forward
322 In addition to the normal writes, garbage collector will move the still valid data from old
324 As we are using the same ID and a big partition size, no data will be moved by the garbage
330 typical set of data.
331 For id/data pair with data <= 8 bytes, effective_size is 16 bytes
332 For id/data pair with data > 8 bytes, effective_size is 16 bytes + sizeof(data)
333 Let's suppose that total_effective_size is the total size of the set of data that is written in
335 having the garbage collector moving blocks all the time.
345 ``SECTOR_EFFECTIVE_SIZE``: is the size sector - header_size(80 bytes)
351 ``TOTAL_EFFECTIVE_SIZE``: Total effective size of the set of written data
353 ``WR_MIN``: Number of writes of the set of data per minute
364 --------
365 - Supports non-erasable devices (only one write operation to erase a sector)
366 - Supports large partition size and sector size (64 bits address space)
367 - Supports 32-bit IDs to store ID/Value pairs
368 - Small sized data ( <= 8 bytes) are stored in the ATE itself
369 - Built-in Data CRC32 (included in the ATE)
370 - Versioning of ZMS (to handle future evolution)
371 - Supports large write-block-size (Only for platforms that need this)
376 - Add multiple format ATE support to be able to use ZMS with different ATE formats that satisfies
378 - Add the possibility to skip garbage collector for some application usage where ID/value pairs
381 - Divide IDs into namespaces and allocate IDs on demand from application to handle collisions
383 - Add the possibility to retrieve the wear out value of the device based on the cycle count value
384 - Add a recovery function that can recover a storage partition if something went wrong
385 - Add a library/application to allow migration from NVS entries to ZMS entries
386 - Add the possibility to force formatting the storage partition to the ZMS format if something
392 but simpler, non-hierarchical ones).
398 - If you are using a non-erasable technology device like RRAM or MRAM, :ref:`ZMS <zms_api>` is defi…
401 - For devices with large write_block_size and/or needs a sector size that is different than the
404 - For classical flash technology devices, :ref:`NVS <nvs_api>` is recommended as it has low footpri…
408 - If your application needs more than 64K IDs for storage, :ref:`ZMS <zms_api>` is recommended here…
409 has a 32-bit ID field.
410 - If your application is working in a FIFO mode (First-in First-out) then :ref:`FCB <fcb_api>` is
416 life expectancy of the device described in this section: :ref:`wear-leveling`.
424 - The total size of the storage partition should be well dimensioned to achieve the best
427 in the documentation. See :ref:`free-space`.
428 We recommend choosing a storage partition that can hold double the size of the key-value pairs
430 - The size of a sector needs to be dimensioned to hold the maximum data length that will be stored.
435 - For some subsystems like :ref:`Settings <settings_api>`, all path-value pairs are split into two …
437 - Using small data to store in the ZMS entries can increase the performance, as this data is
445 - When using ZMS API directly, the recommended cache size should be, at least, equal to
447 - Each additional cache entry will add 8 bytes to your RAM usage. Cache size should be carefully
449 - If you use ZMS through :ref:`Settings <settings_api>`, you have to take into account that each Se…
456 A sample of how ZMS can be used is supplied in :zephyr:code-sample:`zms`.