1Non-volatile storage library 2============================ 3 4:link_to_translation:`zh_CN:[中文]` 5 6Introduction 7------------ 8 9Non-volatile storage (NVS) library is designed to store key-value pairs in flash. This section introduces some concepts used by NVS. 10 11Underlying storage 12^^^^^^^^^^^^^^^^^^ 13 14Currently, NVS uses a portion of main flash memory through the :ref:`esp_partition <flash-partition-apis>` API. The library uses all the partitions with ``data`` type and ``nvs`` subtype. The application can choose to use the partition with the label ``nvs`` through the :cpp:func:`nvs_open` API function or any other partition by specifying its name using the :cpp:func:`nvs_open_from_partition` API function. 15 16Future versions of this library may have other storage backends to keep data in another flash chip (SPI or I2C), RTC, FRAM, etc. 17 18.. note:: if an NVS partition is truncated (for example, when the partition table layout is changed), its contents should be erased. ESP-IDF build system provides a ``idf.py erase-flash`` target to erase all contents of the flash chip. 19 20.. note:: NVS works best for storing many small values, rather than a few large values of the type 'string' and 'blob'. If you need to store large blobs or strings, consider using the facilities provided by the FAT filesystem on top of the wear levelling library. 21 22 23Keys and values 24^^^^^^^^^^^^^^^ 25 26NVS operates on key-value pairs. Keys are ASCII strings; the maximum key length is currently 15 characters. Values can have one of the following types: 27 28- integer types: ``uint8_t``, ``int8_t``, ``uint16_t``, ``int16_t``, ``uint32_t``, ``int32_t``, ``uint64_t``, ``int64_t`` 29- zero-terminated string 30- variable length binary data (blob) 31 32.. note:: 33 34 String values are currently limited to 4000 bytes. This includes the null terminator. Blob values are limited to 508,000 bytes or 97.6% of the partition size - 4000 bytes, whichever is lower. 35 36Additional types, such as ``float`` and ``double`` might be added later. 37 38Keys are required to be unique. Assigning a new value to an existing key works as follows: 39 40- If the new value is of the same type as the old one, value is updated. 41- If the new value has a different data type, an error is returned. 42 43Data type check is also performed when reading a value. An error is returned if the data type of the read operation does not match the data type of the value. 44 45 46Namespaces 47^^^^^^^^^^ 48 49To mitigate potential conflicts in key names between different components, NVS assigns each key-value pair to one of namespaces. Namespace names follow the same rules as key names, i.e., the maximum length is 15 characters. Namespace name is specified in the :cpp:func:`nvs_open` or :cpp:type:`nvs_open_from_partition` call. This call returns an opaque handle, which is used in subsequent calls to the ``nvs_get_*``, ``nvs_set_*``, and :cpp:func:`nvs_commit` functions. This way, a handle is associated with a namespace, and key names will not collide with same names in other namespaces. Please note that the namespaces with the same name in different NVS partitions are considered as separate namespaces. 50 51NVS iterators 52^^^^^^^^^^^^^ 53 54Iterators allow to list key-value pairs stored in NVS, based on specified partition name, namespace, and data type. 55 56There are the following functions available: 57 58- :cpp:func:`nvs_entry_find` returns an opaque handle, which is used in subsequent calls to the :cpp:func:`nvs_entry_next` and :cpp:func:`nvs_entry_info` functions. 59- :cpp:func:`nvs_entry_next` returns iterator to the next key-value pair. 60- :cpp:func:`nvs_entry_info` returns information about each key-value pair 61 62If none or no other key-value pair was found for given criteria, :cpp:func:`nvs_entry_find` and :cpp:func:`nvs_entry_next` return NULL. In that case, the iterator does not have to be released. If the iterator is no longer needed, you can release it by using the function :cpp:func:`nvs_release_iterator`. 63 64 65Security, tampering, and robustness 66^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 67 68NVS is not directly compatible with the {IDF_TARGET_NAME} flash encryption system. However, data can still be stored in encrypted form if NVS encryption is used together with {IDF_TARGET_NAME} flash encryption. Please refer to :ref:`nvs_encryption` for more details. 69 70If NVS encryption is not used, it is possible for anyone with physical access to the flash chip to alter, erase, or add key-value pairs. With NVS encryption enabled, it is not possible to alter or add a key-value pair and get recognized as a valid pair without knowing corresponding NVS encryption keys. However, there is no tamper-resistance against the erase operation. 71 72The library does try to recover from conditions when flash memory is in an inconsistent state. In particular, one should be able to power off the device at any point and time and then power it back on. This should not result in loss of data, except for the new key-value pair if it was being written at the moment of powering off. The library should also be able to initialize properly with any random data present in flash memory. 73 74 75.. _nvs_encryption: 76 77NVS Encryption 78-------------- 79 80Data stored in NVS partitions can be encrypted using AES-XTS in the manner similar to the one mentioned in disk encryption standard IEEE P1619. For the purpose of encryption, each entry is treated as one `sector` and relative address of the entry (w.r.t. partition-start) is fed to the encryption algorithm as `sector-number`. The NVS Encryption can be enabled by enabling :ref:`CONFIG_NVS_ENCRYPTION`. The keys required for NVS encryption are stored in yet another partition, which is protected using :doc:`Flash Encryption <../../security/flash-encryption>`. Therefore, enabling :doc:`Flash Encryption <../../security/flash-encryption>` is a prerequisite for NVS encryption. 81 82The NVS Encryption is enabled by default when :doc:`Flash Encryption <../../security/flash-encryption>` is enabled. This is done because Wi-Fi driver stores credentials (like SSID and passphrase) in the default NVS partition. It is important to encrypt them as default choice if platform level encryption is already enabled. 83 84For using NVS encryption, the partition table must contain the :ref:`nvs_key_partition`. Two partition tables containing the :ref:`nvs_key_partition` are provided for NVS encryption under the partition table option (menuconfig->Partition Table). They can be selected with the project configuration menu (``idf.py menuconfig``). Please refer to the example :example:`security/flash_encryption` for how to configure and use NVS encryption feature. 85 86.. _nvs_key_partition: 87 88NVS key partition 89^^^^^^^^^^^^^^^^^ 90 91 An application requiring NVS encryption support needs to be compiled with a key-partition of the type `data` and subtype `key`. This partition should be marked as `encrypted` and its size should be the minimum partition size (4KB). Refer to :doc:`Partition Tables <../../api-guides/partition-tables>` for more details. Two additional partition tables which contain the :ref:`nvs_key_partition` are provided under the partition table option (menuconfig->Partition Table). They can be directly used for :ref:`nvs_encryption`. The structure of these partitions is depicted below. 92 93.. highlight:: none 94 95:: 96 97 +-----------+--------------+-------------+----+ 98 | XTS encryption key (32) | 99 +---------------------------------------------+ 100 | XTS tweak key (32) | 101 +---------------------------------------------+ 102 | CRC32 (4) | 103 +---------------------------------------------+ 104 105The XTS encryption keys in the :ref:`nvs_key_partition` can be generated in one of the following two ways. 106 1071. Generate the keys on the ESP chip: 108 109 When NVS encryption is enabled the :cpp:func:`nvs_flash_init` API function can be used to initialize the encrypted default NVS partition. The API function internally generates the XTS encryption keys on the ESP chip. The API function finds the first :ref:`nvs_key_partition`. Then the API function automatically generates and stores the NVS keys in that partition by making use of the :cpp:func:`nvs_flash_generate_keys` API function provided by :component_file:`nvs_flash/include/nvs_flash.h`. New keys are generated and stored only when the respective key partiton is empty. The same key partition can then be used to read the security configurations for initializing a custom encrypted NVS partition with help of :cpp:func:`nvs_flash_secure_init_partition`. 110 111 The API functions :cpp:func:`nvs_flash_secure_init` and :cpp:func:`nvs_flash_secure_init_partition` do not generate the keys internally. When these API functions are used for initializing encrypted NVS partitions, the keys can be generated after startup using the :cpp:func:`nvs_flash_generate_keys` API function provided by ``nvs_flash.h``. The API function will then write those keys onto the key-partition in encrypted form. 112 1132. Use pre-generated key partition: 114 115 This option will be required by the user when keys in the :ref:`nvs_key_partition` are not generated by the application. The :ref:`nvs_key_partition` containing the XTS encryption keys can be generated with the help of :doc:`NVS Partition Generator Utility</api-reference/storage/nvs_partition_gen>`. Then the user can store the pre generated key partition on the flash with help of the following two commands: 116 117 i) Build and flash the partition table 118 :: 119 120 idf.py partition-table partition-table-flash 121 122 ii) Store the keys in the :ref:`nvs_key_partition` (on the flash) with the help of :component_file:`parttool.py<partition_table/parttool.py>` (see Partition Tool section in :doc:`partition-tables </api-guides/partition-tables>` for more details) 123 :: 124 125 parttool.py --port /dev/ttyUSB0 --partition-table-offset "nvs_key partition offset" write_partition --partition-name="name of nvs_key partition" --input "nvs_key partition" 126 127Since the key partition is marked as `encrypted` and :doc:`Flash Encryption <../../security/flash-encryption>` is enabled, the bootloader will encrypt this partition using flash encryption key on the first boot. 128 129It is possible for an application to use different keys for different NVS partitions and thereby have multiple key-partitions. However, it is a responsibility of the application to provide correct key-partition/keys for the purpose of encryption/decryption. 130 131Encrypted Read/Write 132^^^^^^^^^^^^^^^^^^^^ 133 134The same NVS API functions ``nvs_get_*`` or ``nvs_set_*`` can be used for reading of, and writing to an encrypted nvs partition as well. 135 136**Encrypt the default NVS partition:** 137To enable encryption for the default NVS partition no additional steps are necessary. When :ref:`CONFIG_NVS_ENCRYPTION` is enabled, the :cpp:func:`nvs_flash_init` API function internally performs some additional steps using the first :ref:`nvs_key_partition` found to enable encryption for the default NVS partition (refer to the API documentation for more details). Alternatively, :cpp:func:`nvs_flash_secure_init` API function can also be used to enable encryption for the default NVS partition. 138 139**Encrypt a custom NVS partition:** 140To enable encryption for a custom NVS partition, :cpp:func:`nvs_flash_secure_init_partition` API function is used instead of :cpp:func:`nvs_flash_init_partition`. 141 142When :cpp:func:`nvs_flash_secure_init` and :cpp:func:`nvs_flash_secure_init_partition` API functions are used, the applications are expected to follow the steps below in order to perform NVS read/write operations with encryption enabled. 143 144 1. Find key partition and NVS data partition using ``esp_partition_find*`` API functions. 145 2. Populate the :cpp:type:`nvs_sec_cfg_t` struct using the :cpp:func:`nvs_flash_read_security_cfg` or :cpp:func:`nvs_flash_generate_keys` API functions. 146 3. Initialise NVS flash partition using the :cpp:func:`nvs_flash_secure_init` or :cpp:func:`nvs_flash_secure_init_partition` API functions. 147 4. Open a namespace using the :cpp:func:`nvs_open` or :cpp:func:`nvs_open_from_partition` API functions. 148 5. Perform NVS read/write operations using ``nvs_get_*`` or ``nvs_set_*``. 149 6. Deinitialise an NVS partition using :cpp:func:`nvs_flash_deinit`. 150 151NVS Partition Generator Utility 152------------------------------- 153 154This utility helps generate NVS partition binary files which can be flashed separately on a dedicated partition via a flashing utility. Key-value pairs to be flashed onto the partition can be provided via a CSV file. For more details, please refer to :doc:`NVS Partition Generator Utility <nvs_partition_gen>`. 155 156Application Example 157------------------- 158 159You can find code examples in the :example:`storage` directory of ESP-IDF examples: 160 161:example:`storage/nvs_rw_value` 162 163 Demonstrates how to read a single integer value from, and write it to NVS. 164 165 The value checked in this example holds the number of the {IDF_TARGET_NAME} module restarts. The value's function as a counter is only possible due to its storing in NVS. 166 167 The example also shows how to check if a read / write operation was successful, or if a certain value has not been initialized in NVS. The diagnostic procedure is provided in plain text to help you track the program flow and capture any issues on the way. 168 169:example:`storage/nvs_rw_blob` 170 171 Demonstrates how to read a single integer value and a blob (binary large object), and write them to NVS to preserve this value between {IDF_TARGET_NAME} module restarts. 172 173 * value - tracks the number of the {IDF_TARGET_NAME} module soft and hard restarts. 174 * blob - contains a table with module run times. The table is read from NVS to dynamically allocated RAM. A new run time is added to the table on each manually triggered soft restart, and then the added run time is written to NVS. Triggering is done by pulling down GPIO0. 175 176 The example also shows how to implement the diagnostic procedure to check if the read / write operation was successful. 177 178:example:`storage/nvs_rw_value_cxx` 179 180 This example does exactly the same as :example:`storage/nvs_rw_value`, except that it uses the C++ NVS handle class. 181 182Internals 183--------- 184 185Log of key-value pairs 186^^^^^^^^^^^^^^^^^^^^^^ 187 188NVS stores key-value pairs sequentially, with new key-value pairs being added at the end. When a value of any given key has to be updated, a new key-value pair is added at the end of the log and the old key-value pair is marked as erased. 189 190Pages and entries 191^^^^^^^^^^^^^^^^^ 192 193NVS library uses two main entities in its operation: pages and entries. Page is a logical structure which stores a portion of the overall log. Logical page corresponds to one physical sector of flash memory. Pages which are in use have a *sequence number* associated with them. Sequence numbers impose an ordering on pages. Higher sequence numbers correspond to pages which were created later. Each page can be in one of the following states: 194 195Empty/uninitialized 196 Flash storage for the page is empty (all bytes are ``0xff``). Page is not used to store any data at this point and does not have a sequence number. 197 198Active 199 Flash storage is initialized, page header has been written to flash, page has a valid sequence number. Page has some empty entries and data can be written there. No more than one page can be in this state at any given moment. 200 201Full 202 Flash storage is in a consistent state and is filled with key-value pairs. 203 Writing new key-value pairs into this page is not possible. It is still possible to mark some key-value pairs as erased. 204 205Erasing 206 Non-erased key-value pairs are being moved into another page so that the current page can be erased. This is a transient state, i.e., page should never stay in this state at the time when any API call returns. In case of a sudden power off, the move-and-erase process will be completed upon the next power-on. 207 208Corrupted 209 Page header contains invalid data, and further parsing of page data was canceled. Any items previously written into this page will not be accessible. The corresponding flash sector will not be erased immediately and will be kept along with sectors in *uninitialized* state for later use. This may be useful for debugging. 210 211Mapping from flash sectors to logical pages does not have any particular order. The library will inspect sequence numbers of pages found in each flash sector and organize pages in a list based on these numbers. 212 213:: 214 215 +--------+ +--------+ +--------+ +--------+ 216 | Page 1 | | Page 2 | | Page 3 | | Page 4 | 217 | Full +---> | Full +---> | Active | | Empty | <- states 218 | #11 | | #12 | | #14 | | | <- sequence numbers 219 +---+----+ +----+---+ +----+---+ +---+----+ 220 | | | | 221 | | | | 222 | | | | 223 +---v------+ +-----v----+ +------v---+ +------v---+ 224 | Sector 3 | | Sector 0 | | Sector 2 | | Sector 1 | <- physical sectors 225 +----------+ +----------+ +----------+ +----------+ 226 227Structure of a page 228^^^^^^^^^^^^^^^^^^^ 229 230For now, we assume that flash sector size is 4096 bytes and that {IDF_TARGET_NAME} flash encryption hardware operates on 32-byte blocks. It is possible to introduce some settings configurable at compile-time (e.g., via menuconfig) to accommodate flash chips with different sector sizes (although it is not clear if other components in the system, e.g., SPI flash driver and SPI flash cache can support these other sizes). 231 232Page consists of three parts: header, entry state bitmap, and entries themselves. To be compatible with {IDF_TARGET_NAME} flash encryption, the entry size is 32 bytes. For integer types, an entry holds one key-value pair. For strings and blobs, an entry holds part of key-value pair (more on that in the entry structure description). 233 234The following diagram illustrates the page structure. Numbers in parentheses indicate the size of each part in bytes. 235 236:: 237 238 +-----------+--------------+-------------+-------------------------+ 239 | State (4) | Seq. no. (4) | version (1) | Unused (19) | CRC32 (4) | Header (32) 240 +-----------+--------------+-------------+-------------------------+ 241 | Entry state bitmap (32) | 242 +------------------------------------------------------------------+ 243 | Entry 0 (32) | 244 +------------------------------------------------------------------+ 245 | Entry 1 (32) | 246 +------------------------------------------------------------------+ 247 / / 248 / / 249 +------------------------------------------------------------------+ 250 | Entry 125 (32) | 251 +------------------------------------------------------------------+ 252 253Page header and entry state bitmap are always written to flash unencrypted. Entries are encrypted if flash encryption feature of {IDF_TARGET_NAME} is used. 254 255Page state values are defined in such a way that changing state is possible by writing 0 into some of the bits. Therefore it is not necessary to erase the page to change its state unless that is a change to the *erased* state. 256 257The version field in the header reflects the NVS format version used. For backward compatibility reasons, it is decremented for every version upgrade starting at 0xff (i.e., 0xff for version-1, 0xfe for version-2 and so on). 258 259CRC32 value in the header is calculated over the part which does not include a state value (bytes 4 to 28). The unused part is currently filled with ``0xff`` bytes. 260 261The following sections describe the structure of entry state bitmap and entry itself. 262 263Entry and entry state bitmap 264^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 265 266Each entry can be in one of the following three states represented with two bits in the entry state bitmap. The final four bits in the bitmap (256 - 2 * 126) are not used. 267 268Empty (2'b11) 269 Nothing is written into the specific entry yet. It is in an uninitialized state (all bytes are ``0xff``). 270 271Written (2'b10) 272 A key-value pair (or part of key-value pair which spans multiple entries) has been written into the entry. 273 274Erased (2'b00) 275 A key-value pair in this entry has been discarded. Contents of this entry will not be parsed anymore. 276 277 278.. _structure_of_entry: 279 280Structure of entry 281^^^^^^^^^^^^^^^^^^ 282 283For values of primitive types (currently integers from 1 to 8 bytes long), entry holds one key-value pair. For string and blob types, entry holds part of the whole key-value pair. For strings, in case when a key-value pair spans multiple entries, all entries are stored in the same page. Blobs are allowed to span over multiple pages by dividing them into smaller chunks. For tracking these chunks, an additional fixed length metadata entry is stored called "blob index". Earlier formats of blobs are still supported (can be read and modified). However, once the blobs are modified, they are stored using the new format. 284 285:: 286 287 +--------+----------+----------+----------------+-----------+---------------+----------+ 288 | NS (1) | Type (1) | Span (1) | ChunkIndex (1) | CRC32 (4) | Key (16) | Data (8) | 289 +--------+----------+----------+----------------+-----------+---------------+----------+ 290 291 Primitive +--------------------------------+ 292 +--------> | Data (8) | 293 | Types +--------------------------------+ 294 +-> Fixed length -- 295 | | +---------+--------------+---------------+-------+ 296 | +--------> | Size(4) | ChunkCount(1)| ChunkStart(1) | Rsv(2)| 297 Data format ---+ Blob Index +---------+--------------+---------------+-------+ 298 | 299 | +----------+---------+-----------+ 300 +-> Variable length --> | Size (2) | Rsv (2) | CRC32 (4) | 301 (Strings, Blob Data) +----------+---------+-----------+ 302 303 304Individual fields in entry structure have the following meanings: 305 306NS 307 Namespace index for this entry. For more information on this value, see the section on namespaces implementation. 308 309Type 310 One byte indicating the value data type. See the :cpp:type:`ItemType` enumeration in :component_file:`nvs_flash/include/nvs_handle.hpp` for possible values. 311 312Span 313 Number of entries used by this key-value pair. For integer types, this is equal to 1. For strings and blobs, this depends on value length. 314 315ChunkIndex 316 Used to store the index of a blob-data chunk for blob types. For other types, this should be ``0xff``. 317 318CRC32 319 Checksum calculated over all the bytes in this entry, except for the CRC32 field itself. 320 321Key 322 Zero-terminated ASCII string containing a key name. Maximum string length is 15 bytes, excluding a zero terminator. 323 324Data 325 For integer types, this field contains the value itself. If the value itself is shorter than 8 bytes, it is padded to the right, with unused bytes filled with ``0xff``. 326 327 For "blob index" entry, these 8 bytes hold the following information about data-chunks: 328 329 - Size 330 (Only for blob index.) Size, in bytes, of complete blob data. 331 332 - ChunkCount 333 (Only for blob index.) Total number of blob-data chunks into which the blob was divided during storage. 334 335 - ChunkStart 336 (Only for blob index.) ChunkIndex of the first blob-data chunk of this blob. Subsequent chunks have chunkIndex incrementally allocated (step of 1). 337 338 For string and blob data chunks, these 8 bytes hold additional data about the value, which are described below: 339 340 - Size 341 (Only for strings and blobs.) Size, in bytes, of actual data. For strings, this includes zero terminators. 342 343 - CRC32 344 (Only for strings and blobs.) Checksum calculated over all bytes of data. 345 346Variable length values (strings and blobs) are written into subsequent entries, 32 bytes per entry. The `Span` field of the first entry indicates how many entries are used. 347 348 349Namespaces 350^^^^^^^^^^ 351 352As mentioned above, each key-value pair belongs to one of the namespaces. Namespace identifiers (strings) are stored as keys of key-value pairs in namespace with index 0. Values corresponding to these keys are indexes of these namespaces. 353 354:: 355 356 +-------------------------------------------+ 357 | NS=0 Type=uint8_t Key="wifi" Value=1 | Entry describing namespace "wifi" 358 +-------------------------------------------+ 359 | NS=1 Type=uint32_t Key="channel" Value=6 | Key "channel" in namespace "wifi" 360 +-------------------------------------------+ 361 | NS=0 Type=uint8_t Key="pwm" Value=2 | Entry describing namespace "pwm" 362 +-------------------------------------------+ 363 | NS=2 Type=uint16_t Key="channel" Value=20 | Key "channel" in namespace "pwm" 364 +-------------------------------------------+ 365 366 367Item hash list 368^^^^^^^^^^^^^^ 369 370To reduce the number of reads from flash memory, each member of the Page class maintains a list of pairs: item index; item hash. This list makes searches much quicker. Instead of iterating over all entries, reading them from flash one at a time, `Page::findItem` first performs a search for the item hash in the hash list. This gives the item index within the page if such an item exists. Due to a hash collision, it is possible that a different item will be found. This is handled by falling back to iteration over items in flash. 371 372Each node in the hash list contains a 24-bit hash and 8-bit item index. Hash is calculated based on item namespace, key name, and ChunkIndex. CRC32 is used for calculation; the result is truncated to 24 bits. To reduce the overhead for storing 32-bit entries in a linked list, the list is implemented as a double-linked list of arrays. Each array holds 29 entries, for the total size of 128 bytes, together with linked list pointers and a 32-bit count field. The minimum amount of extra RAM usage per page is therefore 128 bytes; maximum is 640 bytes. 373 374API Reference 375------------- 376 377.. include-build-file:: inc/nvs_flash.inc 378 379.. include-build-file:: inc/nvs.inc 380
