1############################## 2Mailbox NS Agent Design Update 3############################## 4 5:Organization: Arm Limited 6:Contact: tf-m@lists.trustedfirmware.org 7 8********** 9Background 10********** 11The SPE component that maintains the non-secure clients' request is called 12'NS Agent' in TF-M. Besides the Trustzone-based isolation mechanism, there is 13one other isolation mechanism that implements individual PEs in physically 14isolated cores respectively. NSPE and SPE transfer non-secure client requests 15via inter-processor communication based on mailboxes. The component that 16handles inter-processor communication messages is called ``Mailbox NS Agent``. 17 18.. note:: 19 There may be hardware components and software solutions containing 'mailbox' 20 in their names. The concept ``mailbox`` in this document represent the 21 mechanism described above, which is not referring to the external concepts. 22 23When the first version ``Mailbox NS Agent`` was introduced, the generic FFM 24interrupt handling was not ready. Hence a customized solution 25``Multiple Core`` is implemented. This customized implementation: 26 27- Perform customized operations on SPM internal data in a deferred interrupt 28 handler. 29- Process mailbox operations as special cases in SPM common logic. 30 31These behaviours couple SPM tightly with mailbox logic, which bring issues for 32maintenance. To address the issue, an updated design shall: 33 34- Make SPM manage other components in a unified way (For example, it is 35 simpler for SPM if all non-SPM components under the IPC model act as 36 ``processes``.) 37- Can use FF-M compliant interrupt mechanism and APIs. 38 39Following the above guidelines makes the ``Mailbox NS Agent`` work like a 40``partition``. The agent has an endless loop and waits for signals, calls FFM 41API based on the parsing result on the communication messages. But there are 42still issues after looking closer to the requirements of the agent: 43 44- SPM treats FFM Client API caller's ID as the client ID. While the mailbox NS 45 agent may represent multiple non-secure clients. Hence it needs to tell 46 SPM which non-secure client it is representing, and the default FFM Client 47 API does not have such capability. 48- FFM Client API blocks caller before the call is replied; while the 49 mailbox NS Agent needs to respond to the non-secure interrupts in time. 50 Blocking while waiting for a reply may cause the non-secure communication 51 message not to be handled in time. 52 53Extra design items need to be added to address the issues. 54 55************* 56Design Update 57************* 58The below figure shows the overall design to cover various component types. 59NS Agents are the implementation-defined components that provide FF-M compliant 60Client API to the non-secure clients. Hence from the view of the non-secure 61clients, the FF-M client API behaviour follows the FF-M definition. And NS 62Agent needs customization in SPM since it has extra requirements compared to 63a generic secure partition. 64 65.. figure:: /design_docs/media/mailbox_ns_agent1.* 66 :align: center 67 :name: fig-mailbox1 68 :width: 70% 69 70 Component types and the callable API set 71 72.. note:: 73 3 non-SPM component types here: FFM-compliant Secure Partition 74 (aka ``partition``), Trustzone-based NS Agent (aka ``Trustzone NS Agent``) 75 and mailbox-based NS Agent (aka ``Mailbox NS Agent``). 76 ``Trustzone NS Agent`` is mentioned here for the comparison purpose. The 77 implementation details for this NS agent type is not introduced here. 78 79To make the programming model close to the FFM compliance, the 80``Mailbox NS Agent`` is designed as: 81 82- Working like a standard Secure Partition under the IPC model, has one 83 single thread, can call FFM standard API. 84- Having a manifest file to describe the attributes and resources and a 85 positive valued ``Partition ID`` in the manifest. 86 87Services rely on the ``client_id`` to apply policy-checking, hence SPM 88needs to know which ``client_id`` the mailbox NS Agent is representing when 89mailbox API is calling Client API. The standard API treats the caller as the 90client of the service, which means that a specific API is required to 91support identifying the represented non-secure client. SPM sets the non-secure 92``client_id`` into the message right at the moment the message is 93going to be sent. Before this point, SPM performs the call based on the 94agent's ID. 95 96This specifc ``Agent API`` is non-blocking, unlike the standard FF-M Client 97APIs. This can improve the communication efficiency between NS clients and 98mailbox NS agents. With this mechanism, extra signals and APIs for message 99acknowledges are also required. 100 101.. note:: 102 A standard Secure Partition gets errors when calling the ``Agent API``. 103 104Updated programming interfaces 105============================== 106These Client APIs are expanded from the standard Client APIs: 107 108- ``agent_psa_connect()`` is extended from ``psa_connect()``. 109- ``agent_psa_call()`` is extended from ``psa_call()``. 110 111And to cooperate with the changed behaviour of these APIs, extra defined 112signals and types are also involved. 113 114.. note:: 115 Namespace ``agent`` is involved for NS Agent callable API; namespace ``tfm`` 116 is involved for TF-M specific concepts. Even though ``agent`` is TF-M 117 specific at the current stage, it is proposed to be a common concept for 118 general FF-M compliant implementations, hence assigning ``agent`` for 119 proposed API and data structures. 120 121Input and output vectors 122======================== 123When non-secure clients call ``psa_call()``, a mailbox message containing 124``psa_call()`` parameters is delivered into ``Mailbox NS Agent`` and the agent 125needs to extract parameters from the message and then call ``agent_psa_call()``. 126Revisit the ``psa_call()`` prototype to see the parameters: 127 128.. code-block:: c 129 130 psa_status_t psa_call(psa_handle_t handle, 131 int32_t type, 132 const psa_invec *in_vec, 133 size_t in_len, 134 psa_outvec *out_vec, 135 size_t out_len); 136 137Interface ``agent_psa_call()`` has 4 arguments only, to avoid ABI complexity 138when more than 4 arguments get involved. Then input and output vectors must 139be squashed into a new type to achieve this squashing. There are several 140scenarios to be considered: 141 142- In the shared-memory-based mailbox scheme, the non-secure interface layer 143 puts ``psa_invec`` and ``psa_outvec`` instances and their members in the 144 shared memory. The pointers of these items are delivered to 145 ``Mailbox NS Agent`` for the agent's referencing. All vectors are 146 non-secure in this case. 147- Still, in the shared-memory-based mailbox scheme, but this time not all 148 parameters are prepared by non-secure clients. Some of the items are 149 allocated by the agent. Secure and non-secure vectors get mixed in this case. 150- In the scheme that a memory address can not be delivered (A serial 151 interface e.g.), ``Mailbox NS Agent`` allocates ``psa_invec`` and 152 ``psa_outvec`` in local memory to collect pointers and sizes of received 153 buffers. All the vectors are secure in this case. 154 155Based on these scenarios, and the case that an agent might access depending 156services with the agent itself's identifier and secure vectors, a straight 157conclusion is that the memory source information - secure or non-secure - for 158input and output vectors and their pointing memories need to be indicated by 159the agent so that SPM can perform a memory check towards given vectors. 160 161Most of the SPE platforms have the capability to identify if a given memory 162pointer is secure or not, which makes this indication look duplicated. But the 163indication is necessary for these scenarios: 164 165- The SPE platform identifies the memory source by the address mapped into 166 SPE's address space, but this mapping happens inside the agent instead of 167 SPM. It is the agent who receives the mailbox data and maps it, so it knows 168 which addresses need mapping and the others do not. 169- The SPE platform just does not have such capability. The addresses from the 170 mailbox can be treated as non-secure always but there are cases that the 171 agent itself needs to access services with its own memory instead of 172 representing the non-secure clients. 173 174To cover the above mentioned scenarios, guidelines are listed for input and 175output vector processing: 176 177- The agent needs to tell SPM where vectors and descriptors come from, to 178 assist SPM performs a proper memory checking. The source information is 179 encoded in the parameter ``control``. 180- When SPE platforms have the capability to identify the memory sources, 181 platforms can decide whether to skip the indication or not, in the HAL. 182 183A composition type is created for packing the vectors: 184 185.. code-block:: c 186 187 struct client_param_t { 188 int32_t ns_client_id_stateless; 189 const psa_invec *p_invecs; 190 psa_outvec *p_outvecs; 191 }; 192 193``ns_client_id_stateless`` indicates the non-secure client id when the client 194is accessing a stateless service. This member is ignored if the target service 195is a connection-based one. 196 197.. note:: 198 The vectors and non-secure client ID are recorded in the internal handle. 199 Hence it is safe to claim ``client_param_t`` instance as local variable. 200 201Agent-specific Client API 202========================= 203``agent_psa_connect()`` is the API added to support agent forwarding NS 204requests. 205 206.. code-block:: c 207 208 psa_handle_t agent_psa_connect(uint32_t sid, uint32_t version, 209 int32_t ns_client_id, const void *client_data); 210 211One extra parameter ``ns_client_id`` added to tell SPM which NS client the 212agent is representing when API gets called. It is recorded in the handle 213association data in SPM and requires to be a negative value; ZERO or positive 214values are invalid non-secure client IDs, SPM does not use these invalid IDs 215in the message. Instead, it puts the agent's ID into the messaging in this 216case. This mechanism can provide chances for the agents calling APIs for their 217own service accessing and API works asynchronously. 218 219As mentioned, the standard FFM Client service accessing API are blocked until 220the IPC message gets replied to. While this API returns immediately without 221waiting for acknowledgement. Unless an error occurred, these agent-specific 222API returns PSA_SUCCESS always. The replies for these access requests are 223always fetched initiative by the agent with a ``psa_get()``. 224 225.. code-block:: c 226 227 psa_status_t agent_psa_call(psa_handle_t handle, uint32_t control, 228 const struct client_param_t *params, 229 const void *client_data_stateless); 230 231Compared to the standard ``psa_call()``, this API: 232 233- Squashes the ``psa_invec``, ``psa_outvec``, and ``ns_client_id_stateless`` 234 into parameter ``params``. 235- One extra parameter ``client_data_stateless`` for ``agent_psa_call()`` stands 236 for the auxiliary data added. This member is ignored for connection-based 237 services because ``agent_psa_connect()`` already assigned one in the 238 connected handle. 239- Has a composited agrument ``control``. 240 241The encoding scheme for ``control``: 242 243====== ====== ================================================================ 244Bit(s) Name Description 245====== ====== ================================================================ 24627 NSIV `1`: Input vectors in non-secure memory. `0`: Secure memory. 24724-26 IVNUM Number of input vectors. 24819 NSOV `1`: Output vectors in non-secure memory. `0`: Secure memory. 24916-18 OVNUM Number of output vectors. 2500-15 type signed 16-bit service `type`. 251====== ====== ================================================================ 252 253.. note:: 254 ``control`` is a 32-bit unsigned integer and bits not mentioned in the 255 table are reserved for future usage. 256 257Agent-specific signal 258===================== 259To cooperate with the agent-specific API, one extra acknowledgement signal is 260defined: 261 262.. code-block:: c 263 264 #define ASYNC_MSG_REPLY (0x00000004u) 265 266This signal can be sent to agent type component only. An agent can call 267``psa_get()`` with this signal to get one acknowledged message. This signal is 268cleared when all queued messages for the agent have been retrieved using 269``psa_get()``. SPM assembles the information into agent provided message object. 270For the stateless handle, the internal handle object is freed after this 271``psa_get()`` call. The agent can know what kind of message is acknowledged by 272the ``type`` member in the ``psa_msg_t``, and the ``client_data`` passed in is 273put in member ``rhandle``. If no 'ASYNC_MSG_REPLY' signals pending, calling 274``psa_get()`` gets ``panic``. 275 276Code Example 277============ 278 279.. code-block:: c 280 281 /* 282 * The actual implementation can change this __customized_t freely, or 283 * discard this type and apply some in-house mechanism - the example 284 * here is to introduce how an agent works only. 285 */ 286 struct __customized_t { 287 int32_t type; 288 int32_t client_id; 289 psa_handle_t handle; 290 psa_handle_t status; 291 }; 292 293 void mailbox_main(void) 294 { 295 psa_signal_t signals; 296 psa_status_t status; 297 psa_msg_t msg; 298 struct client_param_t client_param; 299 struct __customized_t ns_msg; 300 301 while (1) { 302 signals = psa_wait(ALL, BLOCK); 303 304 if (signals & MAILBOX_INTERRUPT_SIGNAL) { 305 /* NS memory check needs to be performed. */ 306 __customized_platform_get_mail(&ns_msg); 307 308 /* 309 * MACRO 'SID', 'VER', 'NSID', 'INVEC_LEN', 'OUTVEC_LEN', and 310 * 'VECTORS' represent necessary information extraction from 311 * 'ns_msg', put MACRO names here and leave the details to the 312 * implementation. 313 */ 314 if (ns_msg.type == PSA_IPC_CONNECT) { 315 status = agent_psa_connect(SID(ns_msg), VER(ns_msg), 316 NSID(ns_msg), &ns_msg); 317 } else if (ns_msg.type == PSA_IPC_CLOSE) { 318 psa_close(ns_msg.handle); 319 } else { 320 /* Other types as call type and let API check errors. */ 321 client_param.ns_client_id_stateless = NSID(ns_msg); 322 323 /* 324 * Use MACRO to demonstrate two cases: local vector 325 * descriptor and direct descriptor forwarding. 326 */ 327 328 /* Point to vector pointers in ns_msg. */ 329 PACK_VECTOR_POINTERS(client_param, ns_msg); 330 status = agent_psa_call(ns_msg.handle, 331 PARAM_PACK(ns_msg.type, 332 INVEC_LEN(ns_msg), 333 OUTVEC_LEN(ns_msg))| 334 NSIV | NSOV, 335 &client_param, 336 &ns_msg); 337 } 338 /* 339 * The service access reply is always fetched by a later 340 * `psa_get` hence here only errors need to be dispatched. 341 */ 342 error_dispatch(status); 343 344 } else if (signals & ASYNC_MSG_REPLY) { 345 /* The handle is freed for stateless service after 'psa_get'. */ 346 status = psa_get(ASYNC_MSG_REPLY, &msg); 347 ms_msg = msg.rhandle; 348 ns_msg.status = status; 349 __customized_platform__send_mail(&ns_msg); 350 } 351 } 352 } 353 354.. note:: 355 ``__customized*`` API are implementation-specific APIs to be implemented by 356 the mailbox Agent developer. 357 358Customized manifest attribute 359============================= 360Two extra customized manifest attributes are added: 361 362============= ==================================================== 363Name Description 364============= ==================================================== 365ns_agent Indicate if manifest owner is an Agent. 366------------- ---------------------------------------------------- 367ns_client_ids Possible non-secure Client ID values (<0). 368============= ==================================================== 369 370Attribute 'ns_client_ids' can be a set of numbers, or it can use a range 371expression such as [min, max]. The tooling can detect ID overlap between 372multiple non-secure agents. 373 374.. note:: 375 Per-non-secure-client dependencies scheme is now assumed to be implemented 376 by agent customization. Feedback if there are requirements for a unified 377 scheme. 378 379*********************** 380Manifest tooling update 381*********************** 382The manifest for agents involves specific keys ('ns_agent' e.g.), these keys 383give hints about how to achieve out-of-FFM partitions which might be abused 384easily by developers, for example, claim partitions as agents. Some 385restrictions need to be applied in the manifest tool to limit the general 386secure service development referencing these keys. 387 388.. note:: 389 The limitations can mitigate the abuse but can't prevent it, as developers 390 own all the source code they are working with. 391 392One mechanism: adding a confirmation in the partition list file. 393 394.. parsed-literal:: 395 396 "description": "Non-Secure Mailbox Agent", 397 "manifest": "${CMAKE_SOURCE_DIR}/secure_fw/partitions/ns_agent_mailbox/ns_agent_mailbox.yaml", 398 "non_ffm_attributes": "ns_agent", "other_option", 399 400``non_ffm_attributes`` tells the manifest tool that ``ns_agent`` is valid 401in ns_agent_mailbox.ymal. Otherwise, the manifest tool reports an error when a 402non-agent service abuses ns_agent in its manifest. 403 404*********************************** 405Runtime programming characteristics 406*********************************** 407 408Mailbox agent shall not be blocked by Agent-specific APIs. It can be blocked when: 409 410- It is calling standard PSA Client APIs. 411- It is calling ``psa_wait()``. 412 413IDLE processing 414=============== 415Only ONE place is recommended to enter IDLE. The place is decided based on the 416system topologies: 417 418- If there is one Trustzone-based NSPE, this NSPE is the recommended place no 419 matter how many mailbox agents are in the system. 420- If there are only mailbox-based NSPEs, entering IDLE can happen in 421 one of the mailbox agents. 422 423The solution is: 424 425- An IDLE entering API is provided in SPRTL. 426- A partition without specific flag can't call this API. 427- The manifest tooling counts the partitions with this specific flag, and 428 assert errors when multiple instances are found. 429 430-------------- 431 432*Copyright (c) 2022-2023, Arm Limited. All rights reserved.* 433*Copyright (c) 2023 Cypress Semiconductor Corporation (an Infineon company) 434or an affiliate of Cypress Semiconductor Corporation. All rights reserved.* 435
