1.. SPDX-License-Identifier: GPL-2.0 2 3============ 4Timestamping 5============ 6 7 81. Control Interfaces 9===================== 10 11The interfaces for receiving network packages timestamps are: 12 13SO_TIMESTAMP 14 Generates a timestamp for each incoming packet in (not necessarily 15 monotonic) system time. Reports the timestamp via recvmsg() in a 16 control message in usec resolution. 17 SO_TIMESTAMP is defined as SO_TIMESTAMP_NEW or SO_TIMESTAMP_OLD 18 based on the architecture type and time_t representation of libc. 19 Control message format is in struct __kernel_old_timeval for 20 SO_TIMESTAMP_OLD and in struct __kernel_sock_timeval for 21 SO_TIMESTAMP_NEW options respectively. 22 23SO_TIMESTAMPNS 24 Same timestamping mechanism as SO_TIMESTAMP, but reports the 25 timestamp as struct timespec in nsec resolution. 26 SO_TIMESTAMPNS is defined as SO_TIMESTAMPNS_NEW or SO_TIMESTAMPNS_OLD 27 based on the architecture type and time_t representation of libc. 28 Control message format is in struct timespec for SO_TIMESTAMPNS_OLD 29 and in struct __kernel_timespec for SO_TIMESTAMPNS_NEW options 30 respectively. 31 32IP_MULTICAST_LOOP + SO_TIMESTAMP[NS] 33 Only for multicast:approximate transmit timestamp obtained by 34 reading the looped packet receive timestamp. 35 36SO_TIMESTAMPING 37 Generates timestamps on reception, transmission or both. Supports 38 multiple timestamp sources, including hardware. Supports generating 39 timestamps for stream sockets. 40 41 421.1 SO_TIMESTAMP (also SO_TIMESTAMP_OLD and SO_TIMESTAMP_NEW) 43------------------------------------------------------------- 44 45This socket option enables timestamping of datagrams on the reception 46path. Because the destination socket, if any, is not known early in 47the network stack, the feature has to be enabled for all packets. The 48same is true for all early receive timestamp options. 49 50For interface details, see `man 7 socket`. 51 52Always use SO_TIMESTAMP_NEW timestamp to always get timestamp in 53struct __kernel_sock_timeval format. 54 55SO_TIMESTAMP_OLD returns incorrect timestamps after the year 2038 56on 32 bit machines. 57 581.2 SO_TIMESTAMPNS (also SO_TIMESTAMPNS_OLD and SO_TIMESTAMPNS_NEW) 59------------------------------------------------------------------- 60 61This option is identical to SO_TIMESTAMP except for the returned data type. 62Its struct timespec allows for higher resolution (ns) timestamps than the 63timeval of SO_TIMESTAMP (ms). 64 65Always use SO_TIMESTAMPNS_NEW timestamp to always get timestamp in 66struct __kernel_timespec format. 67 68SO_TIMESTAMPNS_OLD returns incorrect timestamps after the year 2038 69on 32 bit machines. 70 711.3 SO_TIMESTAMPING (also SO_TIMESTAMPING_OLD and SO_TIMESTAMPING_NEW) 72---------------------------------------------------------------------- 73 74Supports multiple types of timestamp requests. As a result, this 75socket option takes a bitmap of flags, not a boolean. In:: 76 77 err = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, &val, sizeof(val)); 78 79val is an integer with any of the following bits set. Setting other 80bit returns EINVAL and does not change the current state. 81 82The socket option configures timestamp generation for individual 83sk_buffs (1.3.1), timestamp reporting to the socket's error 84queue (1.3.2) and options (1.3.3). Timestamp generation can also 85be enabled for individual sendmsg calls using cmsg (1.3.4). 86 87 881.3.1 Timestamp Generation 89^^^^^^^^^^^^^^^^^^^^^^^^^^ 90 91Some bits are requests to the stack to try to generate timestamps. Any 92combination of them is valid. Changes to these bits apply to newly 93created packets, not to packets already in the stack. As a result, it 94is possible to selectively request timestamps for a subset of packets 95(e.g., for sampling) by embedding an send() call within two setsockopt 96calls, one to enable timestamp generation and one to disable it. 97Timestamps may also be generated for reasons other than being 98requested by a particular socket, such as when receive timestamping is 99enabled system wide, as explained earlier. 100 101SOF_TIMESTAMPING_RX_HARDWARE: 102 Request rx timestamps generated by the network adapter. 103 104SOF_TIMESTAMPING_RX_SOFTWARE: 105 Request rx timestamps when data enters the kernel. These timestamps 106 are generated just after a device driver hands a packet to the 107 kernel receive stack. 108 109SOF_TIMESTAMPING_TX_HARDWARE: 110 Request tx timestamps generated by the network adapter. This flag 111 can be enabled via both socket options and control messages. 112 113SOF_TIMESTAMPING_TX_SOFTWARE: 114 Request tx timestamps when data leaves the kernel. These timestamps 115 are generated in the device driver as close as possible, but always 116 prior to, passing the packet to the network interface. Hence, they 117 require driver support and may not be available for all devices. 118 This flag can be enabled via both socket options and control messages. 119 120SOF_TIMESTAMPING_TX_SCHED: 121 Request tx timestamps prior to entering the packet scheduler. Kernel 122 transmit latency is, if long, often dominated by queuing delay. The 123 difference between this timestamp and one taken at 124 SOF_TIMESTAMPING_TX_SOFTWARE will expose this latency independent 125 of protocol processing. The latency incurred in protocol 126 processing, if any, can be computed by subtracting a userspace 127 timestamp taken immediately before send() from this timestamp. On 128 machines with virtual devices where a transmitted packet travels 129 through multiple devices and, hence, multiple packet schedulers, 130 a timestamp is generated at each layer. This allows for fine 131 grained measurement of queuing delay. This flag can be enabled 132 via both socket options and control messages. 133 134SOF_TIMESTAMPING_TX_ACK: 135 Request tx timestamps when all data in the send buffer has been 136 acknowledged. This only makes sense for reliable protocols. It is 137 currently only implemented for TCP. For that protocol, it may 138 over-report measurement, because the timestamp is generated when all 139 data up to and including the buffer at send() was acknowledged: the 140 cumulative acknowledgment. The mechanism ignores SACK and FACK. 141 This flag can be enabled via both socket options and control messages. 142 143 1441.3.2 Timestamp Reporting 145^^^^^^^^^^^^^^^^^^^^^^^^^ 146 147The other three bits control which timestamps will be reported in a 148generated control message. Changes to the bits take immediate 149effect at the timestamp reporting locations in the stack. Timestamps 150are only reported for packets that also have the relevant timestamp 151generation request set. 152 153SOF_TIMESTAMPING_SOFTWARE: 154 Report any software timestamps when available. 155 156SOF_TIMESTAMPING_SYS_HARDWARE: 157 This option is deprecated and ignored. 158 159SOF_TIMESTAMPING_RAW_HARDWARE: 160 Report hardware timestamps as generated by 161 SOF_TIMESTAMPING_TX_HARDWARE when available. 162 163 1641.3.3 Timestamp Options 165^^^^^^^^^^^^^^^^^^^^^^^ 166 167The interface supports the options 168 169SOF_TIMESTAMPING_OPT_ID: 170 Generate a unique identifier along with each packet. A process can 171 have multiple concurrent timestamping requests outstanding. Packets 172 can be reordered in the transmit path, for instance in the packet 173 scheduler. In that case timestamps will be queued onto the error 174 queue out of order from the original send() calls. It is not always 175 possible to uniquely match timestamps to the original send() calls 176 based on timestamp order or payload inspection alone, then. 177 178 This option associates each packet at send() with a unique 179 identifier and returns that along with the timestamp. The identifier 180 is derived from a per-socket u32 counter (that wraps). For datagram 181 sockets, the counter increments with each sent packet. For stream 182 sockets, it increments with every byte. 183 184 The counter starts at zero. It is initialized the first time that 185 the socket option is enabled. It is reset each time the option is 186 enabled after having been disabled. Resetting the counter does not 187 change the identifiers of existing packets in the system. 188 189 This option is implemented only for transmit timestamps. There, the 190 timestamp is always looped along with a struct sock_extended_err. 191 The option modifies field ee_data to pass an id that is unique 192 among all possibly concurrently outstanding timestamp requests for 193 that socket. 194 195 196SOF_TIMESTAMPING_OPT_CMSG: 197 Support recv() cmsg for all timestamped packets. Control messages 198 are already supported unconditionally on all packets with receive 199 timestamps and on IPv6 packets with transmit timestamp. This option 200 extends them to IPv4 packets with transmit timestamp. One use case 201 is to correlate packets with their egress device, by enabling socket 202 option IP_PKTINFO simultaneously. 203 204 205SOF_TIMESTAMPING_OPT_TSONLY: 206 Applies to transmit timestamps only. Makes the kernel return the 207 timestamp as a cmsg alongside an empty packet, as opposed to 208 alongside the original packet. This reduces the amount of memory 209 charged to the socket's receive budget (SO_RCVBUF) and delivers 210 the timestamp even if sysctl net.core.tstamp_allow_data is 0. 211 This option disables SOF_TIMESTAMPING_OPT_CMSG. 212 213SOF_TIMESTAMPING_OPT_STATS: 214 Optional stats that are obtained along with the transmit timestamps. 215 It must be used together with SOF_TIMESTAMPING_OPT_TSONLY. When the 216 transmit timestamp is available, the stats are available in a 217 separate control message of type SCM_TIMESTAMPING_OPT_STATS, as a 218 list of TLVs (struct nlattr) of types. These stats allow the 219 application to associate various transport layer stats with 220 the transmit timestamps, such as how long a certain block of 221 data was limited by peer's receiver window. 222 223SOF_TIMESTAMPING_OPT_PKTINFO: 224 Enable the SCM_TIMESTAMPING_PKTINFO control message for incoming 225 packets with hardware timestamps. The message contains struct 226 scm_ts_pktinfo, which supplies the index of the real interface which 227 received the packet and its length at layer 2. A valid (non-zero) 228 interface index will be returned only if CONFIG_NET_RX_BUSY_POLL is 229 enabled and the driver is using NAPI. The struct contains also two 230 other fields, but they are reserved and undefined. 231 232SOF_TIMESTAMPING_OPT_TX_SWHW: 233 Request both hardware and software timestamps for outgoing packets 234 when SOF_TIMESTAMPING_TX_HARDWARE and SOF_TIMESTAMPING_TX_SOFTWARE 235 are enabled at the same time. If both timestamps are generated, 236 two separate messages will be looped to the socket's error queue, 237 each containing just one timestamp. 238 239New applications are encouraged to pass SOF_TIMESTAMPING_OPT_ID to 240disambiguate timestamps and SOF_TIMESTAMPING_OPT_TSONLY to operate 241regardless of the setting of sysctl net.core.tstamp_allow_data. 242 243An exception is when a process needs additional cmsg data, for 244instance SOL_IP/IP_PKTINFO to detect the egress network interface. 245Then pass option SOF_TIMESTAMPING_OPT_CMSG. This option depends on 246having access to the contents of the original packet, so cannot be 247combined with SOF_TIMESTAMPING_OPT_TSONLY. 248 249 2501.3.4. Enabling timestamps via control messages 251^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 252 253In addition to socket options, timestamp generation can be requested 254per write via cmsg, only for SOF_TIMESTAMPING_TX_* (see Section 1.3.1). 255Using this feature, applications can sample timestamps per sendmsg() 256without paying the overhead of enabling and disabling timestamps via 257setsockopt:: 258 259 struct msghdr *msg; 260 ... 261 cmsg = CMSG_FIRSTHDR(msg); 262 cmsg->cmsg_level = SOL_SOCKET; 263 cmsg->cmsg_type = SO_TIMESTAMPING; 264 cmsg->cmsg_len = CMSG_LEN(sizeof(__u32)); 265 *((__u32 *) CMSG_DATA(cmsg)) = SOF_TIMESTAMPING_TX_SCHED | 266 SOF_TIMESTAMPING_TX_SOFTWARE | 267 SOF_TIMESTAMPING_TX_ACK; 268 err = sendmsg(fd, msg, 0); 269 270The SOF_TIMESTAMPING_TX_* flags set via cmsg will override 271the SOF_TIMESTAMPING_TX_* flags set via setsockopt. 272 273Moreover, applications must still enable timestamp reporting via 274setsockopt to receive timestamps:: 275 276 __u32 val = SOF_TIMESTAMPING_SOFTWARE | 277 SOF_TIMESTAMPING_OPT_ID /* or any other flag */; 278 err = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, &val, sizeof(val)); 279 280 2811.4 Bytestream Timestamps 282------------------------- 283 284The SO_TIMESTAMPING interface supports timestamping of bytes in a 285bytestream. Each request is interpreted as a request for when the 286entire contents of the buffer has passed a timestamping point. That 287is, for streams option SOF_TIMESTAMPING_TX_SOFTWARE will record 288when all bytes have reached the device driver, regardless of how 289many packets the data has been converted into. 290 291In general, bytestreams have no natural delimiters and therefore 292correlating a timestamp with data is non-trivial. A range of bytes 293may be split across segments, any segments may be merged (possibly 294coalescing sections of previously segmented buffers associated with 295independent send() calls). Segments can be reordered and the same 296byte range can coexist in multiple segments for protocols that 297implement retransmissions. 298 299It is essential that all timestamps implement the same semantics, 300regardless of these possible transformations, as otherwise they are 301incomparable. Handling "rare" corner cases differently from the 302simple case (a 1:1 mapping from buffer to skb) is insufficient 303because performance debugging often needs to focus on such outliers. 304 305In practice, timestamps can be correlated with segments of a 306bytestream consistently, if both semantics of the timestamp and the 307timing of measurement are chosen correctly. This challenge is no 308different from deciding on a strategy for IP fragmentation. There, the 309definition is that only the first fragment is timestamped. For 310bytestreams, we chose that a timestamp is generated only when all 311bytes have passed a point. SOF_TIMESTAMPING_TX_ACK as defined is easy to 312implement and reason about. An implementation that has to take into 313account SACK would be more complex due to possible transmission holes 314and out of order arrival. 315 316On the host, TCP can also break the simple 1:1 mapping from buffer to 317skbuff as a result of Nagle, cork, autocork, segmentation and GSO. The 318implementation ensures correctness in all cases by tracking the 319individual last byte passed to send(), even if it is no longer the 320last byte after an skbuff extend or merge operation. It stores the 321relevant sequence number in skb_shinfo(skb)->tskey. Because an skbuff 322has only one such field, only one timestamp can be generated. 323 324In rare cases, a timestamp request can be missed if two requests are 325collapsed onto the same skb. A process can detect this situation by 326enabling SOF_TIMESTAMPING_OPT_ID and comparing the byte offset at 327send time with the value returned for each timestamp. It can prevent 328the situation by always flushing the TCP stack in between requests, 329for instance by enabling TCP_NODELAY and disabling TCP_CORK and 330autocork. 331 332These precautions ensure that the timestamp is generated only when all 333bytes have passed a timestamp point, assuming that the network stack 334itself does not reorder the segments. The stack indeed tries to avoid 335reordering. The one exception is under administrator control: it is 336possible to construct a packet scheduler configuration that delays 337segments from the same stream differently. Such a setup would be 338unusual. 339 340 3412 Data Interfaces 342================== 343 344Timestamps are read using the ancillary data feature of recvmsg(). 345See `man 3 cmsg` for details of this interface. The socket manual 346page (`man 7 socket`) describes how timestamps generated with 347SO_TIMESTAMP and SO_TIMESTAMPNS records can be retrieved. 348 349 3502.1 SCM_TIMESTAMPING records 351---------------------------- 352 353These timestamps are returned in a control message with cmsg_level 354SOL_SOCKET, cmsg_type SCM_TIMESTAMPING, and payload of type 355 356For SO_TIMESTAMPING_OLD:: 357 358 struct scm_timestamping { 359 struct timespec ts[3]; 360 }; 361 362For SO_TIMESTAMPING_NEW:: 363 364 struct scm_timestamping64 { 365 struct __kernel_timespec ts[3]; 366 367Always use SO_TIMESTAMPING_NEW timestamp to always get timestamp in 368struct scm_timestamping64 format. 369 370SO_TIMESTAMPING_OLD returns incorrect timestamps after the year 2038 371on 32 bit machines. 372 373The structure can return up to three timestamps. This is a legacy 374feature. At least one field is non-zero at any time. Most timestamps 375are passed in ts[0]. Hardware timestamps are passed in ts[2]. 376 377ts[1] used to hold hardware timestamps converted to system time. 378Instead, expose the hardware clock device on the NIC directly as 379a HW PTP clock source, to allow time conversion in userspace and 380optionally synchronize system time with a userspace PTP stack such 381as linuxptp. For the PTP clock API, see Documentation/driver-api/ptp.rst. 382 383Note that if the SO_TIMESTAMP or SO_TIMESTAMPNS option is enabled 384together with SO_TIMESTAMPING using SOF_TIMESTAMPING_SOFTWARE, a false 385software timestamp will be generated in the recvmsg() call and passed 386in ts[0] when a real software timestamp is missing. This happens also 387on hardware transmit timestamps. 388 3892.1.1 Transmit timestamps with MSG_ERRQUEUE 390^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 391 392For transmit timestamps the outgoing packet is looped back to the 393socket's error queue with the send timestamp(s) attached. A process 394receives the timestamps by calling recvmsg() with flag MSG_ERRQUEUE 395set and with a msg_control buffer sufficiently large to receive the 396relevant metadata structures. The recvmsg call returns the original 397outgoing data packet with two ancillary messages attached. 398 399A message of cm_level SOL_IP(V6) and cm_type IP(V6)_RECVERR 400embeds a struct sock_extended_err. This defines the error type. For 401timestamps, the ee_errno field is ENOMSG. The other ancillary message 402will have cm_level SOL_SOCKET and cm_type SCM_TIMESTAMPING. This 403embeds the struct scm_timestamping. 404 405 4062.1.1.2 Timestamp types 407~~~~~~~~~~~~~~~~~~~~~~~ 408 409The semantics of the three struct timespec are defined by field 410ee_info in the extended error structure. It contains a value of 411type SCM_TSTAMP_* to define the actual timestamp passed in 412scm_timestamping. 413 414The SCM_TSTAMP_* types are 1:1 matches to the SOF_TIMESTAMPING_* 415control fields discussed previously, with one exception. For legacy 416reasons, SCM_TSTAMP_SND is equal to zero and can be set for both 417SOF_TIMESTAMPING_TX_HARDWARE and SOF_TIMESTAMPING_TX_SOFTWARE. It 418is the first if ts[2] is non-zero, the second otherwise, in which 419case the timestamp is stored in ts[0]. 420 421 4222.1.1.3 Fragmentation 423~~~~~~~~~~~~~~~~~~~~~ 424 425Fragmentation of outgoing datagrams is rare, but is possible, e.g., by 426explicitly disabling PMTU discovery. If an outgoing packet is fragmented, 427then only the first fragment is timestamped and returned to the sending 428socket. 429 430 4312.1.1.4 Packet Payload 432~~~~~~~~~~~~~~~~~~~~~~ 433 434The calling application is often not interested in receiving the whole 435packet payload that it passed to the stack originally: the socket 436error queue mechanism is just a method to piggyback the timestamp on. 437In this case, the application can choose to read datagrams with a 438smaller buffer, possibly even of length 0. The payload is truncated 439accordingly. Until the process calls recvmsg() on the error queue, 440however, the full packet is queued, taking up budget from SO_RCVBUF. 441 442 4432.1.1.5 Blocking Read 444~~~~~~~~~~~~~~~~~~~~~ 445 446Reading from the error queue is always a non-blocking operation. To 447block waiting on a timestamp, use poll or select. poll() will return 448POLLERR in pollfd.revents if any data is ready on the error queue. 449There is no need to pass this flag in pollfd.events. This flag is 450ignored on request. See also `man 2 poll`. 451 452 4532.1.2 Receive timestamps 454^^^^^^^^^^^^^^^^^^^^^^^^ 455 456On reception, there is no reason to read from the socket error queue. 457The SCM_TIMESTAMPING ancillary data is sent along with the packet data 458on a normal recvmsg(). Since this is not a socket error, it is not 459accompanied by a message SOL_IP(V6)/IP(V6)_RECVERROR. In this case, 460the meaning of the three fields in struct scm_timestamping is 461implicitly defined. ts[0] holds a software timestamp if set, ts[1] 462is again deprecated and ts[2] holds a hardware timestamp if set. 463 464 4653. Hardware Timestamping configuration: SIOCSHWTSTAMP and SIOCGHWTSTAMP 466======================================================================= 467 468Hardware time stamping must also be initialized for each device driver 469that is expected to do hardware time stamping. The parameter is defined in 470include/uapi/linux/net_tstamp.h as:: 471 472 struct hwtstamp_config { 473 int flags; /* no flags defined right now, must be zero */ 474 int tx_type; /* HWTSTAMP_TX_* */ 475 int rx_filter; /* HWTSTAMP_FILTER_* */ 476 }; 477 478Desired behavior is passed into the kernel and to a specific device by 479calling ioctl(SIOCSHWTSTAMP) with a pointer to a struct ifreq whose 480ifr_data points to a struct hwtstamp_config. The tx_type and 481rx_filter are hints to the driver what it is expected to do. If 482the requested fine-grained filtering for incoming packets is not 483supported, the driver may time stamp more than just the requested types 484of packets. 485 486Drivers are free to use a more permissive configuration than the requested 487configuration. It is expected that drivers should only implement directly the 488most generic mode that can be supported. For example if the hardware can 489support HWTSTAMP_FILTER_V2_EVENT, then it should generally always upscale 490HWTSTAMP_FILTER_V2_L2_SYNC_MESSAGE, and so forth, as HWTSTAMP_FILTER_V2_EVENT 491is more generic (and more useful to applications). 492 493A driver which supports hardware time stamping shall update the struct 494with the actual, possibly more permissive configuration. If the 495requested packets cannot be time stamped, then nothing should be 496changed and ERANGE shall be returned (in contrast to EINVAL, which 497indicates that SIOCSHWTSTAMP is not supported at all). 498 499Only a processes with admin rights may change the configuration. User 500space is responsible to ensure that multiple processes don't interfere 501with each other and that the settings are reset. 502 503Any process can read the actual configuration by passing this 504structure to ioctl(SIOCGHWTSTAMP) in the same way. However, this has 505not been implemented in all drivers. 506 507:: 508 509 /* possible values for hwtstamp_config->tx_type */ 510 enum { 511 /* 512 * no outgoing packet will need hardware time stamping; 513 * should a packet arrive which asks for it, no hardware 514 * time stamping will be done 515 */ 516 HWTSTAMP_TX_OFF, 517 518 /* 519 * enables hardware time stamping for outgoing packets; 520 * the sender of the packet decides which are to be 521 * time stamped by setting SOF_TIMESTAMPING_TX_SOFTWARE 522 * before sending the packet 523 */ 524 HWTSTAMP_TX_ON, 525 }; 526 527 /* possible values for hwtstamp_config->rx_filter */ 528 enum { 529 /* time stamp no incoming packet at all */ 530 HWTSTAMP_FILTER_NONE, 531 532 /* time stamp any incoming packet */ 533 HWTSTAMP_FILTER_ALL, 534 535 /* return value: time stamp all packets requested plus some others */ 536 HWTSTAMP_FILTER_SOME, 537 538 /* PTP v1, UDP, any kind of event packet */ 539 HWTSTAMP_FILTER_PTP_V1_L4_EVENT, 540 541 /* for the complete list of values, please check 542 * the include file include/uapi/linux/net_tstamp.h 543 */ 544 }; 545 5463.1 Hardware Timestamping Implementation: Device Drivers 547-------------------------------------------------------- 548 549A driver which supports hardware time stamping must support the 550SIOCSHWTSTAMP ioctl and update the supplied struct hwtstamp_config with 551the actual values as described in the section on SIOCSHWTSTAMP. It 552should also support SIOCGHWTSTAMP. 553 554Time stamps for received packets must be stored in the skb. To get a pointer 555to the shared time stamp structure of the skb call skb_hwtstamps(). Then 556set the time stamps in the structure:: 557 558 struct skb_shared_hwtstamps { 559 /* hardware time stamp transformed into duration 560 * since arbitrary point in time 561 */ 562 ktime_t hwtstamp; 563 }; 564 565Time stamps for outgoing packets are to be generated as follows: 566 567- In hard_start_xmit(), check if (skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP) 568 is set no-zero. If yes, then the driver is expected to do hardware time 569 stamping. 570- If this is possible for the skb and requested, then declare 571 that the driver is doing the time stamping by setting the flag 572 SKBTX_IN_PROGRESS in skb_shinfo(skb)->tx_flags , e.g. with:: 573 574 skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS; 575 576 You might want to keep a pointer to the associated skb for the next step 577 and not free the skb. A driver not supporting hardware time stamping doesn't 578 do that. A driver must never touch sk_buff::tstamp! It is used to store 579 software generated time stamps by the network subsystem. 580- Driver should call skb_tx_timestamp() as close to passing sk_buff to hardware 581 as possible. skb_tx_timestamp() provides a software time stamp if requested 582 and hardware timestamping is not possible (SKBTX_IN_PROGRESS not set). 583- As soon as the driver has sent the packet and/or obtained a 584 hardware time stamp for it, it passes the time stamp back by 585 calling skb_hwtstamp_tx() with the original skb, the raw 586 hardware time stamp. skb_hwtstamp_tx() clones the original skb and 587 adds the timestamps, therefore the original skb has to be freed now. 588 If obtaining the hardware time stamp somehow fails, then the driver 589 should not fall back to software time stamping. The rationale is that 590 this would occur at a later time in the processing pipeline than other 591 software time stamping and therefore could lead to unexpected deltas 592 between time stamps. 593 5943.2 Special considerations for stacked PTP Hardware Clocks 595---------------------------------------------------------- 596 597There are situations when there may be more than one PHC (PTP Hardware Clock) 598in the data path of a packet. The kernel has no explicit mechanism to allow the 599user to select which PHC to use for timestamping Ethernet frames. Instead, the 600assumption is that the outermost PHC is always the most preferable, and that 601kernel drivers collaborate towards achieving that goal. Currently there are 3 602cases of stacked PHCs, detailed below: 603 6043.2.1 DSA (Distributed Switch Architecture) switches 605^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 606 607These are Ethernet switches which have one of their ports connected to an 608(otherwise completely unaware) host Ethernet interface, and perform the role of 609a port multiplier with optional forwarding acceleration features. Each DSA 610switch port is visible to the user as a standalone (virtual) network interface, 611and its network I/O is performed, under the hood, indirectly through the host 612interface (redirecting to the host port on TX, and intercepting frames on RX). 613 614When a DSA switch is attached to a host port, PTP synchronization has to 615suffer, since the switch's variable queuing delay introduces a path delay 616jitter between the host port and its PTP partner. For this reason, some DSA 617switches include a timestamping clock of their own, and have the ability to 618perform network timestamping on their own MAC, such that path delays only 619measure wire and PHY propagation latencies. Timestamping DSA switches are 620supported in Linux and expose the same ABI as any other network interface (save 621for the fact that the DSA interfaces are in fact virtual in terms of network 622I/O, they do have their own PHC). It is typical, but not mandatory, for all 623interfaces of a DSA switch to share the same PHC. 624 625By design, PTP timestamping with a DSA switch does not need any special 626handling in the driver for the host port it is attached to. However, when the 627host port also supports PTP timestamping, DSA will take care of intercepting 628the ``.ndo_eth_ioctl`` calls towards the host port, and block attempts to enable 629hardware timestamping on it. This is because the SO_TIMESTAMPING API does not 630allow the delivery of multiple hardware timestamps for the same packet, so 631anybody else except for the DSA switch port must be prevented from doing so. 632 633In the generic layer, DSA provides the following infrastructure for PTP 634timestamping: 635 636- ``.port_txtstamp()``: a hook called prior to the transmission of 637 packets with a hardware TX timestamping request from user space. 638 This is required for two-step timestamping, since the hardware 639 timestamp becomes available after the actual MAC transmission, so the 640 driver must be prepared to correlate the timestamp with the original 641 packet so that it can re-enqueue the packet back into the socket's 642 error queue. To save the packet for when the timestamp becomes 643 available, the driver can call ``skb_clone_sk`` , save the clone pointer 644 in skb->cb and enqueue a tx skb queue. Typically, a switch will have a 645 PTP TX timestamp register (or sometimes a FIFO) where the timestamp 646 becomes available. In case of a FIFO, the hardware might store 647 key-value pairs of PTP sequence ID/message type/domain number and the 648 actual timestamp. To perform the correlation correctly between the 649 packets in a queue waiting for timestamping and the actual timestamps, 650 drivers can use a BPF classifier (``ptp_classify_raw``) to identify 651 the PTP transport type, and ``ptp_parse_header`` to interpret the PTP 652 header fields. There may be an IRQ that is raised upon this 653 timestamp's availability, or the driver might have to poll after 654 invoking ``dev_queue_xmit()`` towards the host interface. 655 One-step TX timestamping do not require packet cloning, since there is 656 no follow-up message required by the PTP protocol (because the 657 TX timestamp is embedded into the packet by the MAC), and therefore 658 user space does not expect the packet annotated with the TX timestamp 659 to be re-enqueued into its socket's error queue. 660 661- ``.port_rxtstamp()``: On RX, the BPF classifier is run by DSA to 662 identify PTP event messages (any other packets, including PTP general 663 messages, are not timestamped). The original (and only) timestampable 664 skb is provided to the driver, for it to annotate it with a timestamp, 665 if that is immediately available, or defer to later. On reception, 666 timestamps might either be available in-band (through metadata in the 667 DSA header, or attached in other ways to the packet), or out-of-band 668 (through another RX timestamping FIFO). Deferral on RX is typically 669 necessary when retrieving the timestamp needs a sleepable context. In 670 that case, it is the responsibility of the DSA driver to call 671 ``netif_rx_ni()`` on the freshly timestamped skb. 672 6733.2.2 Ethernet PHYs 674^^^^^^^^^^^^^^^^^^^ 675 676These are devices that typically fulfill a Layer 1 role in the network stack, 677hence they do not have a representation in terms of a network interface as DSA 678switches do. However, PHYs may be able to detect and timestamp PTP packets, for 679performance reasons: timestamps taken as close as possible to the wire have the 680potential to yield a more stable and precise synchronization. 681 682A PHY driver that supports PTP timestamping must create a ``struct 683mii_timestamper`` and add a pointer to it in ``phydev->mii_ts``. The presence 684of this pointer will be checked by the networking stack. 685 686Since PHYs do not have network interface representations, the timestamping and 687ethtool ioctl operations for them need to be mediated by their respective MAC 688driver. Therefore, as opposed to DSA switches, modifications need to be done 689to each individual MAC driver for PHY timestamping support. This entails: 690 691- Checking, in ``.ndo_eth_ioctl``, whether ``phy_has_hwtstamp(netdev->phydev)`` 692 is true or not. If it is, then the MAC driver should not process this request 693 but instead pass it on to the PHY using ``phy_mii_ioctl()``. 694 695- On RX, special intervention may or may not be needed, depending on the 696 function used to deliver skb's up the network stack. In the case of plain 697 ``netif_rx()`` and similar, MAC drivers must check whether 698 ``skb_defer_rx_timestamp(skb)`` is necessary or not - and if it is, don't 699 call ``netif_rx()`` at all. If ``CONFIG_NETWORK_PHY_TIMESTAMPING`` is 700 enabled, and ``skb->dev->phydev->mii_ts`` exists, its ``.rxtstamp()`` hook 701 will be called now, to determine, using logic very similar to DSA, whether 702 deferral for RX timestamping is necessary. Again like DSA, it becomes the 703 responsibility of the PHY driver to send the packet up the stack when the 704 timestamp is available. 705 706 For other skb receive functions, such as ``napi_gro_receive`` and 707 ``netif_receive_skb``, the stack automatically checks whether 708 ``skb_defer_rx_timestamp()`` is necessary, so this check is not needed inside 709 the driver. 710 711- On TX, again, special intervention might or might not be needed. The 712 function that calls the ``mii_ts->txtstamp()`` hook is named 713 ``skb_clone_tx_timestamp()``. This function can either be called directly 714 (case in which explicit MAC driver support is indeed needed), but the 715 function also piggybacks from the ``skb_tx_timestamp()`` call, which many MAC 716 drivers already perform for software timestamping purposes. Therefore, if a 717 MAC supports software timestamping, it does not need to do anything further 718 at this stage. 719 7203.2.3 MII bus snooping devices 721^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 722 723These perform the same role as timestamping Ethernet PHYs, save for the fact 724that they are discrete devices and can therefore be used in conjunction with 725any PHY even if it doesn't support timestamping. In Linux, they are 726discoverable and attachable to a ``struct phy_device`` through Device Tree, and 727for the rest, they use the same mii_ts infrastructure as those. See 728Documentation/devicetree/bindings/ptp/timestamper.txt for more details. 729 7303.2.4 Other caveats for MAC drivers 731^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 732 733Stacked PHCs, especially DSA (but not only) - since that doesn't require any 734modification to MAC drivers, so it is more difficult to ensure correctness of 735all possible code paths - is that they uncover bugs which were impossible to 736trigger before the existence of stacked PTP clocks. One example has to do with 737this line of code, already presented earlier:: 738 739 skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS; 740 741Any TX timestamping logic, be it a plain MAC driver, a DSA switch driver, a PHY 742driver or a MII bus snooping device driver, should set this flag. 743But a MAC driver that is unaware of PHC stacking might get tripped up by 744somebody other than itself setting this flag, and deliver a duplicate 745timestamp. 746For example, a typical driver design for TX timestamping might be to split the 747transmission part into 2 portions: 748 7491. "TX": checks whether PTP timestamping has been previously enabled through 750 the ``.ndo_eth_ioctl`` ("``priv->hwtstamp_tx_enabled == true``") and the 751 current skb requires a TX timestamp ("``skb_shinfo(skb)->tx_flags & 752 SKBTX_HW_TSTAMP``"). If this is true, it sets the 753 "``skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS``" flag. Note: as 754 described above, in the case of a stacked PHC system, this condition should 755 never trigger, as this MAC is certainly not the outermost PHC. But this is 756 not where the typical issue is. Transmission proceeds with this packet. 757 7582. "TX confirmation": Transmission has finished. The driver checks whether it 759 is necessary to collect any TX timestamp for it. Here is where the typical 760 issues are: the MAC driver takes a shortcut and only checks whether 761 "``skb_shinfo(skb)->tx_flags & SKBTX_IN_PROGRESS``" was set. With a stacked 762 PHC system, this is incorrect because this MAC driver is not the only entity 763 in the TX data path who could have enabled SKBTX_IN_PROGRESS in the first 764 place. 765 766The correct solution for this problem is for MAC drivers to have a compound 767check in their "TX confirmation" portion, not only for 768"``skb_shinfo(skb)->tx_flags & SKBTX_IN_PROGRESS``", but also for 769"``priv->hwtstamp_tx_enabled == true``". Because the rest of the system ensures 770that PTP timestamping is not enabled for anything other than the outermost PHC, 771this enhanced check will avoid delivering a duplicated TX timestamp to user 772space. 773
