1.. SPDX-License-Identifier: GPL-2.0 2 3.. include:: <isonum.txt> 4 5=============================================================== 6Intel Image Processing Unit 3 (IPU3) Imaging Unit (ImgU) driver 7=============================================================== 8 9Copyright |copy| 2018 Intel Corporation 10 11Introduction 12============ 13 14This file documents the Intel IPU3 (3rd generation Image Processing Unit) 15Imaging Unit drivers located under drivers/media/pci/intel/ipu3 (CIO2) as well 16as under drivers/staging/media/ipu3 (ImgU). 17 18The Intel IPU3 found in certain Kaby Lake (as well as certain Sky Lake) 19platforms (U/Y processor lines) is made up of two parts namely the Imaging Unit 20(ImgU) and the CIO2 device (MIPI CSI2 receiver). 21 22The CIO2 device receives the raw Bayer data from the sensors and outputs the 23frames in a format that is specific to the IPU3 (for consumption by the IPU3 24ImgU). The CIO2 driver is available as drivers/media/pci/intel/ipu3/ipu3-cio2* 25and is enabled through the CONFIG_VIDEO_IPU3_CIO2 config option. 26 27The Imaging Unit (ImgU) is responsible for processing images captured 28by the IPU3 CIO2 device. The ImgU driver sources can be found under 29drivers/staging/media/ipu3 directory. The driver is enabled through the 30CONFIG_VIDEO_IPU3_IMGU config option. 31 32The two driver modules are named ipu3_csi2 and ipu3_imgu, respectively. 33 34The drivers has been tested on Kaby Lake platforms (U/Y processor lines). 35 36Both of the drivers implement V4L2, Media Controller and V4L2 sub-device 37interfaces. The IPU3 CIO2 driver supports camera sensors connected to the CIO2 38MIPI CSI-2 interfaces through V4L2 sub-device sensor drivers. 39 40CIO2 41==== 42 43The CIO2 is represented as a single V4L2 subdev, which provides a V4L2 subdev 44interface to the user space. There is a video node for each CSI-2 receiver, 45with a single media controller interface for the entire device. 46 47The CIO2 contains four independent capture channel, each with its own MIPI CSI-2 48receiver and DMA engine. Each channel is modelled as a V4L2 sub-device exposed 49to userspace as a V4L2 sub-device node and has two pads: 50 51.. tabularcolumns:: |p{0.8cm}|p{4.0cm}|p{4.0cm}| 52 53.. flat-table:: 54 :header-rows: 1 55 56 * - Pad 57 - Direction 58 - Purpose 59 60 * - 0 61 - sink 62 - MIPI CSI-2 input, connected to the sensor subdev 63 64 * - 1 65 - source 66 - Raw video capture, connected to the V4L2 video interface 67 68The V4L2 video interfaces model the DMA engines. They are exposed to userspace 69as V4L2 video device nodes. 70 71Capturing frames in raw Bayer format 72------------------------------------ 73 74CIO2 MIPI CSI2 receiver is used to capture frames (in packed raw Bayer format) 75from the raw sensors connected to the CSI2 ports. The captured frames are used 76as input to the ImgU driver. 77 78Image processing using IPU3 ImgU requires tools such as raw2pnm [#f1]_, and 79yavta [#f2]_ due to the following unique requirements and / or features specific 80to IPU3. 81 82-- The IPU3 CSI2 receiver outputs the captured frames from the sensor in packed 83raw Bayer format that is specific to IPU3. 84 85-- Multiple video nodes have to be operated simultaneously. 86 87Let us take the example of ov5670 sensor connected to CSI2 port 0, for a 882592x1944 image capture. 89 90Using the media controller APIs, the ov5670 sensor is configured to send 91frames in packed raw Bayer format to IPU3 CSI2 receiver. 92 93.. code-block:: none 94 95 # This example assumes /dev/media0 as the CIO2 media device 96 export MDEV=/dev/media0 97 98 # and that ov5670 sensor is connected to i2c bus 10 with address 0x36 99 export SDEV=$(media-ctl -d $MDEV -e "ov5670 10-0036") 100 101 # Establish the link for the media devices using media-ctl [#f3]_ 102 media-ctl -d $MDEV -l "ov5670:0 -> ipu3-csi2 0:0[1]" 103 104 # Set the format for the media devices 105 media-ctl -d $MDEV -V "ov5670:0 [fmt:SGRBG10/2592x1944]" 106 media-ctl -d $MDEV -V "ipu3-csi2 0:0 [fmt:SGRBG10/2592x1944]" 107 media-ctl -d $MDEV -V "ipu3-csi2 0:1 [fmt:SGRBG10/2592x1944]" 108 109Once the media pipeline is configured, desired sensor specific settings 110(such as exposure and gain settings) can be set, using the yavta tool. 111 112e.g 113 114.. code-block:: none 115 116 yavta -w 0x009e0903 444 $SDEV 117 yavta -w 0x009e0913 1024 $SDEV 118 yavta -w 0x009e0911 2046 $SDEV 119 120Once the desired sensor settings are set, frame captures can be done as below. 121 122e.g 123 124.. code-block:: none 125 126 yavta --data-prefix -u -c10 -n5 -I -s2592x1944 --file=/tmp/frame-#.bin \ 127 -f IPU3_SGRBG10 $(media-ctl -d $MDEV -e "ipu3-cio2 0") 128 129With the above command, 10 frames are captured at 2592x1944 resolution, with 130sGRBG10 format and output as IPU3_SGRBG10 format. 131 132The captured frames are available as /tmp/frame-#.bin files. 133 134ImgU 135==== 136 137The ImgU is represented as two V4L2 subdevs, each of which provides a V4L2 138subdev interface to the user space. 139 140Each V4L2 subdev represents a pipe, which can support a maximum of 2 streams. 141This helps to support advanced camera features like Continuous View Finder (CVF) 142and Snapshot During Video(SDV). 143 144The ImgU contains two independent pipes, each modelled as a V4L2 sub-device 145exposed to userspace as a V4L2 sub-device node. 146 147Each pipe has two sink pads and three source pads for the following purpose: 148 149.. tabularcolumns:: |p{0.8cm}|p{4.0cm}|p{4.0cm}| 150 151.. flat-table:: 152 :header-rows: 1 153 154 * - Pad 155 - Direction 156 - Purpose 157 158 * - 0 159 - sink 160 - Input raw video stream 161 162 * - 1 163 - sink 164 - Processing parameters 165 166 * - 2 167 - source 168 - Output processed video stream 169 170 * - 3 171 - source 172 - Output viewfinder video stream 173 174 * - 4 175 - source 176 - 3A statistics 177 178Each pad is connected to a corresponding V4L2 video interface, exposed to 179userspace as a V4L2 video device node. 180 181Device operation 182---------------- 183 184With ImgU, once the input video node ("ipu3-imgu 0/1":0, in 185<entity>:<pad-number> format) is queued with buffer (in packed raw Bayer 186format), ImgU starts processing the buffer and produces the video output in YUV 187format and statistics output on respective output nodes. The driver is expected 188to have buffers ready for all of parameter, output and statistics nodes, when 189input video node is queued with buffer. 190 191At a minimum, all of input, main output, 3A statistics and viewfinder 192video nodes should be enabled for IPU3 to start image processing. 193 194Each ImgU V4L2 subdev has the following set of video nodes. 195 196input, output and viewfinder video nodes 197---------------------------------------- 198 199The frames (in packed raw Bayer format specific to the IPU3) received by the 200input video node is processed by the IPU3 Imaging Unit and are output to 2 video 201nodes, with each targeting a different purpose (main output and viewfinder 202output). 203 204Details onand the Bayer format specific to the IPU3 can be found in 205:ref:`v4l2-pix-fmt-ipu3-sbggr10`. 206 207The driver supports V4L2 Video Capture Interface as defined at :ref:`devices`. 208 209Only the multi-planar API is supported. More details can be found at 210:ref:`planar-apis`. 211 212Parameters video node 213--------------------- 214 215The parameters video node receives the ImgU algorithm parameters that are used 216to configure how the ImgU algorithms process the image. 217 218Details on processing parameters specific to the IPU3 can be found in 219:ref:`v4l2-meta-fmt-params`. 220 2213A statistics video node 222------------------------ 223 2243A statistics video node is used by the ImgU driver to output the 3A (auto 225focus, auto exposure and auto white balance) statistics for the frames that are 226being processed by the ImgU to user space applications. User space applications 227can use this statistics data to compute the desired algorithm parameters for 228the ImgU. 229 230Configuring the Intel IPU3 231========================== 232 233The IPU3 ImgU pipelines can be configured using the Media Controller, defined at 234:ref:`media_controller`. 235 236Running mode and firmware binary selection 237------------------------------------------ 238 239ImgU works based on firmware, currently the ImgU firmware support run 2 pipes 240in time-sharing with single input frame data. Each pipe can run at certain mode 241- "VIDEO" or "STILL", "VIDEO" mode is commonly used for video frames capture, 242and "STILL" is used for still frame capture. However, you can also select 243"VIDEO" to capture still frames if you want to capture images with less system 244load and power. For "STILL" mode, ImgU will try to use smaller BDS factor and 245output larger bayer frame for further YUV processing than "VIDEO" mode to get 246high quality images. Besides, "STILL" mode need XNR3 to do noise reduction, 247hence "STILL" mode will need more power and memory bandwidth than "VIDEO" mode. 248TNR will be enabled in "VIDEO" mode and bypassed by "STILL" mode. ImgU is 249running at "VIDEO" mode by default, the user can use v4l2 control 250V4L2_CID_INTEL_IPU3_MODE (currently defined in 251drivers/staging/media/ipu3/include/uapi/intel-ipu3.h) to query and set the 252running mode. For user, there is no difference for buffer queueing between the 253"VIDEO" and "STILL" mode, mandatory input and main output node should be 254enabled and buffers need be queued, the statistics and the view-finder queues 255are optional. 256 257The firmware binary will be selected according to current running mode, such log 258"using binary if_to_osys_striped " or "using binary if_to_osys_primary_striped" 259could be observed if you enable the ImgU dynamic debug, the binary 260if_to_osys_striped is selected for "VIDEO" and the binary 261"if_to_osys_primary_striped" is selected for "STILL". 262 263 264Processing the image in raw Bayer format 265---------------------------------------- 266 267Configuring ImgU V4L2 subdev for image processing 268~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 269 270The ImgU V4L2 subdevs have to be configured with media controller APIs to have 271all the video nodes setup correctly. 272 273Let us take "ipu3-imgu 0" subdev as an example. 274 275.. code-block:: none 276 277 media-ctl -d $MDEV -r 278 media-ctl -d $MDEV -l "ipu3-imgu 0 input":0 -> "ipu3-imgu 0":0[1] 279 media-ctl -d $MDEV -l "ipu3-imgu 0":2 -> "ipu3-imgu 0 output":0[1] 280 media-ctl -d $MDEV -l "ipu3-imgu 0":3 -> "ipu3-imgu 0 viewfinder":0[1] 281 media-ctl -d $MDEV -l "ipu3-imgu 0":4 -> "ipu3-imgu 0 3a stat":0[1] 282 283Also the pipe mode of the corresponding V4L2 subdev should be set as desired 284(e.g 0 for video mode or 1 for still mode) through the control id 0x009819a1 as 285below. 286 287.. code-block:: none 288 289 yavta -w "0x009819A1 1" /dev/v4l-subdev7 290 291Certain hardware blocks in ImgU pipeline can change the frame resolution by 292cropping or scaling, these hardware blocks include Input Feeder(IF), Bayer Down 293Scaler (BDS) and Geometric Distortion Correction (GDC). 294There is also a block which can change the frame resolution - YUV Scaler, it is 295only applicable to the secondary output. 296 297RAW Bayer frames go through these ImgU pipeline hardware blocks and the final 298processed image output to the DDR memory. 299 300.. kernel-figure:: ipu3_rcb.svg 301 :alt: ipu3 resolution blocks image 302 303 IPU3 resolution change hardware blocks 304 305**Input Feeder** 306 307Input Feeder gets the Bayer frame data from the sensor, it can enable cropping 308of lines and columns from the frame and then store pixels into device's internal 309pixel buffer which are ready to readout by following blocks. 310 311**Bayer Down Scaler** 312 313Bayer Down Scaler is capable of performing image scaling in Bayer domain, the 314downscale factor can be configured from 1X to 1/4X in each axis with 315configuration steps of 0.03125 (1/32). 316 317**Geometric Distortion Correction** 318 319Geometric Distortion Correction is used to perform correction of distortions 320and image filtering. It needs some extra filter and envelope padding pixels to 321work, so the input resolution of GDC should be larger than the output 322resolution. 323 324**YUV Scaler** 325 326YUV Scaler which similar with BDS, but it is mainly do image down scaling in 327YUV domain, it can support up to 1/12X down scaling, but it can not be applied 328to the main output. 329 330The ImgU V4L2 subdev has to be configured with the supported resolutions in all 331the above hardware blocks, for a given input resolution. 332For a given supported resolution for an input frame, the Input Feeder, Bayer 333Down Scaler and GDC blocks should be configured with the supported resolutions 334as each hardware block has its own alignment requirement. 335 336You must configure the output resolution of the hardware blocks smartly to meet 337the hardware requirement along with keeping the maximum field of view. The 338intermediate resolutions can be generated by specific tool - 339 340https://github.com/intel/intel-ipu3-pipecfg 341 342This tool can be used to generate intermediate resolutions. More information can 343be obtained by looking at the following IPU3 ImgU configuration table. 344 345https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/master 346 347Under baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/files/gcss 348directory, graph_settings_ov5670.xml can be used as an example. 349 350The following steps prepare the ImgU pipeline for the image processing. 351 3521. The ImgU V4L2 subdev data format should be set by using the 353VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height obtained above. 354 3552. The ImgU V4L2 subdev cropping should be set by using the 356VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as the target, 357using the input feeder height and width. 358 3593. The ImgU V4L2 subdev composing should be set by using the 360VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the target, 361using the BDS height and width. 362 363For the ov5670 example, for an input frame with a resolution of 2592x1944 364(which is input to the ImgU subdev pad 0), the corresponding resolutions 365for input feeder, BDS and GDC are 2592x1944, 2592x1944 and 2560x1920 366respectively. 367 368Once this is done, the received raw Bayer frames can be input to the ImgU 369V4L2 subdev as below, using the open source application v4l2n [#f1]_. 370 371For an image captured with 2592x1944 [#f4]_ resolution, with desired output 372resolution as 2560x1920 and viewfinder resolution as 2560x1920, the following 373v4l2n command can be used. This helps process the raw Bayer frames and produces 374the desired results for the main output image and the viewfinder output, in NV12 375format. 376 377.. code-block:: none 378 379 v4l2n --pipe=4 --load=/tmp/frame-#.bin --open=/dev/video4 380 --fmt=type:VIDEO_OUTPUT_MPLANE,width=2592,height=1944,pixelformat=0X47337069 \ 381 --reqbufs=type:VIDEO_OUTPUT_MPLANE,count:1 --pipe=1 \ 382 --output=/tmp/frames.out --open=/dev/video5 \ 383 --fmt=type:VIDEO_CAPTURE_MPLANE,width=2560,height=1920,pixelformat=NV12 \ 384 --reqbufs=type:VIDEO_CAPTURE_MPLANE,count:1 --pipe=2 \ 385 --output=/tmp/frames.vf --open=/dev/video6 \ 386 --fmt=type:VIDEO_CAPTURE_MPLANE,width=2560,height=1920,pixelformat=NV12 \ 387 --reqbufs=type:VIDEO_CAPTURE_MPLANE,count:1 --pipe=3 --open=/dev/video7 \ 388 --output=/tmp/frames.3A --fmt=type:META_CAPTURE,? \ 389 --reqbufs=count:1,type:META_CAPTURE --pipe=1,2,3,4 --stream=5 390 391You can also use yavta [#f2]_ command to do same thing as above: 392 393.. code-block:: none 394 395 yavta --data-prefix -Bcapture-mplane -c10 -n5 -I -s2592x1944 \ 396 --file=frame-#.out-f NV12 /dev/video5 & \ 397 yavta --data-prefix -Bcapture-mplane -c10 -n5 -I -s2592x1944 \ 398 --file=frame-#.vf -f NV12 /dev/video6 & \ 399 yavta --data-prefix -Bmeta-capture -c10 -n5 -I \ 400 --file=frame-#.3a /dev/video7 & \ 401 yavta --data-prefix -Boutput-mplane -c10 -n5 -I -s2592x1944 \ 402 --file=/tmp/frame-in.cio2 -f IPU3_SGRBG10 /dev/video4 403 404where /dev/video4, /dev/video5, /dev/video6 and /dev/video7 devices point to 405input, output, viewfinder and 3A statistics video nodes respectively. 406 407Converting the raw Bayer image into YUV domain 408---------------------------------------------- 409 410The processed images after the above step, can be converted to YUV domain 411as below. 412 413Main output frames 414~~~~~~~~~~~~~~~~~~ 415 416.. code-block:: none 417 418 raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.out /tmp/frames.out.ppm 419 420where 2560x1920 is output resolution, NV12 is the video format, followed 421by input frame and output PNM file. 422 423Viewfinder output frames 424~~~~~~~~~~~~~~~~~~~~~~~~ 425 426.. code-block:: none 427 428 raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.vf /tmp/frames.vf.ppm 429 430where 2560x1920 is output resolution, NV12 is the video format, followed 431by input frame and output PNM file. 432 433Example user space code for IPU3 434================================ 435 436User space code that configures and uses IPU3 is available here. 437 438https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/master/ 439 440The source can be located under hal/intel directory. 441 442Overview of IPU3 pipeline 443========================= 444 445IPU3 pipeline has a number of image processing stages, each of which takes a 446set of parameters as input. The major stages of pipelines are shown here: 447 448.. kernel-render:: DOT 449 :alt: IPU3 ImgU Pipeline 450 :caption: IPU3 ImgU Pipeline Diagram 451 452 digraph "IPU3 ImgU" { 453 node [shape=box] 454 splines="ortho" 455 rankdir="LR" 456 457 a [label="Raw pixels"] 458 b [label="Bayer Downscaling"] 459 c [label="Optical Black Correction"] 460 d [label="Linearization"] 461 e [label="Lens Shading Correction"] 462 f [label="White Balance / Exposure / Focus Apply"] 463 g [label="Bayer Noise Reduction"] 464 h [label="ANR"] 465 i [label="Demosaicing"] 466 j [label="Color Correction Matrix"] 467 k [label="Gamma correction"] 468 l [label="Color Space Conversion"] 469 m [label="Chroma Down Scaling"] 470 n [label="Chromatic Noise Reduction"] 471 o [label="Total Color Correction"] 472 p [label="XNR3"] 473 q [label="TNR"] 474 r [label="DDR", style=filled, fillcolor=yellow, shape=cylinder] 475 s [label="YUV Downscaling"] 476 t [label="DDR", style=filled, fillcolor=yellow, shape=cylinder] 477 478 { rank=same; a -> b -> c -> d -> e -> f -> g -> h -> i } 479 { rank=same; j -> k -> l -> m -> n -> o -> p -> q -> s -> t} 480 481 a -> j [style=invis, weight=10] 482 i -> j 483 q -> r 484 } 485 486The table below presents a description of the above algorithms. 487 488======================== ======================================================= 489Name Description 490======================== ======================================================= 491Optical Black Correction Optical Black Correction block subtracts a pre-defined 492 value from the respective pixel values to obtain better 493 image quality. 494 Defined in struct ipu3_uapi_obgrid_param. 495Linearization This algo block uses linearization parameters to 496 address non-linearity sensor effects. The Lookup table 497 table is defined in 498 struct ipu3_uapi_isp_lin_vmem_params. 499SHD Lens shading correction is used to correct spatial 500 non-uniformity of the pixel response due to optical 501 lens shading. This is done by applying a different gain 502 for each pixel. The gain, black level etc are 503 configured in struct ipu3_uapi_shd_config_static. 504BNR Bayer noise reduction block removes image noise by 505 applying a bilateral filter. 506 See struct ipu3_uapi_bnr_static_config for details. 507ANR Advanced Noise Reduction is a block based algorithm 508 that performs noise reduction in the Bayer domain. The 509 convolution matrix etc can be found in 510 struct ipu3_uapi_anr_config. 511DM Demosaicing converts raw sensor data in Bayer format 512 into RGB (Red, Green, Blue) presentation. Then add 513 outputs of estimation of Y channel for following stream 514 processing by Firmware. The struct is defined as 515 struct ipu3_uapi_dm_config. 516Color Correction Color Correction algo transforms sensor specific color 517 space to the standard "sRGB" color space. This is done 518 by applying 3x3 matrix defined in 519 struct ipu3_uapi_ccm_mat_config. 520Gamma correction Gamma correction struct ipu3_uapi_gamma_config is a 521 basic non-linear tone mapping correction that is 522 applied per pixel for each pixel component. 523CSC Color space conversion transforms each pixel from the 524 RGB primary presentation to YUV (Y: brightness, 525 UV: Luminance) presentation. This is done by applying 526 a 3x3 matrix defined in 527 struct ipu3_uapi_csc_mat_config 528CDS Chroma down sampling 529 After the CSC is performed, the Chroma Down Sampling 530 is applied for a UV plane down sampling by a factor 531 of 2 in each direction for YUV 4:2:0 using a 4x2 532 configurable filter struct ipu3_uapi_cds_params. 533CHNR Chroma noise reduction 534 This block processes only the chrominance pixels and 535 performs noise reduction by cleaning the high 536 frequency noise. 537 See struct struct ipu3_uapi_yuvp1_chnr_config. 538TCC Total color correction as defined in struct 539 struct ipu3_uapi_yuvp2_tcc_static_config. 540XNR3 eXtreme Noise Reduction V3 is the third revision of 541 noise reduction algorithm used to improve image 542 quality. This removes the low frequency noise in the 543 captured image. Two related structs are being defined, 544 struct ipu3_uapi_isp_xnr3_params for ISP data memory 545 and struct ipu3_uapi_isp_xnr3_vmem_params for vector 546 memory. 547TNR Temporal Noise Reduction block compares successive 548 frames in time to remove anomalies / noise in pixel 549 values. struct ipu3_uapi_isp_tnr3_vmem_params and 550 struct ipu3_uapi_isp_tnr3_params are defined for ISP 551 vector and data memory respectively. 552======================== ======================================================= 553 554Other often encountered acronyms not listed in above table: 555 556 ACC 557 Accelerator cluster 558 AWB_FR 559 Auto white balance filter response statistics 560 BDS 561 Bayer downscaler parameters 562 CCM 563 Color correction matrix coefficients 564 IEFd 565 Image enhancement filter directed 566 Obgrid 567 Optical black level compensation 568 OSYS 569 Output system configuration 570 ROI 571 Region of interest 572 YDS 573 Y down sampling 574 YTM 575 Y-tone mapping 576 577A few stages of the pipeline will be executed by firmware running on the ISP 578processor, while many others will use a set of fixed hardware blocks also 579called accelerator cluster (ACC) to crunch pixel data and produce statistics. 580 581ACC parameters of individual algorithms, as defined by 582struct ipu3_uapi_acc_param, can be chosen to be applied by the user 583space through struct struct ipu3_uapi_flags embedded in 584struct ipu3_uapi_params structure. For parameters that are configured as 585not enabled by the user space, the corresponding structs are ignored by the 586driver, in which case the existing configuration of the algorithm will be 587preserved. 588 589References 590========== 591 592.. [#f5] drivers/staging/media/ipu3/include/uapi/intel-ipu3.h 593 594.. [#f1] https://github.com/intel/nvt 595 596.. [#f2] http://git.ideasonboard.org/yavta.git 597 598.. [#f3] http://git.ideasonboard.org/?p=media-ctl.git;a=summary 599 600.. [#f4] ImgU limitation requires an additional 16x16 for all input resolutions 601
