Figure 4-3 shows the 4-camera deep learning GStreamer pipeline. TI provides a suit of GStreamer plugins that allow offloading some of the media processing and the deep learning inference to the hardware accelerators. Some examples of these plugins include tiovxisp, tiovxmultiscaler, tiovxmosaic, and tidlinferer. The pipeline in Figure 4-3 includes all required plugins for a multipath GStreamer pipeline for 4-camera inputs each with media preprocess, deep learning inference, and postprocess. The duplicated plugins for each of the camera paths are stacked in the graph for easier demonstration.

The available hardware resources are evenly distributed among the four camera paths. For instance, AM62A contains two image multiscalers: MSC0 and MSC1. The pipeline explicitly dedicated MSC0 to process camera 1 and camera 2 paths while MSC1 is dedicated to camera 3 and camera 4.

Figure 4-3 GStreamer Pipeline for Quad CSI IMX219 Camera Deep Learning Inference on AM62A

The output of the four camera pipelines are scaled down and concatenated together using tiovxmosaic plugin. The output is displayed on a single screen. Figure 4-4 shows the output of the four cameras with deep learning model running object detection. Each pipeline (camera) is running at 30 FPS and total 120 FPS.

Figure 4-4 Screenshot of Quad Camera Object Detection Deep Learning Inference using AM62A with Graphical Performance Overlay

Next is the full pipeline script for the multicamera deep learning use case shown in Figure 4-3.

gst-launch-1.0 -v \
v4l2src device=/dev/video2 io-mode=5 ! queue leaky=2 ! video/x-bayer, width=1920, height=1080, format=rggb ! tiovxisp sensor-name=SENSOR_SONY_IMX219_RPI dcc-isp-file=/opt/imaging/imx219/dcc_viss.bin format-msb=7 sink_0::dcc-2a-file=/opt/imaging/imx219/dcc_2a.bin sink_0::device=/dev/v4l-subdev2 ! video/x-raw, format=NV12 ! \
tiovxmultiscaler target=0 name=split_01 \
split_01. ! queue ! video/x-raw, width=1110, height=690 ! tiovxmultiscaler target=0 ! video/x-raw, width=300, height=300 ! tiovxdlpreproc data-type=3 channel-order=1 tensor-format=rgb out-pool-size=4 ! application/x-tensor-tiovx ! tidlinferer target=1 model=/opt/model_zoo/TFL-OD-2000-ssd-mobV1-coco-mlperf-300x300 ! post_0.tensor \
split_01. ! queue ! video/x-raw, width=640, height=360 ! post_0.sink \
tidlpostproc name=post_0 model=/opt/model_zoo/TFL-OD-2000-ssd-mobV1-coco-mlperf-300x300 alpha=0.400000 viz-threshold=0.600000 top-N=5 ! queue ! mosaic_0. \
\
v4l2src device=/dev/video3 io-mode=5 ! queue leaky=2 ! video/x-bayer, width=1920, height=1080, format=rggb ! tiovxisp sensor-name=SENSOR_SONY_IMX219_RPI dcc-isp-file=/opt/imaging/imx219/dcc_viss.bin format-msb=7 sink_0::dcc-2a-file=/opt/imaging/imx219/dcc_2a.bin sink_0::device=/dev/v4l-subdev2 ! video/x-raw, format=NV12 ! \
tiovxmultiscaler target=0 name=split_11 \
split_11. ! queue ! video/x-raw, width=1110, height=690 ! tiovxmultiscaler target=0 ! video/x-raw, width=300, height=300 ! tiovxdlpreproc data-type=3 channel-order=1 tensor-format=rgb out-pool-size=4 ! application/x-tensor-tiovx ! tidlinferer target=1 model=/opt/model_zoo/TFL-OD-2000-ssd-mobV1-coco-mlperf-300x300 ! post_1.tensor \
split_11. ! queue ! video/x-raw, width=640, height=360 ! post_1.sink \
tidlpostproc name=post_1 model=/opt/model_zoo/TFL-OD-2000-ssd-mobV1-coco-mlperf-300x300 alpha=0.400000 viz-threshold=0.600000 top-N=5 ! queue ! mosaic_0. \
\
v4l2src device=/dev/video4 io-mode=5 ! queue leaky=2 ! video/x-bayer, width=1920, height=1080, format=rggb ! tiovxisp sensor-name=SENSOR_SONY_IMX219_RPI dcc-isp-file=/opt/imaging/imx219/dcc_viss.bin format-msb=7 sink_0::dcc-2a-file=/opt/imaging/imx219/dcc_2a.bin sink_0::device=/dev/v4l-subdev2 ! video/x-raw, format=NV12 ! \
tiovxmultiscaler target=1 name=split_21 \
split_21. ! queue ! video/x-raw, width=1110, height=690 ! tiovxmultiscaler target=1 ! video/x-raw, width=300, height=300 ! tiovxdlpreproc data-type=3 channel-order=1 tensor-format=rgb out-pool-size=4 ! application/x-tensor-tiovx ! tidlinferer target=1 model=/opt/model_zoo/TFL-OD-2000-ssd-mobV1-coco-mlperf-300x300 ! post_2.tensor \
split_21. ! queue ! video/x-raw, width=640, height=360 ! post_2.sink \
tidlpostproc name=post_2 model=/opt/model_zoo/TFL-OD-2000-ssd-mobV1-coco-mlperf-300x300 alpha=0.400000 viz-threshold=0.600000 top-N=5 ! queue ! mosaic_0. \
\
v4l2src device=/dev/video5 io-mode=5 ! queue leaky=2 ! video/x-bayer, width=1920, height=1080, format=rggb ! tiovxisp sensor-name=SENSOR_SONY_IMX219_RPI dcc-isp-file=/opt/imaging/imx219/dcc_viss.bin format-msb=7 sink_0::dcc-2a-file=/opt/imaging/imx219/dcc_2a.bin sink_0::device=/dev/v4l-subdev2 ! video/x-raw, format=NV12 ! \
tiovxmultiscaler target=1 name=split_31 \
split_31. ! queue ! video/x-raw, width=1110, height=690 ! tiovxmultiscaler target=1 ! video/x-raw, width=300, height=300 ! tiovxdlpreproc data-type=3 channel-order=1 tensor-format=rgb out-pool-size=4 ! application/x-tensor-tiovx ! tidlinferer target=1 model=/opt/model_zoo/TFL-OD-2000-ssd-mobV1-coco-mlperf-300x300 ! post_3.tensor \
split_31. ! queue ! video/x-raw, width=640, height=360 ! post_3.sink \
tidlpostproc name=post_3 model=/opt/model_zoo/TFL-OD-2000-ssd-mobV1-coco-mlperf-300x300 alpha=0.400000 viz-threshold=0.600000 top-N=5 ! queue ! mosaic_0. \
\
tiovxmosaic src::pool-size=3 name=mosaic_0 \
sink_0::startx="<320>"  sink_0::starty="<180>"  sink_0::widths="<640>"   sink_0::heights="<360>"  \
sink_1::startx="<960>"  sink_1::starty="<180>"  sink_1::widths="<640>"   sink_1::heights="<360>"  \
sink_2::startx="<320>"  sink_2::starty="<560>"  sink_2::widths="<640>"   sink_2::heights="<360>"  \
sink_3::startx="<960>"  sink_3::starty="<560>"  sink_3::widths="<640>"   sink_3::heights="<360>"  \
! video/x-raw,format=NV12, width=1920, height=1080 ! queue ! tiperfoverlay title="AM62A: Quad Camera Object Detection" ! kmssink sync=false driver-name=tidss force-modesetting=true