In this example application, a representative GStreamer pipeline is run for occupancy monitoring, driver monitoring and video telephony on live stream from an OX05B1S camera.

For OMS and DMS, the infrared frames are passed through a segmentation and object detection model, respectively. These models emulate a seatbelt detection and face detection task, respectively. The RGB frames are H.265 encoded and saved into a file. This represents a starting point for an RGB-IR in-cabin monitoring applications. Note that the deep learning models used here are not optimized ⁽¹⁾for DMS/OMS. A final product needs to extend this to include more DMS and OMS models and postprocessing for a robust and feature-rich product.

Figure 5-8 GStreamer Application Flow For Dual-stream RGB-IR For DMS-OMS With Video Recording

In this application, depicted by Figure 5-8, the RGB frames are kept only for file storage by scaling and encoding in H.265 format. Infrared input is processed and scaled down before splitting into OMS and DMS paths. DMS runs at 30 FPS whereas OMS runs more slowly at 5 FPS (with excess frames dropped in OMS path). From here, the infrared frames are prepared for deep learning by scaling, converting colorspace to RGB (although final models can be optimized for grayscale input), preprocessed, and run through a CNN on the C7x deep learning accelerator. Any post-processing for these models run in an application code exposed by the GStreamer appsink plugin.

Figure 5-9 shows the core load on AM62A while running this application. The VISS-ISP hardware accelerator is nearly maxed out for the 315 MP/s capacity. However, there is plenty of headroom left on processing cores. The 4x Arm® Cortex®-A53s are only used approximately 26%, leaving plenty of space for application code and other services. The C7xMMA AI accelerator is less than 50% loaded, leaving headroom for other AI models, and many further optimizations can be performed to maximize IR image analysis. The 32-bit, 3200 MT/s DDR bus shows utilization of 35%, and can reach approximately 50-60% before this shared resource becomes contended for. Note that DDR utilization must be strongly considered when optimizing the overall system.

Individually, the OMS, DMS, and telephony/recording portions of the pipeline have latencies of approximately 85ms, 51ms, and 67ms, respectively (not including frame-capture latency).

Figure 5-9 shows compute, accelerator, DDR utilization for the DMS-OMS reference GStreamer application. Note that models here are off-the-shelf from TI model zoo and used to emulate a realistic load for a simple DMS/OMS application. There is headroom for further deep learning and image analysis algorithms, including opportunity to optimize and reduce DDR load.

Figure 5-9 Core Utilizations for RGB-Ir DMS/OMS Application