This section analyzes an application that visualizes both RGB and Infrared streams simultaneously.

This is composed in both GStreamer and TIOVX to allow comparison between the two. Shown in Figure 5-5 is the application in GStreamer and Figure 5-6 shows the equivalent application constructed with TIOVX. The GST commands and TIOVX code can be found in the GitHub [6].

Figure 5-5 Dual Stream Capture and Display with GStreamer

Figure 5-6 Dual Stream Capture and Display with TIOVX

Frames arrive through v4l2 from the corresponding /dev/videoX entries for RGB and IR streams and are processed by the on-chip ISP. Both streams are downscaled to a resolution that fit into the monitor, and the IR stream (in grayscale) is converted to the same color format as the RGB stream. Then, the streams are combined into a single frame with a mosaic feature before displaying to a monitor through the Linux KMS or DRM interface.

The TIOVX and GStreamer applications are equivalent in terms of the processing functions involved, however, there are a few key differences. The TIOVX application builds a TIOVX graph to handle the inner body of the application, which, in this case, is the ISP, down-scaling, color-conversion, and image-merging (mosaic) features. Input from V4L2 and output through KMS or DRM is handled using Linux-level APIs outside the TIOVX graph. However, GStreamer has numerous plugins available to implement these API calls by plugins. The TIOVX application is compiled into a binary application and run whereas the GStreamer pipeline can be represented using a single string that can be run from the command line.

Performance Statistics and SoC Resource Utilization

This section analyzes resource utilization on AM62A74 while running these applications. The measurement application receives remote core utilizations through TIOVX and reads other information like DDR utilization and temperature through memory-mapped registers. This perf_stats application is part of the SDK under the /opt/edgeai-gst-apps/scripts/perf_stats directory. The sampling interval for SoC utilization is 500ms; these are collected across a 20 second window (600 frames per RGB, IR stream) and averaged together into the double bar-chart shown in Figure 5-7.

Figure 5-7 Utilization Comparison of GStreamer and TIOVX For Dual-stream Visualization Pipeline

Error bars in this chart represent the 25th and 75th percentile. Between the two frameworks, accelerator and DDR utilization is at parity for the application running at 30FPS (per input stream) without frame drops. Notably, the MPU (quad Arm® Cortex®-A53 in SMP mode) denoted as mpu1_0 in Figure 5-7, has higher utilization for GStreamer than TIOVX. This is due to increased signalling between the CPU complex and any remote core or accelerator. C7xMMA is unused in this application. Otherwise, Core or HWA utilizations are very similar between the two application frameworks, with TIOVX being slightly more efficient.

Table 5-1 shows latency through individual components of the pipeline. Frame capture and display latency is not included. For each processing task in the application, compare the latency for GStreamer and TIOVX and the total latencies. From here, the TIOVX is generally faster, especially for color conversion; the infrared path is most affected by the difference in application frameworks.

GStreamer internally implements TIOVX nodes for each plugin, so TIOVX nodes always measure faster than the equivalent plugin in GStreamer. Measurements in GStreamer are captured before and after the plugin runs from Linux, whereas measurements in TIOVX can be captured and reported by the remote core running the operation. There is a noticeable improvement⁽¹⁾ in TIOVX compared to GStreamer, especially for the color conversion from grayscale to NV12 format⁽¹⁾.

Another way to compare these applications is in regards interrupts and how frequent inter-processor communication is (that is, A53 messaging C7x, R5F messaging A53, and so forth). Fewer interrupts is better, as this allows the processor to more quickly address pending signals from different peripherals and accelerators. Measure interrupt count from Linux to see the number of interrupts the A53’s (running Linux) received for each application before and after running for 600 frames.

Number of interrupts to Linux across 20 second duration (600 frames per stream). GStreamer shows more interrupts than TIOVX because Cortex A53 cores running Linux must be notified between each plugin/pipeline element. Interrupt counts for individual core's mailbox are captured from /proc/interrupts.

The data in Table 5-2 reflects the general trend that GStreamer is less efficient in terms of CPU interrupts than TIOVX. This is because TIOVX allows all cores to communicate directly, whereas GStreamer requires the cores to flow through the Linux host (A53). Adding AI processing with TIDL shows a similar pattern for C7x interrupts.