Streaming applications strongly benefit from pipelining such that each component of the application runs in a separate process. This is effective when hardware accelerators and multiple CPU cores are available to allow concurrency. TI’s gstreamer plugins are optimized to leverage the heterogeneous architecture of AM6xA processors and are implemented with zero-buffer copy to reduce memory overhead.

The OpTIFlow plugins (i.e., tiovxdlpreproc, tidlinferer, and tidlpostproc) are instrumental in effective pipelining of an AI application since they allow preprocessing, inference, and postprocessing to happen in parallel. Using the OpTIFlow plugins, the only function that needs to run in dedicated application code is making use of the deep learning output. In the retail-scanner application, this means creating a small image that lists the customer’s order. Figure 7-1 illustrates this pipeline. For the exact gstreamer pipeline and a detailed explanation, see the HOW_ITS_MADE.md readme.

With this pipeline on AM62A (Processor SDK 8.6, E2 revision of the starter-kit EVM), the framerate is 15-18 fps. The main bottleneck is Python application code, as this includes additional memory copies and multiple text-drawing function calls with OpenCV. The deep learning accelerator is loaded 25-30%, quad A53 CPU at 30-40% (avg. across all cores), and 32-bit 3200 MT/s LPDDR4 at 20% load.

Optimizing the pipeline to better offload different tasks of the application represented +200% performance boost in terms of FPS and order of magnitude reduction in latency. Much more can be done to optimize, such as developing in C/C++, creating a more efficient function for writing all text to the frame at once (or removing that portion of the application altogether), or even converting application code into a GST plugin.