Two vision frameworks are supported on AM6xA devices:GStreamer and OpenVX. To be precise, TI has implemented and conforms to the OpenVX standard; this implementation is designated TI OpenVX (TIOVX). These frameworks enable on-chip hardware accelerators, like C7xMMA and ISP. The frameworks handle IPC and data management under the hood to reduce application-level complexity.

GStreamer (GST) is an Open Source, Linux-based framework for multimedia pipelines. Pipelines are constructed of plugins that implement a function, like capturing an image, changing data formats, scaling sizes, compressing, or writing to a file. Many community plugins are available, but the best performance is attained by using TI-provided plugins that leverage on-chip accelerators. These are provided within the Edge AI SDK and source code is available for modifying or extending TI plugins. GST is particularly effective for building and testing pipelines quickly – in addition to C++, Python, and other language support, GST pipelines can be run straight from command line. For example:

v4l2src device=/dev/video-usb-cam0 ! image/jpeg, width=1280, height=720 ! jpegdec ! video/x-raw, format=NV12 ! kmssink driver-name=tidss sync=true

The above pipeline consists of individual plugins separated by ‘!’ delimiters. This pipeline reads images from a USB camera with v4l2, decodes the JPEG-encoded images, converts the frames’ encoding to NV12, and pushes frames to the display subsystem (DSS) through Linux’s KMS/DRM interface for visualization on a monitor. Pipelines can be large and complex, yet still be run from command line with no coding otherwise.

TIOVX is a lower-level framework for building vision pipelines on heterogeneous SoCs. Analogous to GST, pipelines are composed of nodes within an acyclic graph, where each node runs a function on a target core. For example, processing a raw frame on the ISP or running a neural network on C7xMMA. TIOVX applications are written in C/C++, and require more knowledge of the SOC. Under the hood, GST uses TIOVX to communicate with hardware accelerators. GST requires more interaction with Linux to pass control signals from plugin, whereas TIOVX allows cores to communicate more directly. TIOVX is portable between operating systems, including Linux and QNX, and is appropriate for functional safety (FuSa) certified applications. This makes TIOVX an excellent choice in automotive and other ASIL and SIL-rated use cases. The TIOVX framework is slightly more efficient than GST in terms of interrupt usage; however, frame rate, accelerator utilization, and DDR bandwidth are near parity between GST and TIOVX.

Note that TIOVX applications restrict the user to TIDL_RT for deep learning models; open source runtimes (OSRT) like ONNX Runtime cannot be used at runtime through TIOVX. One role of the OSRT is to supply a backup implementation for layers that TIDL does not support. Therefore, a TIOVX-only application's direct use of TIDL_RT implies that all layers in the neural network must be supported by TIDL.

In most cases, GST is sufficient for building vision-processing applications on TI AM6xA SoCs running Linux. If functional safety and another operating system are required, TIOVX is a more appropriate choice.