SPRADP7A February   2025  – March 2025 AM62A3 , AM62A3-Q1 , AM62A7 , AM62A7-Q1 , AM67A , TDA4AEN-Q1

 

  1.   1
  2.   Abstract
  3.   Trademarks
  4. 1Introduction
  5. 2Building Blocks of an RGB-IR Vision Pipeline
    1. 2.1 CSI Receiver
    2. 2.2 Image Signal Processor
    3. 2.3 Video Processing Unit
    4. 2.4 TI Deep Learning Acceleration
    5. 2.5 GStreamer and TIOVX Frameworks
  6. 3Performance Considerations and Benchmarking Tools
  7. 4Reference Design
    1. 4.1 Camera Module
    2. 4.2 Sensor Driver
    3. 4.3 CSI-2 Rx Driver
    4. 4.4 Image Processing
    5. 4.5 Deep Learning for Driver and Occupancy Monitoring
    6. 4.6 Reference Code and Applications
  8. 5Application Examples and Benchmarking
    1. 5.1 Application 1: Single-stream Capture and Visualization with GST
    2. 5.2 Application 2: Dual-stream Capture and Visualization with GST and TIOVX Frameworks
    3. 5.3 Application 3: Representative OMS-DMS + Video Telephony Pipeline in GStreamer
  9. 6Summary
  10. 7References
  11. 8Revision History

2.2 Image Signal Processor

The Image Signal Processor (ISP), also known as the Vision Pre-processing Accelerator (VPAC), on the AM62A device provides essential vision processing functions at the pixel level. The ISP consists of three sub-modules: the Vision Imaging Sub-System (VISS), the Multi-Scalar (MSC), and the Lens Distortion Correction (LDC).

AM62A Image Signal Processor Overview Figure 2-1 AM62A Image Signal Processor Overview

The VISS sub-module processes the raw images and produces demosaiced color images and IR images. The VISS is sometimes also referred as ISP. The MSC sub-module can produce up to 10 downscaled or cropped images. The LDC sub-module performs perspective and geometric transforms and is mostly used to correct lens distortion.

The sub-modules of the AM62A ISP (VPAC) can operate in either memory-to-memory mode or on-the-fly mode:

  • Memory-to-memory mode: The sub-module reads data from memory and writes processed data back to memory.
  • On-the-fly mode: The camera data is sent directly to the Vision Imaging Sub-System (VISS), and the VISS output is sent directly to the Multi-Scalar (MSC) without being stored in memory first.

The VISS processes raw image data in a 4x4 RGB-IR pattern and produces two output streams: one RGB stream and one IR stream, as illustrated in Figure 2-2.

RGB-IR Processing by AM62A ISP Figure 2-2 RGB-IR Processing by AM62A ISP

The 4x4 image data first undergoes raw pixel processing, such as decompanding, WDR merge, defective pixel correction, and lens shading correction. After raw pixel processing, the image data is split into two paths, the RGB processing path and the IR processing path.

  • RGB Processing Path: This path involves typical color processing functions such as demosaicing, auto white balancing, color correction, and noise filtering. Additionally, IR contamination is removed from the RGB data.
  • IR Processing Path: This path is simpler and involves only upsampling and tone mapping.

For applications such as driver monitoring that only need IR images, the IR output is utilized. For applications like video calls or recording that require color images, the YUV or RGB output from the ISP is used. The occupancy (or cabin) monitoring can also use the RGB output to enhance the detection accuracy.