People detection across diverse settings in real-time

People detection is a common vision AI task, and many public datasets are available. Most of TI's pretrained models have been trained on the COCO dataset from Microsoft, which contains nearly 100 different types of objects, including people. Large crowd-sourced datasets like COCO contain images from a wide variety of camera types, lighting conditions and scenes. This variety will improve the robustness and accuracy of the model.

Data can also be collected manually if your use-case requires maximal accuracy in specific scenes or conditions. It is best to collect data with the same type of camera, lens included, that will be used in practice.

Before training, the data must be annotated or labeled. Typically, humans in the frame are detected and localized with an object detection neural network, which uses a bounding box label represented as two pairs of X,Y pixel coordinates. In some applications like blurring the background for video conferencing, segmentation models are the better choice; these will use an image mask (i.e. a class label for each individual pixel) or polygons to represent the precise regions where people are in the image.

When collecting data for people detection, consider carefully what kind of variations will exist in the setting and scene. 

Will the lighting conditions be consistent or variable? Is the same camera and lens always used? Will people be in full view always, or will they be only partly in the frame or even blocked, i.e. occluded, by other objects or people? 

Your dataset should incorporate these variations as needed. Variations can also be artificially introduced using dataset augmentation libraries like albumentations or imgaug, but take care that labels still match where the people are and added noise does not hide anyone.

Developers may start from pretrained models in the TI Model Zoo. If this gives sufficient accuracy for the application, then this is the fastest and simplest way to build your product. 

Models can be trained on new data with CCStudio™ Edge AI Studio or edgeai-modelmaker within the edgeai-tensorlab repository on GitHub. This enables prevalidated object detection models like YOLO X. Your model will be trained on your dataset, starting from a version of the model that was pretrained on the MS COCO dataset. This reduces the number of data samples needed by several orders of magnitude. A proof-of-concept model may only need a few dozen images, though a production model will require several hundred at a minimum, ideally thousands for robust accuracy.

Otherwise, developers should train their model with frameworks like PyTorch or Tensorflow, and export the final version to ONNX or LiteRT (formerly tensorflow-lite) format, respectively. The exported model must be compiled with our deep learning software like edgeai-tidl-tools to be accelerated on the C7 NPU for AM6xA and TDA processors.

Many model architectures, like YOLO-X are defined to have subvariants like nano (n), small (s) or large (l). This will keep the same core structure of the network, but may increase or decrease the number of repeated layers or the size of those layers. Such subvariants makes it simple for developers to scale the model's accuracy and complexity (and thus, latency) without requiring they investigate entirely new architectures. Similarly, larger resolution models will increase latency, but can also improve accuracy.

Subvariants of many models, e.g. N=nano, S=small, L=large will increase the complexity and accuracy, at the cost of inference latency. Similarly, larger resolution images will increase latency, but can also improve accuracy. It is often best to start from the latency requirement, seek models that meet this target and then chose among those with the best accuracy on standard metrics like MS COCO. The TI Model Zoo hosts per-device model benchmarks to aid this selection process.

Model deployment requires the model be compiled beforehand to optimize it for the target hardware. With tools like Edge AI Studio and edgeai-modelmaker, compilation is automatic. Otherwise, compiling models will require a separate step through software packages like edgeai-tidl-tools on the TI GitHub. 

Model artifacts are deployed through runtimes like ONNX Runtime, LiteRT (formerly tensorflow-lite), and TVM using TI Deep Learning (TIDL) as the hardware backend for acceleration. 

To deploy the model into an end-to-end vision application, use edgeai-gst-apps, which composes the pipeline with multiple stages of hardware acceleration for pre-processing and post-processing the image, in addition to accelerating the AI model itself.

In addition to edgeai-gst-apps, on GitHub, a demo application showing people tracking in retail environments that incorporates more application logic and visualization. Live tracking indicates how long a person has been in the scene and shows statistics for occupancy. Additionally, a heatmap displays where occupants have spent their time, which may hold valuable information for advertising or retail product positioning.