How to Use Automated Synthetic Data Generation in AI Model Development for Object Detection

DOWNLOAD PDF VERSION

Introduction

The concept of operations (CONOPS) for aerial platforms has fundamentally changed. Radio frequency (RF) techniques can now effectively jam radio communications and global navigation satellite system (GNSS) navigation systems. This has created a need for fully autonomous platforms with AI perception that rely on passive technologies to operate in today’s contested battlefield and achieve the intended tactical results.

While generalized object detectors may be adequate for many situational awareness applications, other defense applications require fine-grained classification of targets and aim-points. Today, a chasm exists in the data needed to train these AI perception models. Multi-spectral image data are siloed across military, government departments, research organizations, and defense contractors, each with different policies regarding access for commercialization.  Furthermore, the number of military targets is large and growing, and addressing issues such as camouflage, spoofing techniques, and environmental conditions requires rapid data generation and model training in days, rather than weeks or months. In response to this need, Teledyne has developed an automated synthetic data generation and model training process.

Synthetic Data and AI Model Generation

For applications such as autonomous driving, developers typically have access to large training libraries, but for enterprises servicing military technology companies, there are no or limited training datasets, resulting in an acute need for data on targets. While real data is preferred for higher precision classifiers, synthetic data is a practical way to augment existing real datasets or provide data for fine-grain classifiers where no measured (real) data is available.

Teledyne FLIR OEM’s AIMMGen™ (AI Modeling and Model Generation) is a toolchain that combines large-scale automated data generation, intelligent data retrieval, and AI model optimization into a single pipeline designed to meet the needs of Automatic Target Recognition (ATR) algorithm developers. It enables the rapid generation of synthetic data for real-world scenarios, including target object types, positions, orientations, sensor modalities, configurations, trajectories, and environmental states. This toolchain can generate millions of labeled images in days rather than the weeks or months typically required for data collection, without the expense of field data collection and costly labeling services.

AIMMGen automatically generates detailed label information for all generated data, and stores collection metadata with each image set. To support model training and validation across different use cases, the metadata for each image and target is stored in a data lake for rapid curation and construction of datasets. The metadata includes collection parameters, target information, and image parameters. With this information, customized datasets can be curated and prepared for training. Rapid automated optimization of AI models across various use cases and tasks through cloud-scale training and optimization is then enabled by AIMMGen’s extensible and model-agnostic training framework.

Step 1:  AIMMGen Synthetic Data Generation Process

The first stage of the AIMMGen process is data generation, which begins with specifications for location types, sensor resolution, target object types, imaging conditions, sensor configurations, and the desired number of images. Randomized combinations of these specifications are created until the total number of images meets or exceeds the desired number. The data generation pipeline engages a geo-specific simulation engine that includes accurate terrain for the globe, sets the environment conditions, inserts targets sampled from over 20,000 unique models, and begins collecting data. Ground truth is automatically generated as metadata and stored alongside target metadata.

Figure 1.  Synthetic Training Data is Generated in Minutes and at Low Cost

Image metadata, including the final URI of the image location and target metadata, are captured and stored in an Elasticsearch database. These metadata are stored in separate indexes but linked through unique identifiers, enabling users to visualize, filter, and search over data subsets. Once the data has been generated, it can be curated through visualization tools to gain insight and build datasets for specific experiments.

Step 2:  AIMMGen Model Training and Optimization

The second stage of the AIMMGen process is automated model training and optimization. Users specify datasets, desired model performance characteristics, and optimization metrics. AIMMGen orchestrates a cluster of AI training nodes to train, evaluate, and produce high-quality models optimized against baseline and user-specified metrics. The model training framework is agnostic to the specifics of the model architecture and use case, enabling users to create custom data pipelines and model architectures easily within the AIMMGen framework. Output models are tracked and stored in a database for traceability and reproducibility.

Figure 2. Automatic Training and Optimization

Baseline Performance Metrics for AI Perception Model Optimization

In AI perception model development, the following metrics are crucial for evaluating how well the model identifies and classifies objects or instances.

Precision measures the accuracy of the positive predictions made by the model. It is important when the cost of false positives is high. In military applications, misidentifying a non-threat as a threat could lead to unnecessary actions.

Precision = True Positives/(True Positives + False Positives)

Recall, also known as sensitivity or true positive rate, measures the model's ability to identify all relevant instances. This is critical when missing a positive instance has severe consequences, such as the danger of failing to detect an actual threat.

Recall = True Positives/(True Positives + False Negatives)

F1-Score provides a balanced view of the model's performance, ensuring that both precision and recall are considered. This is a good measure of the model's overall performance when dealing with imbalanced datasets and is essential for developing robust perception systems.

F1 - Score = 2 x (Precision * Recall)/(Precision + Recall)

AIMMGen Example Applications and Results

The AIMMGen toolchain has been validated across multiple application and sensor domains, demonstrating its ability to improve air-to-ground detection models, fine-grain classifiers for maritime domain, and more.  

AIMMGen has been shown to increase the performance of detection models in an air-to-ground scenarios. Teledyne FLIR OEM performed a limited experiment to add synthetic electro-optical (EO) and longwave infrared (LWIR) data to an object detection model training dataset alongside the original measured EO and IR data.   The performance was tested with small changes to the training scheme to incorporate the synthetic data. The AIMMGen models achieved a significant improvement in recall. This is demonstrated in the overall improvement in the F1-score.

Table 1.  AIMMGen Models Compared to Real-Only Trained Models

AIMMGen supports the generation of AI models with little to no real training data as shown in a midwave infrared (MWIR) maritime example.  A fine-grained maritime object identifier requires significant, expensive MWIR data with highly detailed labels for vessel type. With limited real data it is only possible to train models to distinguish “small vessel” from “large vessel.”  In contrast, with AIMMGen, a model was produced capable of classifying specific vessel subtypes (Figure 3). This was based exclusively on synthetic data. Performance was validated across multiple real-world collects including fresh water and littoral environments, demonstrating the robustness of a synthetic-only model.

Figure 3: Fine-Grained Identification in MWIR of Maritime Vessels Using Synthetic Data-Trained AI Models

Summary

The challenges of creating large, diverse training datasets for both the EO and IR spectrum and highly specialized applications can be addressed with synthetic imagery. Continuous refinement of synthetic imagery quality and rigorous testing ensure continuous improvement in perception system performance. AIMMGen represents a significant advancement in the field of synthetic data generation and model training, providing a robust solution for developing high-performance AI models in military applications.

Please refer to www.flir.com/prism for more information.