Generative Spatiotemporal Data Augmentation

The gist: Generative spatiotemporal augmentation using video diffusion models can effectively boost object detection performance in data-scarce settings by expanding data along underrepresented axes like camera viewpoint and scene dynamics.

Generative Spatiotemporal Augmentation

The core idea is to leverage off-the-shelf video diffusion models to generate realistic 3D spatial and temporal variations from a given image dataset, which yields richer geometric and temporal coverage than the original dataset. This process involves generating novel viewpoints by selecting a subset of frames indexed by S ⊆ to form Dnew = (It t ∈ S), guided by a camera trajectory C = (ct) where each ct ∈ SE(3) specifies the rotation and translation relative to the initial view. Furthermore, temporal dynamics are diversified by prompting G to synthesize dynamic scene evolution using a temporal control signal τ, such that (It t=1 = G(I0; τ). The resulting augmented dataset Daug is then Dorig ∪ Dnew, which is used for training.

Coverage Expansion and Fidelity Trade-off

The central hypothesis is that generative spatiotemporal augmentation expands the effective coverage of a limited training set over the unseen target distribution by exploring underrepresented axes of variation. Quantifying coverage involves using the precision–recall framework, where recall represents the fraction of validation samples whose features are covered by the training set D. The analysis shows that both image- and object-level recall rise sharply as the number of synthetic frames increases, confirming that generated samples fill underrepresented regions of the data manifold. Because video diffusion models produce temporally correlated frames, Kernel Inception Distance (KID) is used to quantify distribution shift, revealing a coverage–fidelity trade-off. A 1:1 sampling ratio between real and generated images achieves the best balance between expanding coverage substantially while maintaining overall realism.

Annotation Transfer and Disocclusion Handling

To enable training, an automated annotation pipeline transfers bounding boxes across generated views by employing a state-of-the-art video instance segmentation model Mseg (e.g., SAM 2) to produce binary masks for each frame. From these masks, the tightest enclosing axisaligned bounding box, Bjt = BoundingBox(Mj t), serves as the annotation for the corresponding synthetic image. A practical challenge is disocclusion—regions newly revealed and unlabeled in generated views—which can be mitigated through tracking-based masking or pseudo-labeling. Tracking-based masking involves sampling a uniform grid of points P0 on the original image and tracking them across generated frames to define visible regions, which is applied by removing disoccluded pixels as I't = It ⊙ Mt valid.

Performance Gains and Configuration Guidelines

Experiments show significant performance gains, such as +3.8 mAP on a COCO [32] sub-dataset, +2.8 mAP on VisDrone [6], and +5.9 mAP on Semantic Drone [9]. The most effective augmentation strategy depends on the dataset’s characteristics. For COCO5k, spatial augmentation with orbital camera trajectories performs best, while for VisDrone scenes, temporal augmentation with scene dynamics is more effective. The optimal configuration involves a 1:1 ratio between real and generated images, modest camera motion, and CLIP-filtered frames to balance realism and diversity. Disocclusion handling yields only marginal benefits and can typically be ignored.

Scaling with Dataset Size and Model Capacity

The performance scaling with dataset size indicates that the benefit of spatiotemporal augmentation diminishes as the size of the original dataset increases, as real data covers more of the underlying distribution. For VisDrone, where the dataset is limited to 6,471 images, a ∼5% improvement is observed from our augmentation. Furthermore, for low-capacity YOLO models, the initial gap between augmentation strategies diminishes as training data increases. High-capacity YOLO models exhibit a much larger performance gap that persists even at larger dataset sizes. The method remains effective even when applied in pretraining–finetuning pipelines, complementing pretrained representations rather than duplicating them.

Ablation and Practical Recommendations

Ablation studies confirm that enforcing a 1:1 sampling ratio between real and synthetic data and sparsely sampling 8 frames (e.g., sample every 5 frames) is the optimal strategy, achieving a +4.8 gain in mAP@50 over the baseline. Data filtering using a CLIP-based strategy to remove low-quality synthetic samples provides a marginal additional improvement. The paper concludes that spatiotemporal augmentation with video diffusion models is an effective way to boost object detection in data-scarce settings by expanding data along the critical axes of camera viewpoint and scene dynamics.

How it works

The method utilizes a pre-trained image-to-video diffusion model G to generate a video sequence (It) from an original image I0, either by simulating 3D novel views using a camera trajectory C or by synthesizing temporal dynamics using a control signal τ. This generates the synthetic dataset Dnew = (It t ∈ S), which is then combined with the original dataset to form Daug, allowing for training.

Key Components of the Pipeline

The pipeline includes:

1. 3D Novel-View Augmentation, where G generates frames as (It) T t=1 = G(I0, C), and a subset S is selected to form Dnew.

2. Temporal-Dynamics Augmentation, where (It) T t=1 = G(I0; τ), and frames are sub-selected to augment Dorig.

3. Automated Annotation Transfer, which uses a video instance segmentation model Mseg to produce binary masks (Mj t) from the generated video V = (It) T t=1 and initial box Bj0, then computes the tightest enclosing bounding box Bjt.

4. Disocclusion Handling, which employs tracking-based masking by sampling a uniform grid of points P0 on I0 and tracking them across generated frames to define visible regions.

Detector Training Settings

The YOLO11 models are trained from scratch on the target dataset using an input resolution of 640 × 640 pixels and a batch size of 64. Training employs Stochastic Gradient Descent (SGD) with a momentum of 0.937 and a weight decay of 5 × 10−4, trained for 400 epochs with a warmup period of 3 epochs. Standard data augmentation includes Mosaic augmentation (probability 1.0), horizontal flipping (probability 0.5), translation (±10%), scaling (±50%), and color space augmentations in the HSV domain.

Limitations

The limitations acknowledged include the diminishing performance gain as the size of the original dataset increases, especially since real data covers more of the underlying distribution. The computational cost of video diffusion models is significant, and any failures in SAM directly propagate to the annotation process.

Practical Recommendations

The paper provides practical guidelines for choosing an appropriate spatiotemporal generative setup, transferring annotations to synthetic frames, and addressing disocclusion—regions newly revealed and unlabeled in generated views. Effective configurations are guided by the Recall–KID analysis, suggesting a 1:1 ratio between real and generated images and sparse frame selection provides a good balance between realism and diversity<ref:2512.

1. Robust object detection in data-scarce regimes through spatiotemporal augmentation using video diffusion models can achieve substantial gains, as demonstrated by +3.8 mAP on a COCO [32] sub-dataset. This allows AI systems to perform effectively on UAV-captured imagery where annotations are scarce.

2. Automated annotation transfer pipelines leveraging temporal consistency can produce accurate bounding boxes for all generated frames by employing a state-of-the-art video instance segmentation model like SAM 2. This capability enables training object detectors with richer geometric and motion variation without additional manual annotation cost.

3. AI systems can effectively mitigate disocclusion during viewpoint changes by using tracking-based masking, which identifies newly visible regions through point tracking on original frames and generates a binary mask to suppress unlabeled areas. This ensures that the model learns from annotated content while preserving useful visual information, as shown in Figure 4(e).

4. The augmentation strategy can be tailored to dataset characteristics by choosing spatial or temporal augmentation based on the dominant source of variation. For object-centric datasets like COCO5k, orbital camera trajectories are most effective, whereas for aerial imagery like VisDrone, temporal dynamics with scene evolution provide the best performance gains.

5. Performance scaling with dataset size can be exploited to maximize benefits; for instance, in low-data regimes like UAV scenarios (VisDrone), spatiotemporal augmentation yields a ∼5% improvement compared to the baseline when training set size is limited. This suggests that generating additional data is especially valuable where real data coverage is insufficient.

6. Model capacity can be leveraged by using high-capacity models, such as YOLO11x, which exhibit a much larger performance gap that persists even at larger dataset sizes compared to lower-capacity models like YOLO11n. Stronger models can more effectively exploit the expanded coverage provided by spatiotemporal augmentation.

7. Pretraining on large datasets like COCO can be combined with spatiotemporal augmentation during finetuning, yielding consistent gains over both the raw-data baseline and the pretrained baseline when applied to domain-specific data like VisDrone. This demonstrates that our approach complements pretrained representations rather than duplicating them, providing meaningful additional signal during finetuning.

8. The system can be optimized by enforcing a 1:1 sampling ratio between real and synthetic data and sparsely sampling frames (e.g., sample every 5 frames), which was found to achieve the optimal performance of 41.1 mAP@50 on COCO5k over the baseline.

We explore spatiotemporal data augmentation using video foundation models to diversify both camera viewpoints and scene dynamics. Unlike existing approaches based on simple geometric transforms or appearance perturbations, our method leverages off-the-shelf video diffusion models to generate realistic 3D spatial and temporal variations from a given image dataset. Incorporating these synthesized video clips as supplemental training data yields consistent performance gains in low-data settings, such as UAV-captured imagery where annotations are scarce. Beyond empirical improvements, we provide practical guidelines for (i) choosing an appropriate spatiotemporal generative setup, (ii) transferring annotations to synthetic frames, and (iii) addressing disocclusion - regions newly revealed and unlabeled in generated views. Experiments on COCO subsets and UAV-captured datasets show that, when applied judiciously, spatiotemporal augmentation broadens the data distribution along axes underrepresented by traditional and prior generative methods, offering an effective lever for improving model performance in data-scarce regimes.