MLLM-Guided Autonomous Object Detection for Open-World Understanding

The gist: We propose an LLM-guided agentic object detection (LAOD) framework that enables fully label-free, zero-shot detection by prompting a Large Language Model (LLM) to generate scene-specific object names, which are then passed to an open-vocabulary detector for localization.

Introduction and Motivation

Object detection traditionally relies on fixed category sets, requiring costly re-training to handle novel objects While Open-World and Open-Vocabulary Object Detection (OWOD and OVOD) improve flexibility, OWOD lacks semantic labels for unknowns, and OVOD depends on user prompts, limiting autonomy A key drawback of OWOD is its inability to provide specific class names or detailed semantic information for the “unknown” objects it detects; it merely identifies them as novel entities In contrast, our framework shifts the burden of specification from the user to the system itself, exhibiting agentic behavior by interpreting the scene and deciding what to detect.

Proposed Framework (LAOD)

The LAOD framework integrates a powerful multimodal large language model (LLM) with an open-vocabulary object detector to achieve fully autonomous, label-free object detection during inference The system operates sequentially, with an input image first being processed by the multimodal LLM to infer a semantic understanding, followed by the generation of a image specific label list, and finally, the execution of object detection by the specialized vision model.

LLM-Guided Label Generation

The LLM-guided label selection module is built upon a multimodal LLM which possesses the unique capability to directly process raw image data alongside textual instructions For a given image X the aim is to use a multimodal LLM L with a specified prompt p and generate image-specific set of generated class names Cg, such that Cg = L(X, p). The prompt p can be modified to control the granularity of object detection; for instance, a prompt instructing the LLM to identify “main objects” or “primary entities” would likely result in broader categories like “person” and “car,” while a prompt requesting “all visible components” or “fine-grained parts” would encourage the LLM to enumerate more detailed labels such as “shirt,” or “tire.”

Agentic Object Detection

The actual object detection is performed based on the labels dynamically generated by the LLM, achieved by employing an open vocabulary object detector. The processed object class names, derived from the LLM’s output, are directly passed as the query to this detector. Formally, the detection process can be expressed as D = O(X, Cg)

Evaluation Metrics

To evaluate our LAOD framework, we propose two complementary metrics: Class-Agnostic Average Precision (CAAP) and Semantic Naming Average Precision (SNAP). CAAP measures the system’s ability to localize objects regardless of their predicted class names, focusing purely on the spatial overlap between predicted bounding boxes and ground truth bounding boxes. SNAP focuses directly on the correctness of the predicted object class names, isolating the quality of dynamically generated object class names independent of localization.

Experimental Results

Experiments were conducted on three datasets: LVIS-Minival, COCO-Val, and COCO-OOD. The results show that the COCO val set yields the highest SNAP scores overall (e.g., 0.54 at the LO interval), indicating strong semantic alignment between LLMgenerated labels and ground truth categories. Notably, our model performs best in terms of localization on the COCO-OOD dataset, with CAAP scores of 0.61 (LO) and 0.56 (MI). Furthermore, across all datasets and intervals, we consistently observe a meaningful gap between CAAP and SNAP scores. Our method achieves the highest scores across three metrics (U-AP, U-F1, U-PRE, and U-REC) on COCO-OOD compared to existing detectors like Grounding DINO.

Qualitative Capabilities

Qualitative analyses illustrated our method’s capacity to detect and label diverse unknown objects, adapt to different scene requirements, and autonomously identify anomalies. In Figure 4, we compare our method with Detectron2 [20] on a scene from the ComplexVAD dataset [14], which is designed for video anomaly detection. Our method successfully detects and labels the object as a “cardboard box,” whereas traditional methods fail to detect it because it was not in the training set. The agentic nature of our method allows for adjustable detection scopes, a flexibility not offered by traditional OWOD models.

Conclusion and Limitations

In conclusion, we introduced a novel LLM-guided, agentic object detection framework that addresses key limitations of traditional, open-world, and open-vocabulary object detection paradigms. We demonstrated the effectiveness of our framework through comprehensive evaluations using novel metrics: Class-Agnostic Average Precision (CAAP) and Semantic Naming Average Precision (SNAP). Future work will investigate the integration of smaller models to provide a more efficient and streamlined label generation process tailored to object detection. The ability to adjust detection granularity and focus through flexible prompting provides fine-grained control over the scope and semantics of object detection, a crucial advantage in task-specific applications.

Supplementary Material Details

In case of having user-defined class names Cu, we first merge them with the generated labels Cg by defining Cm as Cm = Cu ∪ g ∈ Cg max u∈Cu sim(Emb(g), Emb(u)) < τ (S1). This formulation ensures that a generated class g is added to Cu only if its CLIP embedding is less similar than the threshold τ to the embedding of every existing user-defined class u. The detection process can then be expressed as D = O(X, Cm) (S2). Unknown Average Precision (UAP) is computed as the area under the precision-recall curve obtained by varying the confidence threshold on predictions, expressed as U-AP = Z 1 0 P(U-REC) dU-REC (S6). This formulation explicitly penalizes the over-prediction of object names while rewarding semantically aligned naming, regardless of spatial localization. The Unknown Average Precision (UAP) metric evaluates the ability of an open-world object detection system to detect and localize objects from unknown or novel categories—those not seen during training.

References

[1] Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real-time open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16901–16911, 2024

[2] Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don’t know by virtual outlier synthesis. arXiv preprint arXiv:2202.01197, 2022

[3] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303–338, 2010

[4] Akshita Gupta, Sanath Narayan, KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Ow-detr: Open-world detection transformer.

1. Agentic Object Detection for Open-World Understanding

The improved system can perform fully label-free, zero-shot detection by prompting a Large Language Model to generate scene-specific object names which are then passed to an open-vocabulary detector for localization. This enables the system to adapt its goals dynamically based on the scene context.

2. Dynamic Detection Granularity Control

The framework allows control over detection detail by modifying the prompt, enabling guidance for specific levels of abstraction; for instance, a prompt instructing the LLM to identify main objects versus one requesting all visible components. This capability allows the system to adapt its perception goals on the fly without re-training.

3. Disentangled Performance Analysis

The introduction of Class-Agnostic Average Precision (CAAP) and Semantic Naming Average Precision (SNAP) enables a disentangled assessment of localization accuracy and semantic naming quality. This provides a more nuanced and comprehensive assessment by allowing researchers to understand the strengths and limitations of both components independently.

4. Superior Unknown Object Detection

The system achieves state-of-the-art performance in unknown object detection, surpassing existing detectors across metrics like UAP, U-F1, U-PRE, and U-REC on COCO-OOD. This demonstrates its capability to discover and localize novel object instances effectively while maintaining both spatial and semantic consistency.

5. Robust Vocabulary Refinement

The supplementary material introduces a mechanism that prevents redundancy by constructing the merged class set Cm using the rule: "Cm is defined as: Cu U g in Cg max u in Cu sim(Emb(g), Emb(u)) < tau (S1). This ensures a refined vocabulary for the detector by preventing the inclusion of redundant or semantically overlapping labels."

Object detection typically relies on predefined category vocabularies, while open-world and open-vocabulary approaches remain limited by generic unknown labels or externally specified detection concepts. We propose LAOD, an MLLM-Guided Autonomous Object Detection framework in which a Multimodal Large Language Model (MLLM) generates an image-specific detection vocabulary that is grounded by an open-vocabulary detector. This decomposition separates semantic concept discovery from spatial localization and removes the need to manually enumerate target categories at inference time. We evaluate LAOD using Class-Agnostic Average Precision (CAAP), Semantic Naming Average Precision (SNAP), and Semantic Localization Average Precision (SemLoc-AP), which measure localization, semantic naming, and their correct association, respectively. Experiments on COCO-Val, LVIS-minival, and COCO-OOD across multiple MLLMs and detectors demonstrate strong performance across common, long-tailed, and out-of-distribution objects, while showing that grounding quality remains a major determinant of end-to-end performance. LAOD also substantially outperforms both direct MLLM-based object detection and dedicated models trained specifically for open-ended detection. Overall, the results support a modular formulation of autonomous object detection in which MLLMs determine what to detect and specialized open-vocabulary detectors determine where those concepts occur.