VESSI - VLM-Enhanced Support for Surveillance and Investigations

The gist The proposed VESSI framework is a VLM-based system designed to enhance automated video surveillance analysis through prompt-driven interrogation of video sequences where salient visual features are converted into textual descriptions.

How it works

1. Phase 1: Acquisition involves processing a categorized video dataset, adopting the UCF-Crime dataset, to properly assess the capabilities and performance of VESSI The input to the system is a categorized video dataset  = 1, …, 13>, where each subset  c contains all videos v belonging to the c-th crime class To properly assess the capabilities and performance of VESSI, we adopt the UCF-Crime dataset [16], a large-scale surveillance video collection organized into real-world crimerelated classes, as detailed in Table II.

2. Phase 2: Analysis processes each video v ∈  i by extracting a frame set  using uniform sampling at a fixed rate The goal is to evaluate how well different VLMs detect relevant visual content, given category-specific natural language prompts p ∈  = p1, p2, p3>. For each frame, we apply three prompts p1, p2, and p3 of increasing semantic complexity. These prompts are formulated as a short binary decision, with responses such as “uncertain,” or any output that cannot be parsed as either decision, treated as abstentions and normalized as Unclear.

3. Phase 3: Threat Evaluation assesses label-consistent retrieval by associating affirmative responses with the video-level category. To assess label-consistent retrieval, each affirmative response is associated with the video-level category used to forInput: Video v ∈ categorized video dataset  = 1, …, 13>.

Key Components and Metrics

The framework utilizes several components to ensure operational utility and traceability. The key components include:

** Composite Model Utility Score (CMUS)**

This metric integrates detection coverage, time efficiency, detection rate, and a refined hallucination index into a single, interpretable score. The CMUS in Eq. 4 integrates all four components using a weighted linear combination. For this experiment, we set α = 1.0: Detection coverage is directly rewarded as it reflects the ability of the model to identify relevant activity across various scenarios.

** Detection Coverage (Cm)**

Coverage measures retrieval breadth: a high value means that fewer videos are entirely missed during triage, but it does not imply that every positive response is correct. For each model, coverage is computed as the proportion of videos containing at least one positive response to the category-specific query.

** Time Saved (Tm)**

To evaluate computational efficiency, we compared the average inference duration Tin per model against the total actual playback time Tor, as the latter offers a realistic baseline term for human or real-time monitoring systems. We then computed the average inference time for each model by averaging over the three prompt complexity levels. To quantify the proportional gain, we define the Time Saving as Ts (%) for model  as reported in Eq. 2.

** Refined Hallucination Index (Hm)**

For each Model×Video×Prompt triplet, the detection rate is the proportion of analyzed frames for which the model returns Yes. Because exhaustive frame-level ground truth is unavailable, we treat persistent affirmative activation as an operational proxy for hallucination risk. We group detection rates into five intervals and assign progressively stronger penalties to form the refined hallucination index.

Experimental Evaluation and Findings

The framework was tested with four state-of-the-art models: LLaVA-1.5 [32], InstructBLIP [35], Qwen2.5-VL [37], and DeepSeek-VL [40]. The evaluation involved comparing these models across detection behavior, efficiency, and semantic reliability.

The results show that Qwen2.5-VL obtained the highest CMUS score (1.6031), followed by LLaVA (1.1909). Its low detection rate (0.1579) and negative hallucination index (−0.1598) offset its lower coverage. InstructBLIP showed the strongest persistent affirmative activation, with 2,032 pairs in the ≥ 95% band and only 106 below 10%, resulting in the highest hallucination index, Hm = 1.2030.

Operational Consistency and Trade-offs

The CMUS thus provides an operational aggregate for comparative model selection in security, surveillance, and forensic contexts. The ranking is consistent with the manually referenced evaluation and the normal-video control. This agreement supports the use of persistent affirmative activation as an operational proxy for hallucination risk and supports the comparative CMUS result within the evaluated setting.

The framework successfully answers four research questions, demonstrating that VLMs can improve Law Enforcement frameworks in video surveillance analysis, and that no single architecture dominates every operational dimension, allowing for selection based on deployment priority. The framework supports flexible, lightweight, and traceable scene interrogation for real-world surveillance video.

The final results position VESSI as a practical triage and benchmarking framework whose alerts guide subsequent human assessment. All models processed frames faster than the configured 1 Hz sampling interval, with average time savings from 64.59% to 90.96%. The same model ordering was obtained from Hm, the normal-video control, human-reference FPR, and both crimeand normal-video CMUS evaluations. The results position VESSI as a practical triage and benchmarking framework whose alerts guide subsequent human assessment.

The paper introduces VESSI, a modular pipeline for prompt-based, per-frame surveillance analysis with 7Bparameter VLMs. Four models were evaluated across 13 crime categories and three prompt complexities using detection coverage, time saved, detection rate, and the hallucination index, combined through CMUS. The framework retains the frames, timestamps, normalized responses, and computational logs needed for human inspection. All models processed frames faster than the configured 1 Hz sampling interval, with average time savings from 64.59% to 90.96%. The same model ordering was obtained from Hm, the normal-video control, human-reference FPR, and both crimeand normal-video CMUS evaluations. The results position VESSI as a practical triage and benchmarking framework whose alerts guide subsequent human assessment.

The paper introduces VESSI, a modular pipeline for prompt-based, per-frame surveillance analysis with 7Bparameter VLMs. Four models were evaluated across 13 crime categories and three prompt complexities using detection coverage, time saved, detection rate, and the hallucination index, combined through CMUS. The framework retains the frames, timestamps, normalized responses, and computational logs needed for human inspection. All models processed frames faster than the configured 1 Hz sampling interval, with average time savings from 64.59% to 90.96%.

1. textbfPrompt-Driven Semantic Interrogation for Forensics (VESSI Core): The system can perform prompt-driven interrogation of video sequences where salient visual features are converted into textual descriptions. This allows for category-specific natural language prompts (Easy, Medium, Hard) to query the VLM, enabling users to test direct binary queries, broader phrasing, or reasoning-intensive questions on every frame of a video.

2. textbfComposite Model Utility Score (CMUS) for Robust Model Selection: The system introduces the CMUS score which integrates detection coverage, time efficiency, detection rate, and a refined hallucination index to provide an operational aggregate for comparative model selection. This allows deployment teams to choose the best model based on trade-offs between semantic selectivity, latency, hardware demand, and robustness, as demonstrated by Qwen2.5-VL obtaining the highest score (1.6031).

3. textbfTraceable Evidence Retrieval with Confidence Filtering: The framework ensures that only reliable findings are flagged for human review by implementing a confidence threshold: Only frames with a positive response and zeta ≥ tau are stored. This transforms the output into traceable alert[s] that an officer can review in the context of the original footage, ensuring that alerts are filtered based on model internal uncertainty.

4. textbfReal-Time Operational Efficiency Gains: The system provides significant speed-up for analysis by demonstrating that models like InstructBLIP and Qwen2.5-VL save 90.96% and 85.32% of the inspection time compared to raw playback, enabling faster triage in operational settings.

5. textbfGuaranteed Consistency Across Evaluation Methods: The system achieves cross-validation by showing that the CMUS ranking is reproduced by the normal-video CMUS evaluation and matched the falsepositive-rate ordering obtained from 5,909 manually referenced frames, supporting the use of the score within the evaluated setting.

Automated video surveillance analysis has become a critical component of intelligence infrastructures and Law Enforcement agencies. Traditional systems lack the semantic module for comprehensive situational awareness and forensic tasks, limiting their ability to interpret events meaningfully or support post-incident investigations. This slows operational insight and increases the burden on human analysts. Recent advances in Vision-Language Models (VLMs) offer promising pathways to bridge this gap. To address this, we propose VLM-Enhanced Support for Surveillance and Investigations (VESSI), a VLM-based framework designed to enhance automated video surveillance analysis through prompt-driven interrogation of video sequences where salient visual features are converted into textual descriptions. We test our framework with four state-of-the-art models. Since most datasets for this task are unlabeled, we also propose the Composite Model Utility Score (CMUS) to assess VLM performance. Experimental results show that our solution substantially improves the analysis capabilities of human operators and enhances the flexibility of automated surveillance systems. In our evaluation, the most reliable model flagged potentially relevant activity in more than 66% of the videos while reducing review time by more than 85%, offering a practical balance between selectivity and efficiency. The model ordering produced by the reference-free CMUS evaluation was reproduced by the normal-video CMUS evaluation and matched the false-positive-rate ordering obtained from 5,909 manually referenced frames. This agreement supports the operational use of the score within the evaluated setting.