Less is More: Label-Guided Efficient Summarization of Procedural and Instructional Videos

Video summarization transforms long videos into concise representations that are easier to document and analyze, especially in high-stakes domains such as surgical training. However, long-form videos often require dense frame processing or supervised training pipelines, which can be computationally expensive and may still miss important procedural content. We present PRISM: Procedural Representation via Integrated Semantic and Multimodal Analysis, a zero-shot and training-free framework for frame-efficient procedural video summarization. PRISM selects fewer than 5% of video frames while retaining over 84% semantic content and improving over baselines by up to 7.5%. Rather than exhaustively processing frames, PRISM uses lightweight visual filtering and dynamically generated procedural labels as semantic anchors to select meaningful frame-label groups. This selective inference design preserves key actions, transitions, and contextual details while reducing the number of visual inputs passed to downstream vision-language captioning stages. We evaluate PRISM on YouCook2 and ActivityNet Captions, with additional studies on keyframe selection benchmarks and surgical video datasets. Across procedural and domain-specific video tasks, PRISM achieves strong semantic alignment and accuracy, suggesting that efficient multimodal video understanding can be achieved by grounding generation in dynamically generated semantic anchors.