Cattle-CLIP: A Multimodal Framework for Dairy Cattle Behaviour Recognition from Video

arXiv:2510.09203 · cs.CV · Submitted 2025-10-10 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Cattle-CLIP: A Multimodal Framework for Dairy Cattle Behaviour Recognition from Video".

Jane: Here's the gist of the paper. The gist The Cattle-CLIP framework is a domain-adaptive vision-language model that reformulates cattle behaviour recognition as crossmodal semantic alignment rather than purely visual classification, achieving 96.1% overall accuracy across six behaviours in supervised settings Cattle behaviour is a crucial indicator of animal health, productivity and overall well-being.

Tom: Robust behaviour recognition in real-world farm environments remains challenging due to several data-related limitations, including the scarcity of well-annotated livestock video datasets and the substantial domain gap between large-scale pre-training corpora and agricultural surveillance footage.

Jane: Existing approaches can be broadly classified into three categories based on the type of data source: contact-based, non-contact and hybrid methods.

Tom: Contact-based methods use wearable sensors such as inertial measurement units (IMUs) to collect behavioural data for subsequent analysis.

Title and authors: Tom: So we're looking at a paper called "Cattle-CLIP: A Multimodal Framework for Dairy Cattle Behaviour Recognition from Video." It sounds like it’s tackling the big problem of recognizing what cattle are doing on a farm, but they're doing it using both video and language together.

Jane: Exactly. The title tells you it's not just about looking at pictures; it's about a multimodal framework, meaning they’re combining different types of data to get better results for those six key behaviors we talked about earlier.

Lu: What’s interesting is the authors are moving away from just classifying things visually and instead focusing on aligning the meaning between what the camera sees and what you write down about that behavior.

Meng: So, instead of just training a model to say "this is lying," they're trying to get the visual understanding to match up with the language description of lying across different videos.

Lalam: It suggests that by using this crossmodal alignment, the model gets a more robust way of knowing what’s happening in the video, which is important because those farm environments are so varied.

The paper's summary: Tom: The core idea they propose with Cattle-CLIP is reformulating behavior recognition as this crossmodal semantic alignment instead of just a purely visual classification task. They aren't fine-tuning the visual parts directly in the same way other models might be doing.

Jane: So, instead of just trying to make the vision model better at spotting cows, they’re building a way for the vision and language representations to speak the same semantic language across time.

Lu: They do this by including a lightweight temporal integration module that aggregates frame-level features over time. This is important because it lets them capture those subtle and persistent patterns in behavior that are hard to catch with just single images.

Meng: Capturing subtle patterns across time is what I worry about for deployment at scale, because if the temporal aggregation gets noisy, you'll get inconsistent results when you deploy it on real farm footage.

Lalam: It seems like they’re trying to solve the issue of temporally consistent behavioral patterns which is particularly important for distinguishing between similar states like lying-self-grooming and lying-ruminating.

The paper's improvements: Tom: They introduced a few specific ways they improved the system. First, they added customized data augmentation strategies to handle spatial deformation and appearance variations that happen in real farm settings.

Jane: That’s interesting because we talked about how crops were sometimes removed in other work, but here they specifically removed random cropping because it can lose crucial head information during a random crop operation.

Lu: They decided to fill the frames instead of cropping when aspect ratios vary, which keeps the native aspect ratio before resizing to the target resolution. This is a direct response to how cows move and change posture in surveillance videos.

Meng: From an engineering standpoint, handling those spatial deformations robustly is key for production, because you can't just rely on perfectly centered objects in a video feed.

Lalam: They also introduced tailored behavior prompts to help preserve the semantic completeness of the descriptions and improve that cross-modal alignment between vision and language representations.

Conclusion: Tom: So, to wrap up, Cattle-CLIP shows strong performance when tested in both fully-supervised settings, achieving ninety-six point one percent overall accuracy across six behaviors in those controlled supervised settings <ref:2510.09203#pg2>. They also looked at few-shot learning scenarios where they used a base-to-novel transfer strategy with replay to improve recognition of new behaviors.

Jane: That means that even when you don't have tons of labeled data for a specific new behavior, the model can use what it learned from the base ones to get better results. It points toward sample efficiency being a real thing in precision livestock systems.

Lu: The implication is that multimodal semantic supervision might be a promising direction for making these systems more efficient because it leverages both visual and language understanding simultaneously.

Meng: What I’m concerned about is how this scales to different farms; the paper doesn't really test how well it generalizes across completely different farm management systems or video collection methods.

Lalam: For me, the main thing here is that they released the CattleBehaviours6 dataset with its standardized ethogram, which gives researchers a real resource to do reproducible work on this topic.

Tom: Yeah, so to sum up, Cattle-CLIP is using temporal integration and tailored prompts to achieve high accuracy by aligning vision and language semantically. We’ll be looking at how those temporal modules can get even more expressive in the future.

Jane: That sounds like a good direction for the next steps as we look at this work. We've covered the Cattle-CLIP framework today, and we'll be talking about some other interesting papers next time.

Huimin Liua, Jing Gao, Daria Barana, Axel X Montouta, Neill W Campbell, Andrew W Dowseya

Bristol Veterinary School, University of Bristol

cs.CV

Submitted: 2025-10-10

Updated: 2026-10-16

Journal ref: AgriEngineering 2026, 8(10), 410

DOI: 10.3390/agriengineering8100410

License: http://creativecommons.org/licenses/by/4.0/

The gist: The gist The Cattle-CLIP framework is a domain-adaptive vision-language model that reformulates cattle behaviour recognition as crossmodal semantic alignment rather than purely visual classification,

Key concepts

Crossmodal Semantic Alignment: This is the core idea where the model learns to match visual information (images of cattle) with textual descriptions (labels like 'feeding') in a shared semantic space. Instead of just classifying an image, it ensures that the visual features and the language used to describe them are semantically consistent, making recognition more robust.
Temporal Integration Module: This module extends standard image-based learning to video by integrating features across multiple frames. It helps the model understand how a behavior evolves over time, which is essential for distinguishing between similar actions that happen sequentially, such as 'standing-ruminating' versus 'lying-ruminating'.
Domain Adaptation via Tailored Prompts: The researchers customized the text prompts used to describe cattle behaviors. This involves replacing ambiguous words with more interpretable phrases. This technique helps the model better understand and align its visual features with the language, especially when dealing with complex or nuanced descriptions of animal actions.
Base-to-Novel Few-Shot Learning: This strategy is used to teach the model new, rare behaviors using only a few examples. It works by first learning general knowledge from many well-known base behaviors and then adapting that knowledge to recognize entirely new ones. A 'replay' mechanism helps prevent the model from forgetting what it already knows.

Terminology

Semantic Alignment — 2× in this paper · explained in 6 episodes
Accuracy — 1× in this paper · explained in 1 episode
Benchmark — 1× in this paper · explained in 2 episodes
Distribution Shift — 1× in this paper · explained in 10 episodes
Few-Shot Learning — 1× in this paper · explained in 5 episodes
Fine-Tuning — 1× in this paper · explained in 5 episodes

Summary

The gist The Cattle-CLIP framework is a domain-adaptive vision-language model that reformulates cattle behaviour recognition as crossmodal semantic alignment rather than purely visual classification, achieving 96.1% overall accuracy across six behaviours in supervised settings

Motivation and Problem Context

Cattle behaviour is a crucial indicator of animal health, productivity and overall well-being. Robust behaviour recognition in real-world farm environments remains challenging due to several data-related limitations, including the scarcity of well-annotated livestock video datasets and the substantial domain gap between large-scale pre-training corpora and agricultural surveillance footage. Existing approaches can be broadly classified into three categories based on the type of data source: contact-based, non-contact and hybrid methods. Contact-based methods use wearable sensors such as inertial measurement units (IMUs) to collect behavioural data for subsequent analysis. Fully non-contact methods utilising cameras to collect data have emerged as stress-free (Hlimi et al., 2024) and cost-effective technologies, which have been widely applied for animal behaviour recognition. However, the reliance on handcrafted features and traditional classifiers would restrict adaptability to diverse scenes or large-scale datasets.

Cattle-CLIP Architecture

Cattle-CLIP is a domainadaptive vision-language framework that reformulates cattle behaviour recognition as crossmodal semantic alignment rather than purely visual classification. Instead of directly fine-tuning visual backbones, Cattle-CLIP incorporates a temporal integration module to extend image-level contrastive pre-training to video-based behaviour understanding, enabling consistent semantic alignment across time. The preliminary mechanism involves two parallel Transformers denoted by fimg and ftext to generate image and text representations, respectively. For extending CLIP for video understanding, the framework incorporates temporal integration over frame-level features to produce a unified video representation via average pooling of frame features.

Customized Augmentation and Prompts

The methodology introduces customized augmentation strategies to improve model robustness to spatial deformation and appearance variations in real farm environments. Specifically, the authors opted to remove the crop operation because random crops result in the loss of head information, leading to an incomplete appearance. To address varying aspect ratios, they removed cropping and instead filled frames to maintain the native aspect ratio before resizing them to the target resolution. Furthermore, they introduced tailored behaviour prompts to better preserve the semantic completeness of cattle behaviour descriptions and improve cross-modal semantic alignment between vision and language representations. They modified the text by replacing ruminating with a more interpretable phrase like chewing to facilitate better semantic capture by the model.

Evaluation Scenarios and Results

The model is evaluated under both fully-supervised and few-shot learning scenarios, with a particular focus on data-scarce behaviour recognition. In the fully-supervised setting, Cattle-CLIP achieves 96.1% overall accuracy across six behaviours in supervised settings, with near-perfect recall for feeding, drinking and standing-ruminating behaviours. When compared to state-of-the-art video-based methods like VideoSwin (Liu et al., 2022), X-CLIP (Ni et al., 2022) and EVL (Rasheed et al., 2023), Cattle-CLIP achieves the highest overall accuracy. In the few-shot scenario, a base-to-novel transfer strategy is employed to improve recognition of novel behaviours by leveraging semantic knowledge from well-characterised base behaviours. The study demonstrates that the proposed base-to-novel few-shot learning with replay not only preserves knowledge learned from the base classifier but also improves generalisation to novel behaviours.

Conclusion and Future Directions

The observed performance trends suggest that multimodal semantic supervision may offer a promising direction for improving sample efficiency in precision livestock systems. The release of the CattleBehaviours6 dataset together with its standardised ethogram provides a publicly available resource to support reproducible research and consistent evaluation in cattle behaviour recognition. Future work will explore the integration of more expressive temporal modules to enhance the model’s ability to capture spatio-temporal features, such as temporal attention mechanisms or dedicated spatio-temporal fusion layers. The authors also aim to develop a more detailed ethogram guided by machine vision performance and expand their existing dataset by collecting videos of a broader range of cattle behaviours, including social interactions. Additionally, an additional focus will be on assessing how well the approach generalises to video data collected from different farms and management systems. The research contributes to building a more robust and deployable system for automated behaviour monitoring linked to health, welfare and sustainability outcomes.

Dataset Curation

The CattleBehaviours6 dataset consists of six basic individual behaviours: feeding, drinking, standing-self-grooming, standing-ruminating, lying-self-grooming and lying-ruminating. The dataset contains 1905 video clips for a herd of 200 HolsteinFriesian cows. A standardised ethogram definition is provided to support consistent behaviour annotation and evaluation. The data collection involved recordings across 27th October 2024 and 20th April 2025 at the John Oldacre Centre, Bristol Veterinary School, UK. The video resolution ranged from 1280 × 720 to 1440 pixels, and the frame rate varied between 16 and 25 frames per second (FPS). The data annotation involved using T-Rex Label to annotate the position of cows automatically with minimal manual intervention. The resulting dataset contains 1905 clips (approximately five hours of footage), each labelled with one of the six basic behaviours. The clip durations range from three to ten seconds (at a fixed rate of 5 FPS) and were randomly divided into training, validating and testing using a 6:2:2 split.

Behaviour Definitions

The authors established definitions based on the principle that each behaviour follows a standardised description template: The focal cow, . The six behaviors are defined as follows: standing, lying, feeding, drinking, standing-self-grooming, standing-ruminating, lying-self-grooming and lying-ruminating. Standing is defined as the focal cow supports its body weight with all four limbs on the ground directly below the body. Lying is defined as the focal cow supports its body weight with the sternum in contact with the ground and the limbs can be stretched away from or tucked underneath. Feeding is defined as having its head through the feed rails and inside the feeding trough/area. Standing-ruminating involves regurgitating a bolus (ball of feed from the rumen) followed by chewing in a rhythmic motion and swallowing of the bolus, which repeats over long periods.

Experimental Settings

The experiments adopted the ViT-B/16 variant of the CLIP model for their evaluation. For video sampling, they randomly selected K = 8 frames from each video clip, a commonly used setting in video action recognition benchmarks. The pre-processed frames were resized to a fixed spatial resolution of H × W = 224 × 224.

Improvements for AI systems

Multimodal Pre-training Extension: Extend image-level contrastive pre-training to video understanding by incorporating a temporal integration module to enable consistent semantic alignment across time. This allows the model to capture temporally consistent behavioural patterns, which is crucial for distinguishing between subtly different states like lying-self-grooming and lying-ruminating.
Domain Adaptation via Tailored Prompts: Introduce tailored behaviour prompts to better preserve the semantic completeness of cattle behaviour descriptions and improve cross-modal semantic alignment between vision and language representations, specifically addressing tokenization issues where words like ruminating are split into subword tokens like ru, min and ating.
Robust Data Augmentation Strategy: Remove random crop augmentation to avoid unintentionally truncating these informative regions, instead filling frames to maintain the native aspect ratio before resizing. This preserves critical features such as the head and mouth movement, which are essential for accurately capturing cows' behaviour-related features, especially given the dataset's wide range of aspect ratios.
Few-Shot Learning with Replay: Implement a base-to-novel transfer strategy incorporating a replay mechanism to mitigate catastrophic forgetting during transfer learning. This allows the model to adapt to novel behaviours while retaining recognition performance on base behaviours through training on a new training dataset Dr = Dr U sample(Db).
Enhanced Temporal Modeling for Fine-Grained Behavior: Explore integrating more expressive temporal modules, such as temporal attention mechanisms or dedicated spatio-temporal fusion layers, to enhance the model's ability to capture spatio-temporal features and improve performance in distinguishing between fine-grained behaviours like lying-self-grooming and lying-ruminating.

Abstract

Monitoring cattle behaviour provides valuable insights into animal health, productivity and welfare. However, robust behaviour recognition in real-world farm environments remains challenging because well-annotated livestock video datasets are scarce and agricultural surveillance footage differs substantially from large-scale pretraining data. To address these challenges, we propose Cattle-CLIP, a domain-adaptive vision-language framework based on Contrastive Language-Image Pretraining (CLIP), which reformulates cattle behaviour recognition as a cross-modal semantic alignment task. Cattle-CLIP incorporates a temporal integration module that extends image-level contrastive pretraining to video-based behaviour understanding and promotes consistent semantic alignment across time. To reduce domain discrepancy, customised data augmentation and behaviour-oriented prompts are employed. We also construct CattleBehaviours6, a curated dataset containing 1,905 annotated clips from six indoor behaviours: feeding, drinking, standing-self-grooming, standing-ruminating, lying-self-grooming and lying-ruminating. The dataset provides standardised ethogram definitions for model training, evaluation and future livestock behaviour research. Cattle-CLIP was evaluated under supervised and few-shot learning settings, with particular emphasis on data-constrained behaviour recognition. In supervised experiments, the model achieved 96.1% accuracy across six behaviours and showed high precision and recall for feeding, drinking and standing-ruminating. It also maintained promising few-shot performance with limited labelled data, demonstrating the potential of multimodal learning for cattle behaviour recognition in data-constrained farm environments.