NICE Toolbox: Nonverbal Interpersonal Communication Exploration Toolbox

The NICE Toolbox is a free, open-source Python framework designed to automatically measure nonverbal interpersonal communication signals by combining existing state-of-the-art computer vision and speech-processing methods into a single software package. The tool aims to make data exploration of these signals scalable, reproducible, and accessible for psychological research by unifying disparate models into an integrated analysis workflow.

The gist

The Nonverbal Interpersonal Communication Exploration (NICE) Toolbox is a free, open-source Python framework that automatically measures these signals by combining existing state-of-the-art computer vision and speech-processing methods into a single software package

Statement of Need

Labelling and annotation of nonverbal behaviour signals from video recordings is one of the key methods of psychological research on interpersonal communication Traditionally, this is done manually by trained student assistants or professional labellers rewatching videos multiple times The process is slow, difficult to scale, and results are hard to make consistent across annotators This also limits study dataset size, due to labelling budget and time constraints Computer vision and speech-processing models exist for each of the underlying signals (body pose, gaze, speech transcription, and others), but they are distributed as separate research libraries with different software dependencies, input/output formats, and configuration conventions Combining them into a working analysis workflow requires machine-learning and software-engineering expertise, which usually lies outside the scope of a psychology research project Furthermore, choosing among existing models to determine the most suitable one for a specific study can be difficult for most psychological researchers Data recording setups vary widely in camera placement, lighting, participant distance, appearance, interaction style, and data formats Algorithm rankings on public benchmarks often do not transfer to a specific study, so researchers must compare candidate models and hyperparameter configurations on their own data before committing to one for their study As of yet, no available algorithm framework offers such an integrated performance evaluation pipeline while allowing the combination of different models to capture multimodal data streams

State of the Field

Manual annotation tools such as ANVIL (Kipp, 2001) and ELAN (Wittenburg et al., 2006) have long been the standard for behavioural coding in psychological and linguistic research They provide advanced user interfaces for frame-accurate labelling and are well integrated with the workflows of the communities that use them, but the annotation itself remains a time-consuming and labour-intensive process Automated open-source signal measurement tools usually cover only one specific channel Motion Energy Analysis (MEA) (Ramseyer, 2020) quantifies frame-to-frame pixel change as a proxy for body movement and has been widely used to study interpersonal synchrony in psychotherapy OpenFace (Baltrusaitis et al., 2018) and Py-Feat (Cheong et al., 2023) are open-source libraries for facial analysis used in psychological research CrisperWhisper (Zusag et al., 2024) and WhisperX (Bain et al., 2023) are widely used for audio transcription Combining these tools into a multimodal analysis pipeline is done by the researcher, who must adapt output formats, coordinate systems, and sampling rates using custom post-processing scripts One effort to centralise these tools in a single hub is EnvisionBox (EnvisionBox, 2026) It is a community-driven platform that curates open code modules, tutorials, and datasets for multimodal analysis in social-interaction research Commercial platforms such as FaceReader (Lewinski et al., 2014) and iMotions (iMotions A/S, 2025) provide integrated multimodal analysis with polished user interfaces, but their proprietary licensing, closed algorithms, and costs limit reproducibility and accessibility for academic research Their coverage is also bounded by what the vendor supports: signals or models not offered by the platform cannot be added by the user The closest comparison to NICE Toolbox is DISCOVER (Hallmen et al., 2025), a server-based back-end for computationally driven behaviour analysis Users drive it through its graphical front-end, the NOVA annotation tool (Heimerl et al., 2019), to explore content, inspect recordings, and refine annotations Compared to DISCOVER and other alternatives, NICE Toolbox focuses on automated large-scale dataset processing without human intervention or refinement Each user-created configuration defines a computational pipeline in which detector outputs from different modalities can feed into one another The same pipeline supports second-level measurements built on top of these outputs (for example, kinematics from body pose, or interpersonal distance from two-person tracking) Rather than committing to a single model, NICE Toolbox provides an interchangeable collection of models with a unified output structure This allows users to evaluate algorithms and choose the one that works best for their specific task NICE Toolbox supports multi-camera recordings and fuses per-camera detections into a single 3D space through triangulation, whereas most existing tools either accept only one camera per recording or leave per-camera outputs unfused

Software Design

NICE Toolbox’s design is based on three core principles: (1) to provide a configuration-rich experiment design that lets non-programmers run a full multimodal analysis without writing code, (2) to maintain consistent data structures (components) across different detectors, and (3) to reuse established open-source models and libraries rather than reimplementing them The toolbox is organised into modules with separate responsibilities: detectors runs the computer vision and audio models on recordings, evaluation computes configurable metrics based on the user-specific dataset, visualizer provides the interactive rerun (Rerun Development Team, 2024), and connectors handles import and export to third-party tools such as ELAN and napari-deeplabcut (DeepLabCut, n.d.) for ground-truth annotation Users control the software through TOML configuration files that describe algorithm parameters, evaluation metrics, visualizer layouts (see Figure 2), and dataset structure Configurations are validated against schema definitions before execution, so errors are caught before a long-running pipeline starts Detector algorithms additionally support template inheritance, so users can define a parameter set once and derive variants that override only the fields that differ Detectors can use the output of other detectors as their input, allowing the construction of a directed computational graph The order of execution is defined by topological sorting of nodes Nodes outputting the same components are interchangeable and configurable by the user Most of the detectors run in their own isolated virtual or conda environment The main reason is that ML libraries often pin conflicting versions of shared dependencies (e.g.

1. The system can perform automated large-scale dataset processing without human intervention or refinement. This means researchers can utilize automated large-scale dataset processing to extract behavioral cues for psychological research at a scale previously unattainable due to manual annotation constraints.

2. The AI system enables the evaluation of multiple algorithms on custom data by supporting an interchangeable collection of models with a unified output structure. This allows users to evaluate algorithms and choose the one that works best for their specific task without needing separate expertise for each model's input/output format.

3. The framework supports complex, directed computational graphs where detectors can use the output of others as input, defined by topological sorting of nodes. This allows for the construction of second-level measurements built on top of these outputs (for example, kinematics from body pose).

4. The design allows non-programmers to conduct analysis via a configuration-rich experiment design that lets non-programmers run a full multimodal analysis without writing code. Users control the pipeline through TOML configuration files describing parameters, metrics, and visualizations.

Understanding human communication is a long-standing goal in psychological research. People express themselves through many verbal and nonverbal channels: posture, gesture, gaze, facial expression, and speech. Measurement and annotation of these signals has traditionally required time-consuming manual work by trained human labellers. The Nonverbal Interpersonal Communication Exploration (NICE) Toolbox is a free, open-source Python framework that automatically measures these signals by combining existing state-of-the-art computer vision and speech-processing methods into a single software package. It includes support for multi-camera systems, interchangeable detection algorithms, 2D and 3D visualization of detector outputs, evaluation on the user's custom data, and export to third-party software. With the release of this tool, we aim to make data exploration of interpersonal communication signals scalable, reproducible, and accessible for psychological research.