Rotation-Invariant Multi-IMU Activity Recognition under Independent Per-Location Orientation Shifts

arXiv:2608.15621 · cs.AI, cs.LG · Submitted 2026-08-16 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Rotation-Invariant Multi-IMU Activity Recognition under Independent Per-Location Orientation Shifts".

Jane: The gist: TRI-HAR presents a rotation-invariant framework that makes robustness to independent per-location IMU orientation offsets a structural model property for multi-IMU Human Activity Recognition.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper, "Rotation-Invariant Multi-IMU Activity Recognition under Independent Per-Location Orientation Shifts." The main idea here is tackling a problem where you use multiple inertial measurement units, like in some home rehabilitation or exercise monitoring systems.

Jane: Right, so you have these sensors attached to a person—maybe on the wrist and on the hip—and when you take that data at different times, those sensors can end up with independent orientation offsets from each other because they are reattached differently across sessions.

Lu: Exactly. Conventional scalar models for human activity recognition just don't have a structural way to handle those independent orientation shifts between locations <ref:2608.15621#pg2>. Existing fixes usually involve either augmenting the data with different rotations or using calibration pipelines, but those methods introduce extra assumptions about how the sensors are oriented or where they are placed twelve thirteen thirty-three <ref:2608.15621#pg1>.

Meng: That sounds like a lot of extra work for deployment. If I'm shipping this to an end-user device, I don't want to force them through a calibration pipeline every time they take data <ref:2608.15621#pg2>. We need something that works out of the box.

Lalam: The paper proposes TRI-HAR, which is this rotation-invariant framework where it makes robustness to those independent per-location orientation offsets a structural model property <ref:2608.15621#pg3>. Essentially, it encodes rotational symmetry directly into the architecture so that the features co-rotate predictably with the input nine twenty-five <ref:2608.15621#pg1>.

Tom: So what's TRI-HAR actually doing mechanically? I need to know what they are feeding into that equivariant backbone.

Jane: Well, they treat those raw accelerometer and gyroscope measurements as triaxial vector streams, which is a key step because it moves away from just using single scalar values <ref:2608.15621#pg2>.

Lu: They reshape the scalar channels into these triaxial vectors, which they call X in R T × S×three and then they group all those streams by physical IMU location, so you get groups like X (l) in R T × Sloc×three for each location l <ref:2608.15621#pg3>.

Tom: And then what's the backbone doing with these groups? Is it just running a standard neural network on them?

Paper summary: Jane: Not exactly; they use a shared SO(three)-equivariant backbone, and they apply this to each IMU location group separately <ref:2608.15621#pg3>.

Lu: They use FERVN-DGCNN, which combines Vector Neuron operations with Frequency-based Equivariant Feature Representation, and this backbone maps each location group X (l) to a sequence of equivariant latent features V (l) in R C×N <ref:2608.15621#pg3>.

Meng: Shared backbone across all locations is good for parameter efficiency, but what about the feature extraction itself? How do we get from those equivariant features to something usable without re-calibrating for every new orientation?

Jane: That's where the next step comes in; they convert those equivariant features into invariant scalar representations using a Vector Neuron invariant layer <ref:2608.15621#pg3>. For each location l, they aggregate the time-step features with global max and average pooling to get a single location-level invariant representation h l <ref:2608.15621#pg3>.

Tom: So they're essentially forcing the features from different sensors to agree on what activity is happening locally, before fusing them together?

Jane: Precisely. Then they concatenate all those location features, h1 through hL, and feed that into a final multi-layer perceptron classifier, yˆ = MLPomega (

h1, h2,..., hL: ) <ref:2608.15621#pg3>.

Lu: The core idea is that the ordered tuple in Eq. (one) defines the fixed location order used for fusion, so they maintain invariance to independent per-location orientation shifts <ref:2608.15621#pg3>. This means even if one sensor is rotated independently of another, the final classification stays stable because of this structure nine twenty-five <ref:2608.15621#pg1>.

Tom: So the claim is that they preserve macro-F1 under fixed independent per-location SO(three) rotations and outperform rotation-augmented baselines under that specific shift without needing any augmentation <ref:2608.15621#pg3>. That's a pretty strong claim for a structural property rather than just an augmentation trick.

Jane: It suggests that the way the model is built handles those shifts inherently, which is much better than relying on some external calibration pipeline <ref:2608.15621#pg2>. The experiments show that in benchmarks like the RealDISP Mutual-displacement fold, TRI-HAR achieves a macro-F1 of seven point seven four over the strongest non-augmented baseline and six point eight zero over rotation-augmented ones <ref:2608.15621#pg3>.

Paper summary: Meng: That is a solid result if it holds up when we move it to actual deployment, but I have to ask about the cost. How much overhead is this structural encoding adding?

Lu: The main runtime tradeoff is that you execute the equivariant backbone and invariant projection once per physical IMU location <ref:2608.15621#pg3>.

Tom: So it's not running a massive network for every single orientation possibility, but it's still more complex than a simple scalar model. But the latency numbers are important here too Meng Yeah, they say that the CPU window-to-label latency stays far below the window-update period for both three-IMU and five-IMU streams <ref:2608.15621#pg3>. That means it can keep up with live data from streaming wearable IMUs.

Jane: It seems like they are targeting calibration-light, self-administered multi-IMU HAR where you treat orientation as just a nuisance factor rather than something you need to explicitly correct <ref:2608.15621#pg3>. So, the big implication is that we might be able to build these systems more reliably without relying on perfect reference frames.

Tom: The authors also flag that this framework involves an independent per-location SO(three) shift protocol, meaning each IMU location can receive a distinct fixed orientation offset <ref:2608.15621#pg3>. That's the practical way they model the problem for real-world reattachment scenarios.

Lu: The paper makes it clear that this approach is a structural one, which means robustness to those independent orientation offsets is baked into the model design, not just something you patch on later <ref:2608.15621#pg2>. This is a different kind of solution than just using rotation augmentation where robustness depends on the specific sampled transformations one <ref:2608.15621#pg1>.

Meng: So for someone who only listens to this, it means we don't have to worry as much about how exactly the user physically reattaches their sensors; the model handles that variety structurally <ref:2608.15621#pg3>.

Jane: It points toward a future where multi-sensor systems can be much more flexible for consumer use, provided we can build these equivariant representations efficiently <ref:2608.15621#pg3>. We need to keep an eye on how this scales with more complex sensor setups.

Conclusion: Tom: So we've been looking at this paper, "Rotation-Invariant Multi-IMU Activity Recognition under Independent Per-Location Orientation Shifts." Basically, they built a framework that makes activity recognition robust to those weird orientation shifts you get when you use multiple sensors over time.

Jane: Right, so instead of needing extra calibration or data augmentation to fix the sensor misalignment, this new method builds the rotational invariance right into the model structure itself. It treats those independent location offsets as something it can handle structurally.

Lu: Exactly, they reshape those raw sensor streams into triaxial vectors and use a shared backbone that respects SO(three) equivariance across all locations. The goal is to get invariant features for each sensor spot before fusing them together for the final activity classification.

Meng: I’m interested in the practical side here. They claim this works without needing extra test-time orientation calibration, which is huge if we are shipping this to end users who don't have reference frames handy. What’s the actual cost of running this framework?

Tom: Well, they say it has a runtime tradeoff because you run that equivariant backbone and the invariant projection once for every single physical IMU location. But they did show that even with that extra step, the speed is still good enough for live data from three or five sensors.

Jane: It’s about moving away from treating orientation as a nuisance factor to something we have to perfectly correct first. This suggests that for self-administered wearable monitoring, we can build systems that are much more flexible without needing perfect sensor alignment knowledge.

Lu: The main contribution is formulating an independent per-location SO(three) shift protocol, which means each physical IMU location can have its own different fixed orientation offset modeled separately. They show this works across four public benchmarks, preserving macro-F1 under those shifts and actually beating rotation-augmented baselines without needing that augmentation.

Meng: Beating the augmented ones is a strong claim. I wonder if that performance holds up when we move from controlled benchmarks to messy real-world data where sensor reattachment is totally random?

Tom: That's the next big question, Meng. But what they're pointing toward is a new way to design multi-sensor systems that are inherently robust to those deployment shifts. It’s about building hardware and software that can handle real-world variability better than before.

Jane: Yeah, this paper sets a new direction for how we think about sensor fusion in wearable AI. And it makes me wonder what kind of more complex, multi-sensor setups we could eventually model this way.

Korea University · Massachusetts Institute of Technology

cs.AI, cs.LG

Submitted: 2026-08-16

Updated: 2026-08-16

Comments: 6 pages, 2 figures. ACM International Symposium on Wearable Computing (ISWC) 2026

Journal ref: Proceedings of the 2026 ACM International Symposium on Wearable Computers (ISWC '26), pp. 1-7, 2026

DOI: 10.1145/3830727.3834824

Code: https://github.com/mbaek01/TRI-HAR

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 88/100

The gist: The gist: TRI-HAR presents a rotation-invariant framework that makes robustness to independent per-location IMU orientation offsets a structural model property for multi-IMU Human Activity

Key concepts

Independent Per-Location Orientation Offsets: This occurs when multiple IMUs are attached to different body locations; these locations can have independent rotational misalignment relative to each other. Conventional models fail because they assume a single, consistent orientation for all sensors, which is not true in multi-IMU deployments.
SO(3)-Equivariant Backbone: This is a neural network structure designed to respect the mathematical properties of 3D rotations (SO(3)). It ensures that if the input data rotates by a certain amount, the intermediate features produced by this backbone also rotate predictably, allowing for rotation-invariant representations.
Rotation-Invariant Location Features: The framework converts the equivariant features into fixed, invariant scalar values for each IMU location. This is done by aggregating time steps using pooling methods. By making these location features invariant to orientation changes, the final classification becomes robust against the unknown rotational shifts at each sensor.
Triaxial Vector Representation: Instead of treating accelerometer and gyroscope data as simple scalar streams, TRI-HAR reshapes them into 3D vectors (triaxial streams). This allows the model to capture the full spatial orientation information of the movement at each sensor location, which is crucial for modeling rotational shifts accurately.

Summary

The gist: TRI-HAR presents a rotation-invariant framework that makes robustness to independent per-location IMU orientation offsets a structural model property for multi-IMU Human Activity Recognition.

Problem and Motivation

Human Activity Recognition (HAR) often requires reattaching inertial measurement units (IMUs) across sessions, which can induce independent orientation offsets across body locations, a deployment shift that conventional scalar HAR models do not structurally handle Existing approaches to orientation variability typically rely on either data augmentation or calibration and orientation-normalization pipelines Rotation augmentation exposes a model to synthetic variants of the training data, but its robustness is tied to the sampled transformations and does not guarantee invariance to arbitrary unseen rotations Calibration-based pipelines can reduce orientation mismatch, but they add preprocessing overhead and introduce additional assumptions about available reference frames or explicit procedures A more structural approach is to encode rotational symmetry directly into the model, where SO(3)-equivariant architectures produce intermediate features that co-rotate predictably with the input, enabling rotation-invariant representations to be constructed by design

TRI-HAR Framework

TRI-HAR treats accelerometer and gyroscope measurements as triaxial vector streams, groups them by physical IMU location, and maps each group through a shared SO(3)-equivariant backbone followed by invariant projection The framework reshapes scalar channels into triaxial vector streams, denoted as X in R T × S×3 These location groups define the units over which independent orientation offsets are modeled

Architecture Components

The TRI-HAR architecture involves several key steps to achieve rotation invariance:

(3.1 Location-Grouped IMU Vector Representation)

TRI-HAR reshapes the flattened scalar channels into triaxial vector streams, X in R T × S×3 The triaxial streams are then grouped by physical IMU location: X (l) ∈ R T × Sloc×3, l = 1,..., L These location groups define the units over which independent orientation offsets are modeled

(3.2 Shared SO(3)-Equivariant Backbone)

TRI-HAR uses a single shared SO(3)-equivariant backbone, denoted as B, for all physical IMU locations This backbone is instantiated with FERVN-DGCNN [25], which combines Vector Neuron operations with Frequency-based Equivariant Feature Representation (FER) The backbone maps each location group to a sequence of equivariant latent features: V (l) 1:T = B X (l), V (l) t ∈ R C×N, l = 1,..., L

(3.3 Rotation-Invariant Location Features)

The equivariant backbone produces features that co-rotate with the input TRI-HAR converts these features into invariant scalar representations using the VN invariant layer For a physical IMU location l, the invariant time-step features are aggregated over time with global max and average pooling, denoted by Pool, to obtain a single location-level invariant representation hl The full location encoder satisfies Etheta (Rl · X (l)) = Etheta (X (l)), l = 1,..., L

(3.4 Per-Location Invariant Fusion)

TRI-HAR fuses information across IMU locations only after each location group has been mapped to an invariant feature The resulting location features h1,..., hL are concatenated in a fixed datasetspecific order and passed to the final classifier: yˆ = MLPomega ([h1, h2,..., hL]) This fusion order gives TRI-HAR invariance to independent per-location orientation offsets

Key Findings and Evaluation

TRI-HAR preserves macro-F1 under fixed independent per-location SO(3) rotations and outperforms rotation-augmented baselines under this target shift without requiring rotational augmentation In the RealDISP Mutual-displacement fold, TRI-HAR achieves the highest macro-F1, exceeding the strongest non-augmented and rotation-augmented baselines by 7.74 and 6.80 points, respectively The location-wise formulation creates TRI-HAR’s main runtime tradeoff, as it involves the equivariant backbone and invariant projection being executed once per physical IMU location Despite this cost, the CPU window-to-label latency remains far below the window-update period for both 3-IMU and 5-IMU streams TRI-HAR targets calibration-light, self-administered multi-IMU HAR where orientation should be treated as a nuisance factor

Contributions

The contributions of this work are summarized as follows:

** We propose TRI-HAR, a multi-IMU HAR framework that learns equivariant-to-invariant representations per physical IMU location, enabling structural robustness to independent orientation offsets without rotation augmentation or additional test-time orientation calibration 2. We formulate an independent per-location SO(3) shift protocol, where each IMU location can receive a distinct fixed orientation offset, matching self-administered multi-IMU reattachment settings 3. Across four public benchmarks, we show that TRI-HAR preserves macro-F1 under fixed independent per-location rotations and outperforms rotation-augmented baselines under this target orientation-shift condition The results support architecture-level rotation invariance for calibration-light, self-administered wearable HAR 4. The latency results show that the current host-side implementation can keep up with live 3-IMU and 5-IMU streams, supporting off-sensor inference from streaming wearable IMUs This work is licensed under a Creative Commons Attribution 4.0 International License The paper is published in Proceedings of the 2026 ACM International Symposium on Wearable Computers (ISWC ’26), October 11–15, 2026, Shanghai, China The ACM ISBN is 979-8-4007-2872-3/2026/10 This work is licensed under a Creative Commons Attribution 4.0 International License The ACM Reference Format lists the authors and publication details for this work The paper is published in Proceedings of the 2026 ACM International Symposium on Wearable Computers (ISWC ’26), October 11–15, 2026, Shanghai, China The ACM ISBN is 979-8-4007-2872-3/2026/10 This work is licensed under a Creative Commons Attribution 4.0 International License The paper is published in Proceedings of the 2026 ACM International Symposium on Wearable Computers (ISWC ’26), October 11–15, 2026, Shanghai, China This work is licensed under a Creative Commons Attribution 4.

Improvements for AI systems

Bold header: Structural Robustness to Independent Per-Location Offsets

This improvement enables a multi-IMU HAR system to maintain high accuracy when sensors are reattached across different body locations with independent orientation offsets, as TRI-HAR makes robustness to independent per-location IMU orientation offsets a structural model property.

Bold header: Elimination of Calibration and Augmentation Overhead

The system can operate without requiring rotational augmentation or an additional test-time orientation-calibration step, simplifying deployment for self-administered monitoring where explicit calibration is difficult.

Bold header: Location-Wise Feature Invariance

TRI-HAR achieves invariance by design, as it applies invariant projection before cross-location fusion, which cancels the location-specific rotation action before combining features, unlike joint fusion methods that only cancel shared global rotation.

Bold header: Enhanced Performance Under Realistic Displacement

The improved system is shown to exceed strongest baselines under challenging physical shifts, achieving a macro-F1 of 85.29% on the RealDISP Mutual-displacement fold, demonstrating robustness against both orientation and positional changes simultaneously.

Bold header: Efficient Host-Side Inference for Streaming Data

The architecture maintains feasibility for real-time deployment because the CPU window-to-label latency remains far below the window-update period, supporting off-sensor inference from streaming wearable IMUs.

Abstract

Human Activity Recognition (HAR) with self-administered wearables, such as at-home rehabilitation and exercise monitoring, often requires reattaching inertial measurement units (IMUs) across sessions. In multi-IMU settings, this can induce independent orientation offsets across body locations, a deployment shift that conventional scalar HAR models do not structurally handle. Existing remedies rely on rotation augmentation, whose robustness depends on sampled transformations, or calibration and orientationnormalization pipelines requiring additional reference-frame assumptions or explicit procedures. We present Truly Rotation-Invariant HAR (TRI-HAR), a rotation-invariant framework that makes robustness to independent per-location IMU orientation offsets a structural model property. TRI-HAR reshapes accelerometer and gyroscope streams into triaxial vectors, applies a shared SO(3)-equivariant backbone and invariant projection to each IMU location, and fuses the resulting invariant features for activity classification. Across four multi-IMU benchmarks, TRI-HAR preserves macro-F1 under fixed independent per-location SO(3) rotations and outperforms rotation-augmented baselines under this target shift without requiring rotational augmentation.