One target to align them all: LiDAR, RGB and event cameras extrinsic calibration for Autonomous Driving

arXiv:2511.12291 · cs.CV · Submitted 2025-11-15 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "One target to align them all".

Jane: Here's the gist of the paper. The gist: A novel multi-modal extrinsic calibration framework is presented that simultaneously estimates relative poses between event cameras, LiDARs, and RGB cameras using a custom 3D calibration target.

Tom: In autonomous driving, integrating diverse sensing modalities like RGB cameras for semantic information, LiDARs for 3D geometry, and event cameras for high-speed or low-light conditions critically relies on accurate extrinsic calibration to enable consistent sensor fusion and scene reconstruction.

Jane: Event cameras pose significant calibration challenges due to their fundamentally different sensing mechanism and sparse, asynchronous output.

Tom: While LiDAR-RGB calibration is established, robust event camera calibration in multi-sensor contexts remains an open problem that this work addresses.

Title and authors: Tom: They’re calling it "One target to align them all: LiDAR, RGB and event cameras extrinsic calibration for Autonomous Driving." It sets the stage clearly for what they are tackling.

Jane: It immediately tells us the focus is on achieving spatial alignment across three specific sensor types in the context of autonomous driving, which is a really practical application.

Lu: The title highlights the central mechanism: using one target to align all three sensors simultaneously, which implies a unified calibration approach rather than treating each pair as an independent problem.

Meng: I wonder how much effort goes into designing this single target that needs to speak the language of LiDAR, RGB, and event cameras all at once. It sounds like a lot of engineering upfront just to get that physical setup right.

Lalam: The authors mention that this design enables "simultaneous features detection across all reference frames," meaning they are looking for features from every sensor in one go on the target itself.

Tom: Exactly. And they point out that this is a contrast to existing methods where you have to rely on separate, pairwise calibrations, which usually means running those alignments separately and then stitching the results together later.

Jane: So what’s the actual problem they are solving here? It sounds like the main hurdle is that establishing correspondences between features from different modalities is inherently non-trivial because there's no direct one-to-one mapping between them.

Lu: They acknowledge that this task is inherently challenging because the diverse sensor modalities make it difficult to establish unambiguous and reliable correspondences, especially between sparse events from an event camera and a dense LiDAR pointcloud.

The paper's summary: Tom: The paper summarizes their framework by focusing on this novel calibration target as the central component for estimating relative poses, specifically rotation matrices Rij and translation vectors tij for each sensor pair.

Jane: So, they are aiming to produce these specific transformation parameters between every sensor combination—LiDAR to event camera, LiDAR to RGB, and presumably the reverse too—all from this single setup.

Lu: They detail their pipeline as a multi-stage process involving feature detection modules for each modality: extracting keypoints from blinking LEDs on the cube for events, identifying three dee corner positions using planes on the LiDAR point cloud, and finding ArUco markers using RGB camera frames <ref:2511.12291#pg1>.

Meng: That sounds like a very involved feature extraction pipeline. They’ve got to handle frequency analysis for the event camera data, fitting ellipses in frequency maps, and then doing three dee geometric computations for the LiDAR points <ref:2511.12291#pg1>.

Lalam: They describe how they get those three dee corner positions by fitting three planes with Sequential Ransac on a cropped version of the LiDAR point cloud to find E0, and then deriving the other six corners based on known cube dimensions <ref:2511.12291#pg1>.

Tom: And for finding the ArUco markers from RGB, they are using the known measures of each cube’s face to retrieve those markers from the RGB camera frames. They also use a perspective-n-points algorithm to do it for both LiDAR and event camera calibrations.

Jane: So, in short, they are using a specific physical target to extract correspondences from all three sensors—LEDs for events, ArUco boards for RGB, and geometry for LiDAR—to compute all the necessary relative poses.

The paper's improvements: Tom: One of the main improvements they highlight is moving away from separate calibrations toward this one-shot joint calibration process enabled by their novel target design.

Jane: They emphasize that this approach provides higher accuracy and repeatability, which is crucial because target-based methods are generally preferred over target-free approaches when you need high precision in controlled environments.

Lu: They note that for LiDAR to event camera calibration specifically, the literature on extrinsic calibration between those two is quite scarce, and they are addressing that gap.

Meng: But the paper also admits a limitation regarding direct correspondence; they say there isn't a direct one-to-one mapping between features of each modality, which makes finding reliable correspondences hard.

Lalam: To address that difficulty, they describe their feature detection pipeline as an improvement because it provides "an accurate and simultaneous feature matching between the two" sensors involved in the calibration.

Tom: And they show quantitative results on this, saying that for the RGB camera and LiDAR calibration, their performance is only zero point zero three pixels off from what they consider the best method available.

Jane: That’s a concrete number to anchor—a very small reprojection error of zero point zero three pixels suggests their unified framework is performing very well in practice compared to the previous state-of-the-art methods they tested against.

Conclusion: Tom: So, to wrap up, the paper successfully addresses the challenge of event camera calibration in multi-sensor autonomous driving systems by using a novel calibration target and feature extraction pipeline.

Jane: The implication is that this framework lays a foundation for more robust and accurate extrinsic calibrations across multiple modalities, which could be useful for any complex sensor system needing unified alignment.

Lu: They also mention that the approach is extensible; they suggest it can be extended to include stereo calibration between the event camera and an RGB camera by reconstructing a temporally synchronized frame-like representation of the event stream.

Meng: From a deployment standpoint, if this works reliably on this target, it suggests we could integrate these diverse sensors more effectively into real autonomous driving frameworks without needing dozens of bespoke calibration procedures.

Lalam: The paper's main contribution is that it takes both the calibrations directly and uses a single calibration target with a unique pipeline, which is something they claim is unique among existing methods.

Tom: It really shows how designing the physical interaction between sensors, through this target, can fundamentally improve the accuracy and robustness of sensor fusion in autonomous driving.

Jane: We've seen how they tackle the complexities of aligning sparse event data with dense point clouds using this single calibration approach described in "One target to align them all: LiDAR, RGB and event cameras extrinsic calibration for Autonomous Driving."

Andrea Bertogalli, Giacomo Boracchi, Luca Magri

Politecnico di Milano

cs.CV

Submitted: 2025-11-15

Updated: 2025-11-15

Journal ref: Proceedings of the 36th British Machine Vision Conference (BMVC), 2025, pp. 1-15, Paper 499

Project page: https://andberto.github.io/One-target-to-align-them-all

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

The gist: The gist: A novel multi-modal extrinsic calibration framework is presented that simultaneously estimates relative poses between event cameras, LiDARs, and RGB cameras using a custom 3D calibration

Key concepts

Multi-modal Extrinsic Calibration
This is the process of determining the precise spatial relationship (rotation and translation) between two or more different types of sensors (like a LiDAR, an RGB camera, and an event camera). The goal is to align their coordinate systems so that data from all sensors can be accurately fused into a single, coherent view of the scene.
Custom 3D Calibration Target
A specially designed physical object used during calibration. This target is engineered to present distinct features—like planes for LiDAR, ChArUco markers for RGB cameras, and blinking LEDs for event cameras—all at once. This allows the system to gather necessary data from all three sensors simultaneously in one setup.
Event Camera Feature Extraction
This involves analyzing the unique output of an event camera stream. The method uses frequency analysis on changes in pixel brightness to detect specific patterns related to blinking LEDs on the target, allowing for precise location estimation of these features within the event data.

Terminology

Summary

The gist: A novel multi-modal extrinsic calibration framework is presented that simultaneously estimates relative poses between event cameras, LiDARs, and RGB cameras using a custom 3D calibration target.

Introduction to the Problem

In autonomous driving, integrating diverse sensing modalities like RGB cameras for semantic information, LiDARs for 3D geometry, and event cameras for high-speed or low-light conditions critically relies on accurate extrinsic calibration to enable consistent sensor fusion and scene reconstruction. Event cameras pose significant calibration challenges due to their fundamentally different sensing mechanism and sparse, asynchronous output. While LiDAR-RGB calibration is established, robust event camera calibration in multi-sensor contexts remains an open problem that this work addresses. Extrinsic calibration defines the spatial transformation that aligns the coordinate systems of different sensors, computing rotation matrices Rij and translation vectors tij for each sensor pair.

Novel Calibration Target Design

The core of the approach is a novel 3D calibration target specifically designed to be concurrently perceived by all three sensing modalities. This target encodes features in planes, ChArUco, and active LED patterns, each tailored to the unique characteristics of LiDARs, RGB cameras, and event cameras respectively. The target consists of a cube with blinking LEDs at its corners as illustrated in Figure 1. On each face we place a different 3 × 3 ChArUco board to uniquely identify the markers’ features from the RGB camera, while the 3D geometry allows for identifying the corners Ei and Aruco’s 3D positions Ai in the same reference system of the pointcloud. On the seven visible corners Ei of the three faces we installed 7 ellipsoidal LED diodes. The blinking behavior is essential to trigger events while the blinking frequency is used to uniquely identify each LED.

Feature Detection and Correspondence

The calibration framework is structured into three main keypoints extraction modules, one for each modality involved. This involves detecting blinking LEDs placed on the cube’s corners Ei to extract keypoints ei from the event stream through frequency analysis. The procedure to identify the position of the keypoints in the event stream is composed by three main stages:

  1. Frequency analysis, where pixels respond to changes in logarithmic brightness intensity L = log(I) and an event is triggered when the change exceeds a threshold C since the last event at uk.

  2. Best frequency map estimation, which involves computing n frequency maps Mi and selecting the one whose bounding box R¯i minimizes the Manhattan distance (L1-norm) with R.

  3. Ellipses fitting and location estimation, where seven ellipses are fitted inside the selected bounding box M¯i using a constrained least squares Fitzgibbon et al. [1999] on the edges Canny [1986] detected in the frequency map.

3D Corner Identification and Pose Estimation

To identify the 3D positions Ei of the cube’s corners we proceed by fitting three planes π1, π2, π3 using Sequential Ransac on a cropped version of the LiDAR pointcloud. E0 is computed as the intersection of all three planes π1, π2, and π3. The remaining six points E1 through E6 are derived through standard geometric computations since the dimensions of the cube are known by design. ArUco markers Ai are identified exploiting the known measures of each cube’s face to retrieve corresponding markers a0 through a59 from the RGB camera frames. For both calibrations, the one between LiDAR and event camera and the one between LiDAR and RGB camera, we employ the well-known Perspective-n-points algorithm Zheng et al. [2013] applied on the correspondences found.

Experimental Validation

Experiments were conducted on a custom recorded dataset featuring the calibration target in different light conditions and positions. The performance is quantitatively evaluated using the average reprojection error Emean = 1/n Σ p(xi − xˆi)2 + (yi − yˆi)2. For the more challenging calibration between the event camera and the LiDAR, our multi-modal calibration target benefits are more evident as our performances are considerably better than those obtained by similar methods. The results show that for the RGB camera and LiDAR calibration, the difference with the best method is only 0.03 pixels.

Practical Implementation

The work has been implemented to calibrate a real autonomous driving framework of a fully autonomous Maserati GrancCabrio Folgore. The sensors employed include two Ouster LiDARs, one Seyond Falcon K Ultra-Long Range sensor, up to three Lucid Vision IP68 RGB Cameras, and one EVK4 Prophesee with the IMX636 sensor. Data is exchanged with the elaboration system through the well-known ROS2 Macenski et al. [2022].

Conclusion

This work successfully addressed the challenge of event camera calibration in multi-sensor autonomous driving systems using a novel calibration target and feature extraction pipeline. The approach lays the foundation for a wide range of applications that require a robust, accurate, and flexible extrinsic calibration as a starting point for complex sensor systems. The framework is extensible to include stereo calibration between the two cameras following an approach similar to Garcia et al. [2025]. The calibration marker could be extended to include more feature points by adding more LEDs in order to increase the accuracy of the PnP algorithm. The approach enables precise and unified extrinsic calibration across multiple modalities, showing improved accuracy and robustness compared to state-of-the-art alternatives.

References

Song et al. Song et al. [2018] 4.691 -

Xing et al. Xing et al. [2023] 3.161 -

Yan et al. Yan et al. [2023] - 1.104

Grammatikopoulos et al. Grammatikopoulos et al. [2022] - 11.4826

Zhou et al. Zhou et al. [2018] - 21.69332

Our method is the only one that takes into account both the calibrations directly, a single calibration target, and a unique pipeline

Song et al. Song et al. [2018] - 4.691

Xing et al. Xing et al. [2023] - 3.161

Yan et al. Yan et al. [2023] - 1.104

Grammatikopoulos et al. Grammatikopoulos et al. [2022] - 11.4826

Zhou et al. Zhou et al. [2018] - 21.69332

Our method is the only one that takes into account both the calibrations directly, a single calibration target, and a unique pipeline

Song et al. Song et al. [2018] - 4.691

Xing et al. Xing et al. [2023] - 3.161

Yan et al. Yan et al. [2023] - 1.104

Grammatikopoulos et al. Grammatikopoulos et al. [2022] - 11.4826

Zhou et al. Zhou et al. [2018] - 21.69332

Our method is the only one that takes into account both the calibrations directly, a single calibration target, and a unique pipeline

Song et al. Song et al. [2018] - 4.691

Xing et al. Xing et al. [2023] - 3.161

Yan et al. Yan et al. [2023] - 1.104

Grammatikopoulos et al.

Improvements for AI systems

  1. Multi-sensor extrinsic calibration for autonomous driving systems can be achieved in a single data acquisition by using a custom 3D calibration target that is concurrently perceived by event cameras, LiDARs, and RGB cameras. This allows for a one-shot, joint extrinsic calibration process instead of relying on separate pairwise calibrations.

  2. The improved system can perform precise and unified extrinsic calibration across multiple modalities by leveraging the novel feature detection approach for both LiDAR and events to provide an accurate and simultaneous feature matching between the two.

  3. Autonomous driving perception frameworks can integrate event camera data with LiDAR pointclouds more robustly, overcoming the challenge where the diverse sensor modalities make it non-trivial to establish unambiguous and reliable correspondences.

  4. The system enables enhanced object detection in autonomous driving by utilizing the accurate spatial alignment of heterogeneous sensors for tasks such as object detection Sgaravatti et al. [2025], Bai et al. [2022], Yin et al. [2023].

  5. For stereo calibration between an event camera and an RGB camera, the system can reconstruct a temporally synchronized frame-like representation of the event stream to perform feature matching, leading to more accurate pose estimation for fast 6-dof object tracking.

Sources

Related papers