Sensor fusion annotation: aligning LiDAR and RGB at scale

odern robotics systems don't rely on a single sensor. They fuse LiDAR, RGB cameras, thermal, and IMU data into a single perception pipeline. Annotating this multi-modal data is straightforward if your sensors are perfectly calibrated; it's a nightmare if they're not.

The sensor fusion annotation problem

LiDAR gives you sparse 3D geometry. RGB gives you rich visual features — colour, texture, edges. To train a fusion model, you need labels that are consistent across modalities: a point cloud label and the corresponding image region must refer to the same object.

If your LiDAR points misalign with image pixels by 2–3 pixels, annotators will see a "car" label on the point cloud that corresponds to empty road in the RGB image. The model learns to distrust the fused representation, and performance suffers.

This alignment problem is often overlooked in annotation planning. Vendors quote a cost-per-frame without mentioning calibration, and projects discover the problem 30% through annotation.

Pre-annotation calibration: non-negotiable

Before a single annotation happens, you need:

Intrinsic calibration: Each camera and LiDAR must be internally calibrated (focal length, principal point, distortion). Use checkerboard patterns or aruco markers in 5–10 frames per sensor per environment.

Extrinsic calibration: The relative pose (translation, rotation) between LiDAR and cameras must be known to submillimetre accuracy. Use a calibration target (checkerboard or April tags) visible to both sensors. Run this in each new environment (different rooms, outdoor vs. indoor).

Temporal synchronisation: LiDAR, camera, and IMU run at different frame rates and may have clock skew. Synchronise on a known event (hardware trigger or visual pulse) and compute precise inter-sensor delays.

Skip any of these, and you'll spend 20–30% of annotation time chasing alignment issues rather than labelling objects.

Annotation workflow for fused data

Display strategy: Show the LiDAR point cloud rendered as a 3D scene, with the RGB image projected onto it (or vice versa). Annotators label in one view; the tool propagates labels to the other.

Example (LiDAR-primary workflow):

1. Render point cloud as 3D mesh, coloured by RGB projection
2. Annotator places a 3D cuboid around a car in the point cloud
3. Tool automatically converts cuboid to 2D bounding box in RGB image
4. Annotator visually verifies: does the 2D box match the car in the image?
5. If misalignment >2 pixels, flag for calibration check

Quality check: For every Nth frame (e.g., every 10th), project cuboid boundaries from LiDAR onto RGB and vice versa. Annotators verify alignment. Misalignment indicates calibration drift.

Cross-modal consistency checks

Same-object detection across views: An object visible in both LiDAR and RGB must get the same label and object ID. Common error: the same car labeled as two different IDs (one from point cloud, one from image).

Occlusion consistency: If an object is occluded in RGB but visible in LiDAR (or vice versa), annotators must handle this consistently. Define a rule: "If occluded in >80% of visible sensors, label as occluded. Otherwise label normally."

Confidence scoring: Some regions of the scene are easier to annotate in one modality than the other. A distant car is sharp in LiDAR but blurry in RGB; a reflective object is visible in RGB but appears as noise in LiDAR. Allow annotators to mark confidence per modality.

Common pitfalls and fixes

Calibration drift over time: Sensors shift slightly with temperature, vibration, or impact. Recalibrate every 100–200 frames of continuous operation. If you see a pattern of misalignment (all labels drift right by 3 pixels), stop annotation and recalibrate.

Annotation tool limitations: Many 3D annotation tools don't handle sensor fusion well. They show point clouds and images separately, requiring annotators to mentally align them. Invest in or build a tool that shows fused data natively (web-based viewer with WebGL rendering).

Object boundary ambiguity at the edge: Near the LiDAR's range limit, points become sparse and noisy. Object boundaries are hard to define. Use a rule: "If <50% of object surface is visible in point cloud, don't label it." This prevents low-confidence labels.

Inconsistent frame rates: If LiDAR runs at 10 Hz and camera at 30 Hz, which frames do you annotate? Define a strategy: "Annotate LiDAR frames only; image is auxiliary for visual confirmation." Or: "Annotate on a 10 fps resampled sequence aligned to LiDAR."

Handling partial observability

In real scenes, some objects are visible only in one sensor:

• Highly reflective surfaces (metal, glass) show up in LiDAR but not RGB, or vice versa
• Dynamic objects (moving people) may be captured by one sensor but not the other due to timing
• Shadows and glare obscure objects in one modality

Define explicit rules:

• "Label an object if visible in >50% of active sensors"
• "If visible in LiDAR only, mark as 'LiDAR-only'; if visible in RGB only, mark as 'RGB-only'"
• "If visible in both but attributes differ (size, pose), use majority vote or annotator discretion"

Cost implications and dataset size

Multi-sensor fusion annotation requires significant upfront investment in calibration and cross-modal consistency verification. This cost is justified by improved model performance, as fusion models can resolve ambiguities that single-modality models cannot. The upfront investment pays back through faster convergence and fewer model refinement cycles.

What this means for you

If you're building a multi-sensor robot, sensor fusion annotation is essential. The upfront cost is high, but you can't skip calibration — it's the foundation of everything downstream.

Plan 2–3 weeks for initial calibration, ongoing 1–2 days per 500 frames for recalibration. Use a tool that displays fused data natively. Hire annotators who understand 3D geometry and sensor physics, not generalists.

And remember: a misaligned dataset won't fail obviously. Your model will train, but it will be 20–30% worse than expected. By the time you discover the issue, you've invested months.

This is why multi-sensor annotation works best with stable, experienced teams. A specialist who has calibrated and annotated 500 fusion datasets will spot misalignment errors in seconds. New annotators take weeks to develop this intuition. Retention is your biggest lever on calibration quality and overall data quality.

Learn more about robotics data pipelines or explore our data enrichment services.