The Data Layer for Physical AI: A Unified Foundation for Robotics

Overview

Rerun has evolved into a unified data layer designed specifically for physical AI and robotics. This system provides the essential data primitives required to build, understand, and improve the data loop for multi-rate, multimodal data. From the initial recording stage to massive-scale deployment, Rerun offers a comprehensive suite of tools to manage complex robotics datasets.

Benefits

Rerun enables users to visualize everything within the data pipeline. This includes reviewing datasets at a high level, debugging issues at the detail level, and extending functionality with custom views and tools tailored to every stage of the pipeline. The platform supports full dataframe or SQL queries over any robotics data, allowing for deep inspection beyond just metadata. Users can execute SQL or dataframe queries across the catalog, drilling down into columns, time ranges, and values inside recordings. They can also refine data without creating copies by adding derived columns and evolving schemas while maintaining historical integrity. Transforms are run via the SDK, keeping derived data and raw recordings organized together. Rerun handles the storage and management of multi-rate, multimodal data efficiently. Data is stored as column-chunks in .rrd files. Users can log data directly or easily convert from any other format. Rerun Hub serves as the production backend, offering cataloging, byte-range indexing, and retrieval capabilities. This turns object stores into a queryable, streamable foundation, allowing transforms to run on the edge or close to the data. Rerun streamlines the training process by eliminating the need for an export step. Users can express a dataset mix as a query and stream it directly to GPUs. The dataloader is column-aware and video-codec-aware, ensuring efficient training directly on recordings. To ensure team alignment, Rerun promotes a shared data environment. One viewer accesses the same recordings across the entire team. Teams can explore, annotate, and trace failures back to the specific data that caused them.

Use Cases

Rerun is used in various robotics and AI projects. LeRobot, Hugging Face's robotics project, utilizes Rerun as an integrated visualization tool to inspect and debug training runs. Brush, a 3D reconstruction engine from DeepMind using Gaussian splatting, is written in Rust. It is portable, flexible, fast, and leverages Rerun for training visualization. PyCuVSLAM is the official Python wrapper for NVIDIA's cuVSLAM library. It provides GPU-accelerated visual SLAM and camera tracking for real-time localization, with Rerun handling visualization. Project Aria, Meta Reality Labs Research's egocentric AI platform, uses Rerun to visualize sequences within the Aria Dataset Explorer.

Pricing

Pricing details are not available.

Vibes

Pricing details are not available.

Additional Information

Pricing details are not available.