Document · 75 blocks

Training Dyna-2 at million-hour scale, repeatably — DYNA

§ 1 Introduction

Earlier this week, we announced our flagship world-action model Dyna-2, trained on more than one million hours of egocentric video data. Training a robotics foundation model at this scale is unprecedented in the robotics literature, and introduced a new set of challenges across our data and training infrastructure. Most of what worked at ten thousand hours did not hold up at a million.

Robotics training is constrained by the data in a particular way. An episode is not one stream but many. Several cameras run alongside proprioceptive and action streams, each sampled at its own rate, and a training sample has to be assembled from all of them at a consistent point in time. Robotics foundation models are also more compact due to onboard deployment constraints. As a result, training can consume data quickly enough for the dataloader itself to become a bottleneck.

Robotics discussion tends to focus on models and data more than the infrastructure underneath them. We have weighted our focus the other way from the start. For most of the last year, the number of experiments we could run was limited not by a shortage of ideas, but by how long each experiment had to wait for its data. More compute would not have fixed that on its own, whether GPUs or additional CPU workers. As the data grows, the bottleneck keeps moving: first the storage format, then the ingestion pipeline, then the training manifest. The work is finding the current bottleneck, not simply adding machines behind it.

We picked concrete examples from across the training lifecycle, along with the job resilience that spans all of it, and describe what we actually changed to make one million hours work repeatedly:

§ 2 Scaling challenges

From collection to training on the GPU cluster, robot data moves through four stages: collection into a landing bucket, ingestion into training-ready episodes, curation into the dataset for a given experiment, and loading into training batches. The high-level paradigm is common across industries. But what differs is that existing infrastructure falls short on the specific requirements of large-scale robotics training, and each stage breaks in its own way. We take them in that order, and finish on the GPU cluster itself, where the same kind of scale-dependent bottleneck shows up in training.

Episode container: MCAP and topic-group chunking

For Dyna-2, which predicts both video and actions, each training sample needs a few decoded video frames but a much longer sequence of proprioceptive state, sized to the action chunk. That asymmetry isn't unique to Dyna-2: data for robotics is inherently multi-modal, and any data collection method usually carries a suite of sensors, all producing data continuously at different rates and structures, so the read pattern itself becomes a training hyperparameter that differs across modalities.

These I/O patterns force a direct tradeoff in the storage format. Independent, per-frame access makes an arbitrary-offset read trivial, but costs far more to store and stream, on the order of a hundred MB per camera-minute for per-frame JPEG, depending on resolution and frame rate. Inter-frame video compression (H.264 with GOPs) shrinks that dramatically, but a compressed frame is only decodable starting from its GOP's keyframe, which is at odds with random access to an arbitrary timestamp. Scaling up meant we needed a format that was both efficient to store and easy to inspect. Our earlier storage, H5 holding per-frame JPEG, fell short on both counts:

We moved fully to MCAP, which is widely used in autonomous driving and stands out for its random access and flexible chunking. However, MCAP does not meet our training requirements out of the box, largely because its default chunking pattern is not optimized for video-action training. We heavily tuned its compression, video encoding, and chunking specific to our access pattern. Two choices mattered most:

Figure 2. Default MCAP writes each topic's whole window in turn, so one training sample costs a read per topic. We instead group topics that share a read pattern — cameras with cameras, state with state — and write each group time-major. A sample is then two reads, one per group, however many topics there are. The figure uses four topics at two different rates for clarity. A real episode carries more of both, so the measured saving below is larger.

On real teleop episodes, the compression cut the storage size by roughly 68% against a per-frame JPEG baseline. Topic chunking further reduced I/O round trips, roughly 3.4× fewer chunk fetches per sample, and in turn reads roughly 2.9× faster than the default chunking under the same reader:

Ingestion: DAG decomposition, staggered starts, and bin-packed batches

Our ingestion pipeline was once capped at 14,000 episode-hours per week. At that rate, a million hours would have taken over a year.

A typical ingestion pipeline to process robot data has three stages: data transformation, quality check, and feature enrichment. Data transformation resamples all camera and proprioception streams onto a common timestamp grid, computes derived signals, encodes video into a canonical MCAP, and indexes the resulting metadata for manifest queries. During quality checks specifically, we detected and filtered out quality issues such as camera blackout, choppy joint states, missing/occluded hand positions, bad frames, etc. Feature enrichment generates performance labels, video captions, segmentations, and other data used in training. These are standard practice; the hard part is running it at scale.

Our original data processing pipelines, implemented as a single Kubernetes job, had multiple issues beyond scalability. Earlier this year we rewrote our data processing pipelines with Airflow, where each step was defined explicitly in a DAG. This bought us three things:

On top of the DAG orchestrator, we also backed the pipeline states with durable storage rather than keeping them in a live process. This greatly increased the operability of the pipelines: checkpointable multi-day runs, replay from any step, cross-DAG artifact sharing, concurrency across dozens of active runs, and selective reprocessing. Additionally, the same design applies to both data collection use case where one run processes one episode, and batch use case for millions of episodes, down to the same set of sub-DAG steps. A run profile decides which of those steps actually run (full processing, a metadata-only backfill, a re-label pass), so neither a new trigger type nor a partial rerun needs a forked pipeline.Every processed episode also carries a stamp of what produced it: the schema version, the ingestion pipeline version, and the software version running on the robot when it was recorded. Reprocessing a million hours takes weeks, so any pipeline change leaves the corpus mixed-version for a while, sometimes permanently. The stamp is what makes that survivable. We can find exactly which episodes are stale and reprocess only those, and a curation query can ask for whatever version range an experiment needs.

However, this design does not necessarily solve the scalability issues since throughput does not improve linearly by simply throwing more compute to the pool. There are two problems at scale. First, when millions of runs happen concurrently, the scheduler becomes the choke point, because bursts of writes flood the scheduler database and stall the whole process. Second, data files vary widely in size, which creates imbalanced workloads and leaves resources underused.

The fix addresses each wall directly:

Together with the storage-level optimizations, these changes fully unlocked horizontal scaling and increased throughput from 14,000 to 440,000 episode-hours per week, a 31× improvement. That takes one million hours from about 16 months of processing down to under three weeks.

Curation: warehouse queries and memory-mapped tables

At a million hours, building the training manifest took about 48 hours before a training run could start.

Every training run begins by building a training manifest: exactly which episodes are in this run, and where each one starts and stops. Batching, sharding across GPUs, and epoch length all depend on it. And because each experiment filters the corpus differently- by task, by robot, by whether the run succeeded, by whether a camera dropped out- the manifest gets rebuilt every time.

Initially we built it from the files themselves. The metadata DB gave us a list of candidate paths, but a path alone is not a manifest. We still had to confirm each file was really there, read its sidecar for the quality flags, open its header for the time range, and then open it once more to count the steps it held. That is four trips to storage per episode, and our million-hour dataset is 43 million episodes. A single pre-training run can absorb that cost once. Paying it again for every experiment, however, becomes a significant bottleneck to iteration speed.

Our metadata DB does store this information. But it is also on the critical path of our data collection and annotation operations, and transactional databases are just not optimized to support large-scale columnar scans. So rather than push one database to be good at two opposed jobs, we split them by workload and keep them in sync with near-real-time change-data-capture:

The warehouse trails the production DB by seconds, which we can afford, because a curation selects over episodes that finished processing well before anyone trains on them.A curation is now a SQL query. It writes the manifest as a columnar file. One rank downloads that file once, and every rank then memory-maps the local copy at the start of training, the downloading rank included. Building it became one query instead of tens of millions of lookups, and cold startup dropped from about 48 hours to under a minute. What changed is not just the constant. The query plans over a table instead of walking a file list, so its cost no longer tracks the number of episodes the dataset holds, and a curation over the full table, now past 50 million rows, comes back in a few seconds. (That is building the manifest. Loading it is a separate cost, and what Figure 7 measures.)

That fixed how the manifest is built. Loading it was a separate problem: it was fine at 100,000 hours, but at one million it broke down in two distinct ways:

We addressed both with three changes that only work as a set:

Together, these changes took the manifest off the startup critical path: building the manifest dropped from days of filesystem crawling to a single query, and loading it went from minutes to seconds of memory mapping.

Delivery: cluster-local cache on node NVMe

Our training data lives in cloud object storage. But our training clusters do not necessarily live next to it. Since the LLM boom, GPU capacity has been scarce enough that we take it wherever we can get it, which usually means several vendors at once.

Copying the corpus to each cluster is not a real option at that point. Our million hours of training data is PB scale, where even a simple copy becomes a serious undertaking:

So the corpus stays in one place and the compute moves around it. But reading straight from cloud storage leaves the GPUs exposed to egress latency and packet losses. A short job can shrug off the occasional stall, whereas a run that has to hold throughput for weeks cannot afford any of them. The fix was to stop reading from the cloud during training and keep the working set on the cluster itself.

On-cluster data orchestration

The idea is to use the NVMe already sitting in the GPU nodes. A GPU node usually ships with plenty of it, and the cost is already in the price of the node, so the capacity is there whether we use it or not. We have used Alluxio's solution as our on-cluster caching layer since last year, for three reasons:

Building on top of their solution, we developed an on-cluster data orchestration service. Before training launches, the service resolves its manifest and warms the exact working set into cluster local storages. Training launches after the storage is warmed, and during training, cached data is served close to compute, while cache misses fall back to cloud storage.

Remote read speed depends mostly on how far the cluster sits from the bucket, and a single reader against object storage sustains roughly 200 MB/s. The bucket itself is not the bottleneck. Its aggregate bandwidth is enormous, but one connection pays a round trip per request, and a rank reading one file at a time is exactly one connection. The cache serves roughly 2 GB/s per node regardless, about ten times faster, and the NVMe underneath is quicker still, so the ceiling is the read path rather than the disk. That predictability mattered as much as the speed. Loading stayed uniform across clusters, held for weeks, and hid the mount differences, so training configurations stay cluster-independent. More on our multi-cloud architecture in a future report.

Training: topology-aware optimizer sharding

Feeding the GPUs is one problem. What runs on them is another, and an optimization tuned at one scale does not necessarily survive the next. We made many changes on that side of the system, and rather than walk through all of them, here is the one that shows the pattern most clearly. Muon, our optimizer, was taking roughly half the wall-clock step time. So we split its state across every rank in the job, with each rank updating every Nth parameter and the results all-reduced afterwards. On a handful of nodes that worked very well.

Connecting many nodes together started to slow this down. Intra-node GPU to GPU communication is blazing-fast with NVLink, roughly 1.8TB/s per GPU on our B200 nodes. But separate nodes are connected via InfiniBand, which is more than an order of magnitude slower per GPU. We found that scaling up the number of nodes dramatically increases the slower inter-node comms. Taking inspiration from FSDP Hybrid sharding:

Once a job is big enough for the switch to kick in, the optimizer step runs about 3x faster on average than the fully-sharded design it replaced, and the gap grows as nodes are added, though the figure below measures a single node count. That only holds at scale, though. Small jobs have less inter-node traffic to begin with, so the original global sharding still wins there. The trainer therefore picks between the two at runtime, from the node count it sees.

Figure 11: measured optimizer step cost by sharding strategy, at the node count where the switch triggers. Fully sharded looks competitive at the median but pays 2.6x that on the mean, because it moves 7.6x the broadcast traffic and moves it across InfiniBand rather than keeping it on NVLink. In synchronous training a slow step is paid by every rank, and wall-clock is the sum of all steps, so the mean is the number to compare.

Job resilience: preflight gating and auto-restart

When a job is small, a node dying isn't really a problem. You restart it and you've lost a few minutes. That changes once a run holds the fleet for weeks. The run only moves when every rank is healthy, so each machine you add is one more thing that can stall all the others. And when the job does go down, you lose everything since the last checkpoint. You will hit bad nodes, and most never show up in an alert:

So we build assuming some part of the fleet is broken at any given time:

§ 3 Looking forward

Million-hour training is not a single scaling problem. Instead, it is an end-to-end vertical system re-design with different considerations at each stage. The lasting result goes beyond a single large training run. Ad-hoc scripts get one run out the door, then have to be written again for the next. Reusable infrastructure lets each experiment start where the last one finished, and that is what makes this scale repeatable rather than a one-off. It is why we built the foundation instead of scripting our way through: researchers can now ingest, curate, and experiment with orders of magnitude larger data without rebuilding the path each time, which accelerates our iteration cycles and widens the range of experiments worth designing.

Robotics will continue to place unusual demands on both research and infrastructure. If working on these systems sounds interesting, visit dyna.co/careers.

§ References