NVIDIA cuts repeated weight transfers in reinforcement learning NVIDIA cuts repeated weight transfers in reinforcement learning

One model, two arrangements

Reinforcement learning repeatedly uses a model to generate attempts, scores those attempts and updates the model's weights. Training and generation can run on separate GPU groups because the two jobs have different computing needs. Each update eventually has to reach the generation machines. That transfer is called a refit. The complication is that the same weights can be divided differently across the two groups. A piece stored on one training GPU may overlap pieces needed by several inference GPUs. More devices do not remove this mismatch. The communication system still has to put the right values in the right places before the updated model can generate useful work.

Send the slice that will be used

The conventional route first gathers weight fragments into a complete parameter tensor, then broadcasts that tensor to the receiving GPUs. Each receiver keeps its required slice. Reshard instead describes both layouts and calculates where their slices overlap. In NVIDIA's example, an eight-by-eight matrix is split across two training GPUs and four inference GPUs. Each receiver takes a four-by-two section from each sender to assemble its own eight-by-two section. The abstraction comes from PyTorch distributed tensors, which describe a tensor's shape, device arrangement and placement. Applications provide those descriptions while the communication library handles the transfer plan.

The network is part of the algorithm

The implementation also accounts for replicated weights. Sending the same slice separately to every replica wastes network capacity. NCCL M2N groups receivers within an NVLink domain, divides the incoming network traffic between them and exchanges the missing pieces locally over NVLink. These transfers overlap. NeMo RL takes a mixed approach: large feed-forward projection weights use Reshard, while smaller weights keep the conventional transfer path. This matters because support for every layer and parallel layout remains a separate engineering task. NVIDIA estimates that feed-forward projection weights represent around 95 percent of parameters in the latest language models, making them the main target for reduced traffic.

Seconds saved in a specific stage

NVIDIA reports that refitting DeepSeek v3 671B on DGX H100 fell from 14.9 seconds to 2.6 seconds in its tested layout. The same model's GB200 configuration went from 6.1 seconds to 2.3 seconds. Qwen 3 235B on H100 fell from 6.4 seconds to 1.9 seconds. These comparisons isolate a recurring transfer step. They do not measure how much faster a whole training job finishes. For teams running disaggregated reinforcement learning, the practical benefit depends on how much time that step already consumes. The design changes the movement of existing weights, rather than the learning objective or the model's answers. Its open reference implementation gives framework developers a concrete alternative to repeatedly reconstructing and distributing complete tensors.