New Parallelism Method Enhances Long-Context Diffusion Model Training

VERA Brief

AI-generated. Grounded in the article and its cited sources.

Researchers have developed block parallelism (BP) and context-sharded block parallelism (CSBP) to enhance the training efficiency of large diffusion language models with extended contexts. These methods address limitations in distributed attention communication and memory usage, leading to improved throughput and speedups.

Key facts

• Block parallelism (BP) assigns each corrupted-block computation to a separate rank for training block diffusion language models with long contexts.
• Context-sharded block parallelism (CSBP) shards the shared clean sequence across ranks to keep corrupted K/V and gradients local and eliminate replicated clean prefixes.
• CSBP demonstrated improved throughput on H200 GPUs at 256K context, with gains ranging from 1.18x to 1.45x for supervised fine-tuning.
• CSBP accelerated DFlash2 speculative-decoder training by 2.48x at 512K and 7.59x at 1M context on H100 GPUs.
• In supervised fine-tuning of DiffusionGemma 26B-A4B, CSBP achieved higher pass rates on SWE-bench Verified and Terminal-Bench Lite within 12 hours.