Masked diffusion language models

The study of the fundamental properties of masked diffusion language models — a new class of models in which text is generated not sequentially from left to right, but through the iterative restoration of several parts of a sequence at once. We investigate how such a generation mechanism affects the dependencies between tokens, the structure of internal representations and reasoning, as well as the choice of tokenization, training objectives, and decoding algorithms. The goal of this line of research is to understand the limitations of existing methods and to develop principles for building diffusion language models that genuinely leverage the advantages of parallel generation.

Scroll to Top