Discrete Diffusion for Text Infilling

Abstract

Generative modeling of text is a fundamental challenge in natural language processing. While autoregressive models have achieved remarkable success, they face limitations in parallelizability and flexible control. Discrete diffusion models offer a promising alternative paradigm, leveraging iterative refinement and potentially enabling bidirectional context use, parallel generation, and flexible prompting. However, existing discrete text diffusion models typically assume fixed token positions, hindering their application to tasks requiring dynamic sequence lengths, such as unconstrained text infilling where ground-truth positional information is absent. vspace{baselineskip} This thesis introduces textbf{D}iscrete textbf{D}iffusion with textbf{O}ptimal textbf{T}ransport Position Coupling (DDOT) to overcome this critical limitation. DDOT is presented as the first discrete diffusion framework capable of handling flexible-length text infilling. At its core, DDOT employs a novel diffusion process that jointly models discrete token identities and continuous token positions. To maintain sequence coherence during the iterative generation process, a sample-level optimal transport (OT) coupling is integrated, ensuring consistent relative ordering of tokens. vspace{baselineskip} The methodology developed in this thesis is designed to be compatible with various underlying discrete diffusion techniques and pretrained denoising models. Comprehensive experimental validation on challenging constrained text generation benchmarks demonstrates DDOT's effectiveness. Results show that DDOT achieves performance competitive with state-of-the-art non-autoregressive methods, nears the quality of autoregressive models, and provides significant gains in training efficiency and flexibility for position-aware generation tasks. This research thus advances the capabilities of discrete diffusion models for complex text generation scenarios.