Memorization and generalization in generative diffusion under the manifold hypothesis
Beatrice Achilli, Luca Ambrogioni, Carlo Lucibello, Marc Mézard, Enrico Ventura · Journal of Statistical Mechanics Theory and Experiment · 2025
Abstract We study the memorization and generalization capabilities of a diffusion model (DM) in the case of structured data defined on a latent manifold. We specifically consider a set of P data points in N dimensions lying on a latent subspace of dimension D = α D N , according to the hidden manifold model. Our analysis considers a reverse process given by the empirical score function as a proxy of the true one, and then precisely characterizes the process in the high-dimensional limit in which P = exp ( α N ) and N is large, by exploiting a connection with the random energy model (REM). We provide evidence for the existence of an onset time, t o , when traps appear in the time-varying potential, although they do not affect typical trajectories. The size of the basins of attraction of such traps is computed at any time. Moreover, we derive the collapse time, t c < t o , at which trajectories fall in the basin of one of the training points, implying memorization. An explicit formula for t c is given as a function of P and the ratio α D , proving that the curse of dimensionality issue does not hold for highly structured data, i.e. α D ≪ 1 , regardless of the non-linearity of the manifold surface. We also prove that collapse coincides with the condensation transition in the REM. Finally, the degree of generalization of DMs is formulated in terms of the Kullback–Leibler divergence between the exact distribution and the one obtained at time t of the reverse process. We show the existence of an additional time t g < t c < t o such that the distance between the reverse distribution and the ground-truth is minimal. Counter-intuitively, the best generalization performance is found within the memorization phase of the model. We conclude that the generalization performance of DMs benefit from highly structured data since t g approaches zero faster than t c when α