Latent-Autoregressive GP-VAE Language Model
By: Yves Ruffenach
We investigate a fully Latent AutoRegressive scheme based on a Gaussian Process (GP) integrated into a Variational Autoencoder (VAE). In this setting, sequential dynamics are transferred from the observation space to a continuous latent space, while linguistic generation remains parallel through a non-autoregressive decoder. We present a complete methodological formulation, including a causal GP prior, a structured amortized posterior, and a training protocol based on a regularized ELBO. Empirical evaluation, conducted within a deliberately constrained proof-of-concept (POC) framework, shows that the model can be trained stably and that the sequential and parallel sampling variants exhibit consistent behavior. Overall, the results suggest that part of the temporal structure in a language model can be supported by the probabilistic geometry of the latent space rather than by explicit neural operations.
Similar Papers
Physically Interpretable Representation Learning with Gaussian Mixture Variational AutoEncoder (GM-VAE)
Machine Learning (CS)
Finds hidden patterns in messy science data.
Integrating Predictive and Generative Capabilities by Latent Space Design via the DKL-VAE Model
Machine Learning (CS)
Creates new designs with desired features.
From Classical Probabilistic Latent Variable Models to Modern Generative AI: A Unified Perspective
Machine Learning (CS)
Unifies AI tools by showing how they learn.