Event
DSReg aligns JEPA representations with real-world variables
1 reports · 1 sources
Researchers from UIUC, Cold Spring Harbor Laboratory, Brown University, and the Max Planck Institute propose DSReg (Dependency-Sparsity Regularization), which can be added directly to an already-trained LeJEPA so that each learned dimension corresponds to a real-world variable, without the need to reconstruct observations, decoders, or labels. The method requires the generative process to satisfy Structural Diversity, i.e., the sets of observations affected by any two variables must not be exactly the same. The researchers state that DSReg can reuse existing LeJEPA checkpoints, with results similar to joint training; the local Jacobian does not need to be stored explicitly, and a single 48GB GPU can handle 8192 dimensions. On synthetic data, the researchers report that LeJEPA's representations are consistently mixed, and after adding DSReg the recovery performance approaches the upper bound across dimensions.
Report timeline
- UIUC and others propose DSReg, making each JEPA representation dimension correspond to a real-world latent variable
Researchers from UIUC, Cold Spring Harbor Laboratory, Brown University, and the Max Planck Institute propose DSReg (Dependency-Sparsity Regularization), which can be added directly to an already-trained LeJEPA so that each learned dimension corresponds to a real-world variable, without reconstruction of observations, a decoder, or labels.
机器之心 · 微信公众号 ·