To the Test Set and Beyond: Generalizing Far From Your Training Data with Compositional Energy Based Models

To the Test Set and Beyond: Generalizing Far From Your Training Data with Compositional Energy Based Models

Abstract: A critical issue when training machine learning models in the embodied environments is a lack of comprehensive data. In my talk, I'll illustrate how by factoring the data distribution into a set of learnable subdistributions, we can learn models that generalize far from the training distribution. I'll first introduce the concept of Energy Based Models, and illustrate the algebra through which these models enable compositionality. I'll then discuss how such compositionality enables generalization far from the training dataset in both vision and robotics domains. Finally, I'll discuss how such operations can be used to construct agents with multiple-sources of commonsense knowledge, by combining different foundation models at prediction time.

Bio: Yilun Du is PhD student at MIT EECS advised by Leslie Kaelbling, Tomas Lozano-Perez, and Joshua Tenenbaum. Previously, he has done a research fellowship at OpenAI, and held visiting researcher/research internship positions at FAIR and Google Deepmind. He is supported by the NSF Graduate Fellowship, and has received outstanding paper awards at NeurIPS and ICRA workshops.