1

We study the best-achievable identification guarantees and provable identification algorithms for causal representation learning when hard interventions are not available.

We investigate the “grokking” phenomenon in deep learning on some simple setups, and show that it is caused by a dichotomy of the implicit biases between the early phase and late phase during training.

We prove that GD applied to the matrix sensing problem has intriguing properties – with small initialization and early stopping, it follows an incremental/greedy low-rank learning procedure. This form of simplicity bias allows GD to recover the ground-truth, despite over-parameterization and non-convexity.

We consider the problem of learning a linear operator between Sobolev RKHSs from noisy data. Different from its finite-dimensional counterpart where regularized least squares is optimal, we prove that estimators with a certain multilevel structure is necessary (and sufficient) to achieve optimality.

We provide theoretical evidence that the hardness of robust generalization may stem from the expressive power of deep neural networks. Even when standard generalization is easy, robust generalization provably requires the size of DNNs to be exponentially large.

We provide an improved analysis of the convergence rates of clipping algorithms, theoretically justifying their superior performance in deep learning.

We proposed the first non-asymptotic analysis of algorithms for DRO with non-convex losses. Our algorithm incorporates momentum and adaptive step size, and has superior empirical performance.

We provide an improved analysis of the convergence rates of clipping algorithms, theoretically justifying their superior performance in deep learning.