[Statlist] ETH/UZH Research Seminar by Weijie Su, Wharton, University of Pennsylvania, 16.12.2022

[Maurer Letizia]

We are glad to announce the following talk in the ETH/UZH Research Seminar:

"Some Geometric Patterns of Real-World Deep Neural Networks"   
Weijie Su, Wharton, University of Pennsylvania

Time: Friday,  16.12.2022 at 15.15 h
Place: ETH Zurich, HG G 19.1

Abstract: In this talk, we will investigate the emergence of geometric patterns in well-​trained deep learning models by making use of the layer-​peeled model and the law of equi-​separation. The former is a nonconvex optimization program that models the last-​layer features and weights. We use the model to shed light on the neural collapse phenomenon of Papyan, Han, and Donoho, and to predict a hitherto-​unknown phenomenon that we term minority collapse in imbalanced training. This is based on joint work with Cong Fang, Hangfeng He, and Qi Long (arXiv:2101.12699). In the second part, we study how real-​world deep neural networks process data in the interior layers. Our finding is a simple and quantitative law that governs how deep neural networks separate data according to class membership throughout all layers for classification. This law shows that each layer improves data separation at a constant geometric rate, and its emergence is observed in an authoritative collection of network architectures and datasets during training. This law offers practical guidelines for designing architectures, improving model robustness and out-​of-sample performance, as well as interpreting the predictions. This is based on joint work with Hangfeng He (arXiv:2210.17020).


Seminar website: https://math.ethz.ch/sfs/news-and-events/research-seminar.html

Research Seminar – Seminar for Statistics | ETH Zurich
math.ethz.ch


Organisers: A. Bandeira, P. L. Bühlmann, R. Furrer, L. Held, T. Hothorn, D. Kozbur, M. H. Maathuis, N. Meinshausen, S. van de Geer, M. Wolf