Abstract: Artificial neural networks are a highly successful tool in machine learning and can be mathematically analyzed through the lens of approximation theory. However, many of the constructive approximation techniques used for their analysis do not correspond to how training is carried out in practice: typically, by gradient-based minimization of a nonconvex loss function. In this talk, we aim to connect classical approximation techniques, which allow for precise error guarantees, with optimization-based training procedures.
We focus on the task of constructing and training a shallow neural network of small size. To ensure that a desired approximation tolerance is achieved we rely on a convex reformulation of the loss term. To address the competing goal of limiting the size of the network, we add a sparsity promoting penalization term for the weights. A mathematical analysis reveals that a nonconvex penalization does a better job of limiting the size of the optimal network without compromising accuracy. Finally, we describe how to combine greedy node insertion, gradient based training, and node deletion to adaptively construct such optimal networks in practice and give numerical examples.
Speaker’s Bio: Konstantin Pieper is a staff member in the Computational and Applied Mathematics group, which he joined in June 2019. Dr. Pieper obtained his Ph.D. in Mathematics in 2015 from the Technical University of Munich. Prior to joining ORNL, he worked as a Postdoctoral Researcher at the Florida State University. In his research, he is motivated to incorporate new optimization and discretization methods in the context of applications involving machine learning, optimal design of experiments, and climate modeling. Currently, he is applying infinite dimensional optimization techniques to enhance our understanding of neural networks and improve their training procedures.
Host: Eirik Endeve, endevee@ornl.gov
About the Seminar: The Computational and Applied Math Seminar features talks by invited speakers, local mathematicians, and domain scientists working on problems of mathematical interest. The seminar is held weekly, every Thursday from 3:00pm-4:00pm. If you are interested in giving a seminar, please contact Eirik Endeve, endevee@ornl.gov. To subscribe to the CAM Seminar mailing list, please contact Kasi Arnold, arnoldkl@ornl.gov. To see the full list of previous and upcoming seminars, go to https://csmd.ornl.gov/events/9/seminars.
Last Updated: May 28, 2020 - 4:01 pm