Project

Black-box training for scientific machine learning models

Project Status: Active

Project Summary: We propose to develop a scalable black-box training framework for scientific machine learning (SciML) models that are non-trainable with existing automatic differentiation-based algorithms. Our particular interest to this effort is to study how to train data-driven SciML models to learn missing physics of a complex system for advancing forward simulations.  Specifically, this effort aims at achieving the following objectives: (1) develop a novel non-local gradient with structured sampling to enable non-local exploration for escaping from local minima and to achieve sufficient accuracy in gradient estimation; (2) advance theoretical analyses for a class of non-convex training problems to help domain scientists tune the hyper-parameters of the proposed training framework; and (3) exploit high-performance computing to accelerate the time to solution for black-box training problems for which loss functions involve computationally expensive black-box simulators.  The proposed framework will be demonstrated on two distinct applications.  The first is to train machine learning-based constitutive models to predict mercury dynamics in the Spallation Neutron Source mercury target, and the second is to train heat-source models to predict time-dependent laser scan paths that yield desirable micro-structures in three-dimensional printed metal components. This effort will advance the state of the art of several machine learning areas such as reinforcement learning and variational inference.

Principal Investigator: Guannan Zhang (CSMD, ORNL)

Senior Investigators: , Jiaxin Zhang (CSMD, ORNL), Hoang Tran (CSMD, ORNL), Dan Lu (CSED, ORNL), Matthew Bement (CSED, ORNL), Yousub Lee (CSED, ORNL), Bejamin Stump (MSTD, ORNL), Sirui Bi (CSED, ORNL)

Funding Period: Sept. 2020 to Aug. 2022

DGS_photonics

In Dec. 2020, Jiaxin Zhang gave a presentation on our DGS gradient optimization method at NeurIPS Workshop on Machine Learning for Engineering Modeling, Simulation and Design

Black-box adversarial attack

In May 2021, Dr. Hoang Tran gave a presentation on "Boosting black-box adversarial attack via exploiting loss smoothness" at the ICLR 2021 Workshop on Security and Safety in Machine Learning Systems.

DGS for global optimization

In July 2021, Guannan Zhang gave a presentation on our work on DGS method at The 37th conference on uncertainty in artificial intelligence. 

Publications: 

  • Hoang Tran, and Guannan Zhang, AdaDGS: An adaptive black-box optimization method with a nonlocal directional Gaussian smoothing gradient, submitted. (https://arxiv.org/abs/2011.02009).
  • Sirui Bi, Benjamin Stump, Yousub Lee, John Coleman, Matt Bement, Guannan Zhang, Blackbox Optimization for High-fidelity Heat Conduction Model Approximation in Metal Additive Manufacturing, submitted, 2021.
  • Jiaxin. Zhang, Hoang. Tran, and Guannan. Zhang, Accelerating Reinforcement Learning with a Directional-Gaussian-Smoothing Evolution Strategy, Electronic Research Archive (doi:10.3934/era.2021075), 2021.
  • Jiaxin. Zhang, Hoang. Tran, Dan. Lu, and Guannan. Zhang, Enabling long-range exploration in minimization of multimodal functions, Proceedings of 37th Conference on Uncertainty in Artificial Intelligence (UAI), 2021. (https://arxiv.org/abs/2002.03001)
  • Hoang Tran, Dan Lu and Guannan Zhang, Boosting black-box adversarial attack via exploiting loss smoothness, Proceedings of ICLR Workshop on Security and Safety in Machine Learning Systems, 2021.
  • Jiaxin Zhang, Sirui Bi, and Guannan Zhang, A directional Gaussian smoothing optimization method for computational inverse design in nanophotonics, Materials & Design, 197 (1), pp. 109213, 2021.
  • Jiaxin Zhang, Sirui Bi, and Guannan Zhang, A nonlocal-gradient descent method for inverse design in nanophotonics, Proceedings of NeurIPS Workshop on Machine Learning for Engineering Modeling, Simulation and Design, Dec. 2020.

Activities:

  • In July 2021, Sirui Bi gave a presentation on "A hybrid blackbox optimization for efficient calibration of heat conduction models in additive manufacturing" at the U.S. National Congress on Computational Mechanics
  • In July 2021, Guannan Zhang gave a presentation on "Enabling long-range exploration in minimization of multimodal functions" at the 37th Conference on Uncertainty in Artificial Intelligence (UAI 2021). 
  • In March 2021, Guannan Zhang gave a presentation on "A nonlocal gradient for high-dimensional blackbox optimization in scientific machine learning", at the SIAM Conference on CS&E
  • In May 2021, Hoang Tran gave a presentation on "Boosting black-box adversarial attack via exploiting loss smoothness" at the ICLR 2021 Workshop on Security and Safety in Machine Learning Systems.
  • In December 2020, Sirui Bi gave a presentation on "Directional Gaussian Smoothing Optimization for Inverse Design in Nanophotonics" at The Conference on Machine Learning in Science and Engineering (MLSE 2020).
  • In December 2020, Jiaxin Zhang gave a presentation on "A nonlocal-gradient descent method for inverse design in nanophotonics" at NeurIPS 2020 Workshop on Machine Learning for Engineering Modeling, Simulation and Design. [Download our poster] [A short video presentation]

Last Updated: October 4, 2021 - 9:26 am