Surrogate construction via weight parameterization of residual neural networks

Surrogate model development is a critical step for uncertainty quantification or other sample-intensive tasks for complex computational models. Here, in this work, we develop a multi-output surrogate form using a class of neural networks (NNs) that employ shortcut connections, namely Residual NNs (ResNets). ResNets are known to regularize the surrogate learning problem and improve the efficiency and accuracy of the resulting surrogate. Inspired by the continuous, Neural ODE analogy, we augment ResNets with weight parameterization strategy with respect to ResNet depth. Weight-parameterized ResNets regularize the NN surrogate learning problem and allow better generalization with a drastically reduced number of learnable parameters. We demonstrate that weight-parameterized ResNets are more accurate and efficient than conventional feed-forward multi-layer perceptron networks. We also compare various options for parameterization of the weights as functions of ResNet depth. We demonstrate the results on both synthetic examples and a large scale earth system model of interest.. Authors: Diaz-Ibarra, Oscar Homero [Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)]; Sargsyan, Khachik [Sandia National Lab. (SNL-CA), Livermore, CA (United States)] (ORCID:000000021037786X); Najm, Habib N. [Sandia National Lab. (SNL-CA), Livermore, CA (United States)] (ORCID:0000000263387260). DOE Contract: NA0003525. Subjects: 97 MATHEMATICS AND COMPUTING; Climate; E3SM; MLP; Neural network; ResNet; Surrogate model; Weight parameterization