Centre d'apprentissage

Surrogate Modeling Theory

---

This course offers supplementary material that complements the surrogate modeling and uncertainty quantification courses by providing more in-depth coverage of the theory behind the three types of surrogate models. The course begins with a discussion of deep neural network (DNN) surrogate models, then continues with Gaussian process (GP) surrogate models, and concludes with polynomial chaos expansion (PCE) surrogate models.

DNN, GP, and PCE surrogate models represent a wide range of approaches to surrogate modeling, each with its unique strengths and application areas. DNNs are particularly useful for capturing complex, nonlinear relationships in large sets of high-dimensional data, making them ideal for scenarios where traditional methods might not work. GP models, on the other hand, offer a probabilistic framework that provides not only predictions but also a measure of uncertainty, which is important for uncertainty quantification analyses. GP models are typically more suitable than DNN models for small datasets, such as when a model is computationally very expensive and the number of model evaluations needs to be kept at a minimum. PCE models are well suited for representing the influence of random variables on system responses and are particularly useful for sensitivity analysis. These three types of surrogate models are addressed in the following respective supplements:

• Parts 1-2 cover DNN surrogate models
• Parts 3-6 cover GP surrogate models
• Parts 6-10 cover PCE surrogate models

An example of a typical neural network architecture, including the input layer (left cyan nodes), hidden layers (center blue nodes), and output layer (right cyan nodes).

Additional details summarizing the contents of each part of the course are as follows:

Part 1: More on Deep Neutral Network Surrogate Models

• Neural network optimizers
• Overview of hyperparameters
• Neural network architectures
• Network parameters and evaluating complexity of neural networks

Part 2: More on Deep Neural Network Activation Functions

• Overview of activation functions
• Activation function options
• Activation functions for regression tasks

Part 3: More on Gaussian Process Surrogate Models

• Overview of and theory behind GP regression
• Comparison of GP regression to radial basis function (RBF) fitting
• Theory behind the uncertainty estimate

Part 4: More on Gaussian Processes

• Introduction to the theory behind Gaussian processes
• GP regression
• Gaussian process mean value function and variance

Part 5: More on Covariance Functions

• Introduction to the theory behind covariance functions
• Significance of covariance functions for defining GP surrogate models
• Covariance function options
• Gaussian processes in cases of higher dimensions

Part 6: More on Hyperparameters for Gaussian Processes

• Introduction to the theory of hyperparameters
• How hyperparameter optimization is handled
• Overview of software settings for hyperparameter optimization and validation
• Overview of iterations and termination criteria for uncertainty quantification analyses using a GP surrogate model

Part 7: More on PCE Surrogate Models

• Introduction to the theory behind PCE surrogate models
• Overview of PCE case with one variable
• Calculating the PCE for an analytical function
• Computing the mean and variance
• Comparison of KDE plots

Part 8: More on PCE Surrogate Models in Two Variables

• Overview of PCE case with two variables
• Calculating the PCE for an analytical function
• Computing the mean and variance
• Comparison of KDE plots

Part 9: More on Sparse PCE Methods

• Overview of PCE case with several variables
• Software settings for creating PCE surrogate models
• Enhancing accuracy of the PCE approximation

Part 10: More on PCE Surrogate Models, Sensitivity Analysis, and Sobol Indices

• How Sobol indices are used for sensitivity analysis
• How Sobol indices are computed using a PCE surrogate model
• Overview of first-order Sobol indices and total Sobol indices

Upon completing the course, you will have a comprehensive understanding of the underlying theory of surrogate modeling. You will be knowledgeable in the different type of surrogate models available, the use cases and considerations for each, and related terminology. You will also be familiarized with the various settings in the software that are used to control aspects of each type of surrogate model.