The least-squares ReLU neural network (LSNN) method was introduced and studied for solving linear advection-reaction equation with discontinuous solution in [4, 5]. The method is based on an equivalent least-squares formulation and [5] employs ReLU neural network (NN) functions with ⌈${\rm log}_2(d+1)$⌉$+1$-layer representations for approximating solutions. In this paper, we show theoretically that the method is also capable of accurately approximating non-constant jumps along discontinuous interfaces that are not necessarily straight lines. Theoretical results are confirmed through multiple numerical examples with $d = 2, 3$ and various non-constant jumps and interface shapes, showing that the LSNN method with ⌈${\rm log}_2 (d + 1)$⌉$+1$ layers approximates solutions accurately with degrees of freedom less than that of mesh-based methods and without the common Gibbs phenomena along discontinuous interfaces having non-constant jumps.

Singularities, distinctive features signifying abrupt changes in function behavior, hold pivotal importance across numerous scientific disciplines. Accurate detection and characterization of these singularities are essential for understanding complex systems and performing data analysis. In this manuscript, we introduce a novel approach that employs neural networks and machine learning for the automated detection and characterization of singularities based on spectral data obtained through fast Fourier transform (FFT). Our methodology uses neural networks trained on known singular functions, along with the corresponding singularity information, to efficiently identify the location and characterize the nature of singularities within FFT data from arbitrary functions. Several tests have been provided to demonstrate the performance of our approach, including singularity detection for functions with single singularities and multiple singularities.

In this paper, we propose the Legendre-Galerkin Network (LGNet), a novel machine learning-based numerical solver for parametric partial differential equations (PDEs) using spectral methods. Spectral methods leverage orthogonal function expansions, such as Fourier series and Legendre polynomials, to achieve highly accurate solutions with a reduced number of grid points. Our framework combines the advantages of spectral methods, including accuracy, efficiency, and generalization, with the capabilities of deep neural networks. By integrating deep neural networks into the spectral framework, our approach reduces computational costs that enable real-time predictions. The mathematical foundation of the LGNet solver is robust and reliable, incorporating a well-developed loss function derived from the weak formulation. This ensures precise approximation of solutions while maintaining consistency with boundary conditions. The proposed LGNet solver offers a compelling solution that harnesses the strengths of both spectral methods and deep neural networks, providing an effective tool for solving parametric PDEs.

Fully connected deep neural networks are successfully applied to classification and function approximation problems. By minimizing the cost function, i.e., finding the proper weights and biases, models can be built for accurate predictions. The ideal optimization process can achieve global optima. However, do global optima always perform well? If not, how bad can it be? In this work, we aim to: 1) extend the expressive power of shallow neural networks to networks of any depth using a simple trick, 2) construct extremely overfitting deep neural networks that, despite having global optima, still fail to perform well on classification and function approximation problems. Different types of activation functions are considered, including ReLU, Parametric ReLU, and Sigmoid functions. Extensive theoretical analysis has been conducted, ranging from one-dimensional models to models of any dimensionality. Numerical results illustrate our theoretical findings.

In this paper, we present a deep surrogate model for learning the Green’s function associated with the reaction-diffusion operator in rectangular domain. The U-Net architecture is utilized to effectively capture the mapping from source to solution of the target partial differential equations (PDEs). To enable efficient training of the model without relying on labeled data, we propose a novel loss function that draws inspiration from traditional numerical methods used for solving PDEs. Furthermore, a hard encoding mechanism is employed to ensure that the predicted Green’s function is perfectly matched with the boundary conditions. Based on the learned Green’s function from the trained deep surrogate model, a fast solver is developed to solve the corresponding PDEs with different sources and boundary conditions. Various numerical examples are also provided to demonstrate the effectiveness of the proposed model.

Reduced order modeling (ROM) has been widely used to solve parametric PDEs. However, most existing ROM methods rely on linear projections, which face efficiency challenges when dealing with complex nonlinear problems. In this paper, we propose a convolutional neural network-based ROM method to solve parametric PDEs. Our approach consists of two components: a convolutional autoencoder (CAE) that learns a low-dimensional representation of the solutions, and a convolutional neural network (CNN) that maps the model parameters to the latent representation. For time-dependent problems, we incorporate time $t$ into the surrogate model by treating it as an additional parameter. To reduce computational costs, we use a decoupled training strategy to train the CAE and latent CNN separately. The advantages of our method are that it does not require training data to be sampled at uniform time steps and can predict the solution at any time $t$ within the time domain. Extensive numerical experiments have shown that our surrogate model can accurately predict solutions for both time-independent and time-dependent problems. Comparison with traditional numerical methods further demonstrates the computational effectiveness of our surrogate solver, especially for solving nonlinear parametric PDEs.

The impressive expressive power of deep neural networks (DNNs) underlies their widespread applicability. However, while the theoretical capacity of deep architectures is high, the practical expressive power achieved through successful training often falls short. Building on the insights gained from Neural ODEs, which explore the depth of DNNs as a continuous variable, in this work, we generalize the traditional fully connected DNN through the concept of continuous width. In the Generalized Deep Neural Network (GDNN), the traditional notion of neurons in each layer is replaced by a continuous state function. Using the finite rank parameterization of the weight integral kernel, we establish that GDNN can be obtained by employing the Integral Activation Transform (IAT) as activation layers within the traditional DNN framework. The IAT maps the input vector to a function space using some basis functions, followed by nonlinear activation in the function space, and then extracts information through the integration with another collection of basis functions. A specific variant, IAT-ReLU, featuring the ReLU nonlinearity, serves as a smooth generalization of the scalar ReLU activation. Notably, IAT-ReLU exhibits a continuous activation pattern when continuous basis functions are employed, making it smooth and enhancing the trainability of the DNN. Our numerical experiments demonstrate that IAT-ReLU outperforms regular ReLU in terms of trainability and better smoothness.

This paper uses neural networks and machine learning to study the optimal choice of the interior penalty parameter of the discontinuous Galerkin finite element methods for both the elliptic problems and Biot’s systems. It is crucial to choose the optimal interior penalty parameter, which is not too small or too large for the stability, robustness, and efficiency of the approximated numerical solutions. Both linear regression and nonlinear artificial neural network methods are employed and compared using several numerical experiments to illustrate the capability of our proposed computational framework. This framework is integral to developing automated numerical simulation because it can automatically identify the optimal interior penalty parameter. Real-time feedback could also be implemented to update and improve model accuracy on the fly.