TY - JOUR
T1 - How Can Deep Neural Networks Fail Even with Global Optima? 
AU - Guan , Qingguang
JO - International Journal of Numerical Analysis and Modeling
VL - 5
SP - 674
EP - 696
PY - 2024
DA - 2024/10
SN - 21
DO - http://doi.org/10.4208/ijnam2024-1027
UR - https://global-sci.org/intro/article_detail/ijnam/23448.html
KW - Deep neural network, global optima, binary classification, function approximation, overfitting.
AB - <p style="text-align: justify;">Fully connected deep neural networks are successfully applied to classification and
function approximation problems. By minimizing the cost function, i.e., finding the proper weights
and biases, models can be built for accurate predictions. The ideal optimization process can
achieve global optima. However, do global optima always perform well? If not, how bad can
it be? In this work, we aim to: 1) extend the expressive power of shallow neural networks
to networks of any depth using a simple trick, 2) construct extremely overfitting deep neural
networks that, despite having global optima, still fail to perform well on classification and function
approximation problems. Different types of activation functions are considered, including ReLU,
Parametric ReLU, and Sigmoid functions. Extensive theoretical analysis has been conducted,
ranging from one-dimensional models to models of any dimensionality. Numerical results illustrate
our theoretical findings.</p>