TY - JOUR
T1 - Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
AU - John Xu , Zhi-Qin
AU - Zhang , Yaoyu
AU - Luo , Tao
AU - Xiao , Yanyang
AU - Ma , Zheng
JO - Communications in Computational Physics
VL - 5
SP - 1746
EP - 1767
PY - 2020
DA - 2020/11
SN - 28
DO - http://doi.org/10.4208/cicp.OA-2020-0085
UR - https://global-sci.org/intro/article_detail/cicp/18395.html
KW - Deep learning, training behavior, generalization, Jacobi iteration, Fourier analysis.
AB - <p style="text-align: justify;">We study the training process of Deep Neural Networks (DNNs) from the
Fourier analysis perspective. We demonstrate a very universal Frequency Principle
(F-Principle) — DNNs often fit target functions from low to high frequencies — on
high-dimensional benchmark datasets such as MNIST/CIFAR10 and deep neural networks such as VGG16. This F-Principle of DNNs is opposite to the behavior of Jacobi
method, a conventional iterative numerical scheme, which exhibits faster convergence
for higher frequencies for various scientific computing problems. With theories under an idealized setting, we illustrate that this F-Principle results from the smoothness/regularity of the commonly used activation functions. The F-Principle implies
an implicit bias that DNNs tend to fit training data by a low-frequency function. This
understanding provides an explanation of good generalization of DNNs on most real
datasets and bad generalization of DNNs on parity function or a randomized dataset.</p>