Physics-Informed Neural Network (PINN) represents a new approach to solve Partial Differential Equations (PDEs). PINNs aim to solve PDEs by integrating governing equations and the initial/boundary conditions (I/BCs) into a loss function. However, the imbalance of the loss function caused by parameter settings usually makes it difficult for PINNs to converge, e.g. because they fall into local optima. In other words, the presence of balanced PDE loss, initial loss and boundary loss may be critical for the convergence. In addition, existing PINNs are not able to reveal the hidden errors caused by non-convergent boundaries and conduction errors caused by the PDE near the boundaries. Overall, these problems have made PINN-based methods of limited use on practical situations. In this paper, we propose a novel physics-informed neural network, i.e. an adaptive physics-informed neural network with a two-stage training process. Our algorithm adds spatio-temporal coefficient and PDE balance parameter to the loss function, and solve PDEs using a two-stage training process: pre-training and formal training. The pre-training step ensures the convergence of boundary loss, whereas the formal training process completes the solution of PDE by balancing various loss functions. In order to verify the performance of our method, we consider the imbalanced heat conduction and Helmholtz equations often appearing in practical situations. The Klein-Gordon equation, which is widely used to compare performance, reveals that our method is able to reduce the hidden errors. Experimental results confirm that our algorithm can effectively and accurately solve models with unbalanced loss function, hidden errors and conduction errors. The codes developed in this manuscript are publicy available at https://github.com/callmedrcom/ATPINN.