This paper develops a high-order accurate gas-kinetic scheme in the framework of the finite volume method for the one- and two-dimensional flow simulations, which is an extension of the third-order accurate gas-kinetic scheme [Q.B. Li, K. Xu, and S. Fu, J. Comput. Phys., 229(2010), 6715-6731] and the second-order accurate gas-kinetic scheme [K. Xu, J. Comput. Phys., 171(2001), 289-335]. It is formed by two parts: quartic polynomial reconstruction of the macroscopic variables and fourth-order accurate flux evolution. The first part reconstructs a piecewise cell-center based quartic polynomial and a cell-vertex based quartic polynomial according to the "initial" cell average approximation of macroscopic variables to recover locally the non-equilibrium and equilibrium single particle velocity distribution functions around the cell interface. It is in view of the fact that all macroscopic variables become moments of a single particle velocity distribution function in the gas-kinetic theory. The generalized moment limiter is employed there to suppress the possible numerical oscillation. In the second part, the macroscopic flux at the cell interface is evolved in fourth-order accuracy by means of the simple particle transport mechanism in the microscopic level, i.e. free transport and the Bhatnagar-Gross-Krook (BGK) collisions. In other words, the fourth-order flux evolution is based on the solution (i.e. the particle velocity distribution function) of the BGK model for the Boltzmann equation. Several 1D and 2D test problems are numerically solved by using the proposed high-order accurate gas-kinetic scheme. By comparing with the exact solutions or the numerical solutions obtained the second-order or third-order accurate gas-kinetic scheme, the computations demonstrate that our scheme is effective and accurate for simulating invisid and viscous fluid flows, and the accuracy of the high-order GKS depends on the choice of the (numerical) collision time.