Pressure dislocation might be observed when the modified ghost fluid method (MGFM) is applied to simulate radially symmetric multi-medium fluid flows for a long time. We disclose the insightful reason that the MGFM cannot satisfy balanced boundary conditions when applied to simulate the radially symmetric compressible multi-medium flows, resulting in a first-order temporal error in the interface region. To impose these balanced boundary conditions, we develop an improved MGFM in this work. The reconstruction for the initial value of the multi-medium Riemann problem at the interface is specially designed in combination with these balanced boundary conditions. In addition, the predicted instantaneous interface states and balanced boundary conditions are then utilized to define the ghost fluid states. Theoretical analysis shows that the improved MGFM can satisfy these balanced boundary conditions and effectively eliminate the first-order temporal error at the interface. Its extension to two dimensions is also presented. Numerical results show that the proposed improved MGFM can restraint pressure dislocation and overheating at the material interface very well and effectively improve mass conservation.