Considering the individual difference, this paper deals with an infection-age structured epidemic model coupling within-host and between-host for environmentally-driven infectious disease. The full system with two time scales, the cellular level and population level, is first separated into the isolated fast and slow systems. For the isolated fast and slow systems, combined with the within-host and between-host reproduction numbers, $R_{w0}$ and $R_{b0},$ we give the complete global dynamics by using Lyapunov function respectively. Our results indicate that when there is no virus in environment the disease can be not only controlled, but also eliminated. However, when there is always virus in environment the disease is only controlled but not eliminated. Furthermore, the coupled slow system has complex dynamics with multiple positive equilibria and backward bifurcation. The virus contaminated environment plays a critical role on backward bifurcation. When the initial environmental virus is below some threshold the disease will be eliminated, when it is above the threshold the disease will develop an endemic disease. Some numerical simulations are performed to illustrate these results. The age structured model is more general, and this work includes some previous results.