We propose a multiscale multilevel Monte Carlo (MsMLMC) method to solve multiscale elliptic PDEs with random coefficients in the multi-query setting. Our method consists of offline and online stages. In the offline stage, we construct a small number of reduced basis functions within each coarse grid block, which can then be used to approximate the multiscale finite element basis functions. In the online stage, we can obtain the multiscale finite element basis very efficiently on a coarse grid by using the pre-computed multiscale basis. The MsMLMC method can be applied to multiscale RPDE starting with a relatively coarse grid, without requiring the coarsest grid to resolve the smallest-scale of the solution. We have performed complexity analysis and shown that the MsMLMC offers considerable savings in solving multiscale elliptic PDEs with random coefficients. Moreover, we provide convergence analysis of the proposed method. Numerical results are presented to demonstrate the accuracy and efficiency of the proposed method for several multiscale stochastic problems without scale separation.