Spectral element methods are well established in the field of wave propagation, in particular because they inherit the flexibility of finite element methods and have low numerical dispersion error. The latter is experimentally acknowledged, but has been theoretically shown only in limited cases, such as Cartesian meshes. It is well known that a finite element mesh can contain distorted elements that generate numerical errors for very large distortions. In the present work, we study the effect of element distortion on the numerical dispersion error and determine the distortion range in which an accurate solution is obtained for a given error tolerance. We also discuss a double-grid calculation of the spectral element matrices that preserves accuracy in deformed geometries.