We present a new conservative semi-Lagrangian finite difference weighted essentially non-oscillatory scheme with adaptive order. This is an extension of the conservative semi-Lagrangian (SL) finite difference WENO scheme in [Qiu and Shu, JCP, 230 (4) (2011), pp. 863-889], in which linear weights in SL WENO framework were shown not to exist for variable coefficient problems. Hence, the order of accuracy is not optimal from reconstruction stencils. In this paper, we incorporate a recent WENO adaptive order (AO) technique [Balsara et al., JCP, 326 (2016), pp. 780-804] to the SL WENO framework. The new scheme can achieve an optimal high order of accuracy, while maintaining the properties of mass conservation and non-oscillatory capture of solutions from the original SL WENO. The positivity-preserving limiter is further applied to ensure the positivity of solutions. Finally, the scheme is applied to high dimensional problems by a fourth-order dimensional splitting. We demonstrate the effectiveness of the new scheme by extensive numerical tests on linear advection equations, the Vlasov-Poisson system, the guiding center Vlasov model as well as the incompressible Euler equations.