In this paper, we propose a new conservative semi-Lagrangian (SL) finite difference (FD) WENO scheme for linear advection equations, which can serve as a base scheme for the Vlasov equation by Strang splitting [4]. The reconstruction procedure in the proposed SL FD scheme is the same as the one used in the SL finite volume (FV) WENO scheme [3]. However, instead of inputting cell averages and approximate the integral form of the equation in a FV scheme, we input point values and approximate the differential form of equation in a FD spirit, yet retaining very high order (fifth order in our experiment) spatial accuracy. The advantage of using point values, rather than cell averages, is to avoid the second order spatial error, due to the shearing in velocity (v) and electrical field (E) over a cell when performing the Strang splitting to the Vlasov equation. As a result, the proposed scheme has very high spatial accuracy, compared with second order spatial accuracy for Strang split SL FV scheme for solving the Vlasov-Poisson (VP) system. We perform numerical experiments on linear advection, rigid body rotation problem; and on the Landau damping and two-stream instabilities by solving the VP system. For comparison, we also apply (1) the conservative SL FD WENO scheme, proposed in [22] for incompressible advection problem, (2) the conservative SL FD WENO scheme proposed in [21] and (3) the non-conservative version of the SL FD WENO scheme in [3] to the same test problems. The performances of different schemes are compared by the error table, solution resolution of sharp interface, and by tracking the conservation of physical norms, energies and entropies, which should be physically preserved.