Error estimates for the Crank-Nicolson in time, Finite Element in space (CNFE) discretization of the Navier-Stokes equations require application of the discrete Gronwall inequality, which leads to a time-step $(\Delta t)$ restriction. All known convergence analyses of the fully discrete CNFE with linear extrapolation rely on a similar  $\Delta t$-restriction. We show that CNFE with arbitrary-order extrapolation (denoted CNLE) is convergences optimally in the energy norm without any $\Delta t$-restriction. We prove that CNLE velocity and corresponding discrete time-derivative converge optimally in $l^∞(H^1)$ and $l^2(L^2)$ respectively under the mild condition  $\Delta t \leq Mh^{1/4}$ for any arbitrary $M > 0$ (e.g. independent of problem data, $h$, and  $\Delta t$) where $h > 0$ is the maximum mesh element diameter. Convergence in these higher order norms is needed to prove convergence estimates for pressure and the drag/lift force a fluid exerts on an obstacle. Our analysis exploits the extrapolated convective velocity to avoid any  $\Delta t$-restriction for convergence in the energy norm. However, the coupling between the extrapolated convecting velocity of usual CNLE and the a priori control of average velocities (characteristic of CN methods) rather than pointwise velocities (e.g. backward-Euler methods) in $l^2(H^1)$ is precisely the source of  $\Delta t$-restriction for convergence in higher-order norms.