In this paper, we extend the unified gas kinetic particle (UGKP) method to the frequency-dependent radiative transfer equation with both absorption-emission and scattering processes. The extended UGKP method could capture the diffusion and free transport limit and provide a smooth transition in the physical and frequency space in the regime between the above two limits. The proposed scheme has the properties of asymptotic-preserving and regime-adaptive, which make it an accurate and efficient scheme in the simulation of multiscale photon transport problems. In the UGKP formulation of flux construction and distribution closure, the coefficients of the non-equilibrium free stream distribution and near-equilibrium Planck expansion are independent of the time step. Therefore, even with a large CFL number, the UGKP can preserve a physically consistent ratio of the non-equilibrium and the near-equilibrium proportion. The methodology of scheme construction is a coupled evolution of the macroscopic energy equation and the microscopic radiant intensity equation, where the numerical flux in the macroscopic energy equation and the closure in the microscopic radiant intensity equation are constructed based on the integral solution. Both numerical dissipation and computational complexity are well controlled, especially in the optically thick regime. 2D multi-thread code on a general unstructured mesh has been developed. Several numerical tests have been simulated to verify the numerical scheme and code, covering a wide range of flow regimes. The numerical scheme and code we developed are highly demanded and widely applicable in high-energy engineering applications.