To realize the visual tracking under the complicated condition, an efficient color object tracking algorithm based on the adaptive is presented in this paper. Based on an adaptive kernel model, the nonparametric statistical modeling of a moving target was carried out with the intensity difference between the target and the background. The search region is extended for searching objects with the background-weighted histogram for statistics of the target feature on the premise of taking the relevance between the target and background into account in order to realize the tracking of the moving gargets in a large area. According to the change of the object and environment, the target model is updated to improve the adaptive ability for environment variation of object tracking. Experimental results on real image sequences show that the algorithm can efficiently track the moving gargets, and the average iteration number reduces 37.28% in comparison with other method.