In recent years, with the continuous development of education in China, the physical education in colleges and universities is also improving constantly. The health level of some college students continues to rise. However, the physique of the majority of college students maintains unchanged or even decline. This paper analyzes the current situation of the quality of physical education in colleges and universities, promotes the balanced development of the quality of physical education, optimizes the standard and evaluation method of physical education and then constructs a comprehensive evaluation system of the quality of physical education in colleges and universities. This system not only can improve the quality of physical education, but also benefits the sports management in colleges and universities, ultimately promoting the all-round development of physical education in colleges and universities.