Spoken English learning has always been the largest obstacle to the English communicative competence of Chinese students. This paper adopts the in-depth learning method based on computer-aided speech recognition technology to achieve the automatic speech recognition of spoken English and to overcome the defects of spoken speech evaluation in the traditional mode such as fairly high subjectivity and low evaluation efficiency. Meanwhile, the evaluation method of the traditional computer-aided spoken speech quality has been improved, and multi-parameter indicators such as accuracy, speed, rhythm and intonation have been taken into account. Through the establishment of the objective and efficient spoken English speech recognition and evaluation model, we are able to provide true and credible evaluations and timely feedback guidance for learners. The research results effectively guide learners to study spoken English, which uplifts the existing level of spoken English learning in China.