Development of selection criteria
As the Ministry of Education has gained experience, there have been continuous changes from year to year of the evaluation criteria for Top Level Courses. These range from minor variations in the rating rubrics used, to more large-scale changes in priorities. The actual rubric used to evaluate courses has five comprehensive indicators: teaching teams, teaching content, teaching conditions, teaching methods and practice and instructional outcome. These five are subdivided into 15 secondary indicators (one was removed in 2008) as follows:
teaching team, including responsible person and main lecturer
the composition and quality of teaching team
educational reform and educational research
the course content
the organization and planning of the course content
the practical components (removed in 2008)
teaching materials and other resources
conditions for practical components
online teaching environment
instructional outcomes - evaluation by peers, by university supervisor,
evaluation of recorded materials
(Wang Peng, 2008)
When it comes to large scale changes in focus, Professor B3 commented that initially the internet resources were not that important, because the Internet was not very developed at the time that the Top-Level Courses Project started. This changed, as the web grew in importance and sophistication, and in 2004 and 2005, requirements about the functionality and design of the course website increased.
Likewise, Mr. B0 stated that the provinces and the Ministry of Education have focused increasingly on the sharing of resources and providing a service to others. This is a marked change from the first few years, when the application process was the main focus. Initially there were not many courses, and attracting a number of high-quality courses was the focus. Later, as the number of courses increased, it became important to offer better access to these resources, which led to the creation of the Jingpinke.com portal (see the later section on this).
Wang Peng (2008) who studied how the evaluation criteria had changed from 2003-2008, also identified a number of trends. He reaffirmed the added focus on instructional design, pedagogy, use of technology, impact and sharing, and the building of local teams. It is often more interesting to look at the explanations given for the different criteria, than the weight assigned. For example, the explanation for the criteria “evaluation of recorded materials” changed from specifying “dignified bearing, clear sound, full of teaching enthusiasm, lively class atmosphere, high student participation rate, and effective student-teacher interaction” in 2003 to asking for "influential teaching which attracts students attention, enlightens and inspires thinking, association and creativity” in 2004 (ibid. 39).
As new categories of courses were added, new rubrics also had to be developed. For example, the evaluation of online courses put more focus on course resources for self-learners, the teaching and learning process, and learning support design. Zeng, Zeng and Fan (2007) believe this to be natural, and mention high student numbers, and a focus on adult learners that need practical and vocationally oriented training as key features distinguishing distance education from traditional universities.