Briefing note: Student questionnaires on courses and teaching

Download a pdf of this document.

It is a common practice at universities to have students complete end-of-term questionnaires about their courses and instructors. Sometimes called student evaluations of teaching (SETs) or student questionnaires on courses and teaching (SQCTs), these are often used to make decisions about faculty tenure and promotion without an appreciation of their limitations. These questionnaires could be good for capturing the student experience, but responses are inherently influenced by factors outside of the professor’s control, including the subject being taught, class size, and the professor’s gender, race, or accent. Further, the comment sections in these anonymous questionnaires can and have been vehicles of harassment.

Ontario’s faculty understand the value of student feedback, but the manner in which this feedback is sought, and the ends to which it is used are problematic. The goal of student questionnaires should be to inform the understanding of the teaching and learning experience, not to punish faculty for their class size, instructional innovations, gender, or skin colour.

To consider these issues, the Ontario Confederation of University Faculty Associations (OCUFA) has set up a working group with experts in methodology, research ethics, and human rights. The group has been tasked with developing a deeper understanding of how student questionnaires are currently being used at Ontario’s universities, defining the limitations of these questionnaires, and developing proposals for ensuring that these questionnaires are used appropriately. The working group is expected to release its report and recommendations later this year. What follows is a summary of the group’s findings so far.

Student questionnaire results are skewed by factors outside of faculty control

When completing a questionnaire, students are influenced not just by their impression of their professor’s instruction, but by their more general experiences in the class, program, and institution. So many factors influence the classroom experience that it is very difficult to determine whether ratings are the result of faculty performance or other contributing factors. For instance, students in larger classes, lectures early in the morning, or more difficult upper year courses are more likely to give low ratings than those in smaller classes, mid-day lectures, or easier first-year courses. In fact, in multiple large studies it has been shown that instructors who help students achieve higher outcomes in future learning receive relatively poor ratings compared with instructors of the same course whose students later attained lower academic outcomes.

Student questionnaire results are skewed by systemic discrimination and bias

Systemic discrimination based on gender, skin colour, and accent is a very real issue on Ontario’s campuses, and one of the places it manifests itself is in student questionnaire results. Research conducted in several countries over the past two decades has shown that women, people of colour, and those with accents receive lower evaluation ratings than their white male peers – regardless of ability. This discrimination and bias even plays out on the basis of course content, with classes about gender and racial issues more likely to receive lower ratings.

Student questionnaires facilitate anonymous harassment

Course evaluation questionnaires are composed of mostly multiple-choice questions. As such, they provide a very limited type of feedback. Accordingly, many such questionnaires include room for comments, allowing students to address topics not captured in the multiple-choice section. Unfortunately, as these questionnaires are anonymous, the comment section has become a means by which many faculty are being subjected to racial and sexual harassment. In the absence of effective precautions, moving the questionnaires online only facilitates this kind of threatening behaviour.

Student questionnaire results can compromise educational quality

Student questionnaires can provide important feedback about the student experience in a course, but not necessarily about a faculty member’s teaching performance. Determining whether a professor is conducting class according to student expectations is not the same as assessing how well students are learning in that class nor whether effective instructional methods have been used. Innovation in the classroom often results in lower SQCT scores even when it improves learning outcomes. If the employment status of faculty is tied to the results of these questionnaires, professors are incentivized to gain favour with their students and make course work less rigorous. That compromises the integrity of courses. Contract faculty are especially vulnerable in this scenario, as many have to reapply for their jobs each term, and the results of these questionnaires could be used to determine whether they are hired again.

Student questionnaire results should not be used to determine university funding levels

With the Ontario government’s intention to expand the portion of provincial funding based on performance indicators in later rounds of Strategic Mandate Agreements (SMA), it is important to be clear that student questionnaire ratings have no more place in allocating funding than they do in setting tuition fees, as proposed in the UK. The aggregation of ratings would compound the effect of biases and would not provide reliable indicators of program quality or respect qualitative differences between programs or institutions. It would ultimately penalize universities for achieving faculty diversity, instructional innovation, and true challenge and long-term learning for students.

Conclusion

Student feedback is important, but the purpose of student questionnaires on courses and teaching should be to help faculty develop their teaching, not to undermine their standing as employees, subject them to harassment, or punish them for factors outside of their control.