Briefing note: Student questionnaires on courses and teaching

| 6 comments so far

Download a pdf of this document.

It is a common practice at universities to have students complete end-of-term questionnaires about their courses and instructors. Sometimes called student evaluations of teaching (SETs) or student questionnaires on courses and teaching (SQCTs), these are often used to make decisions about faculty tenure and promotion without an appreciation of their limitations. These questionnaires could be good for capturing the student experience, but responses are inherently influenced by factors outside of the professor’s control, including the subject being taught, class size, and the professor’s gender, race, or accent. Further, the comment sections in these anonymous questionnaires can and have been vehicles of harassment.

Ontario’s faculty understand the value of student feedback, but the manner in which this feedback is sought, and the ends to which it is used are problematic. The goal of student questionnaires should be to inform the understanding of the teaching and learning experience, not to punish faculty for their class size, instructional innovations, gender, or skin colour.

To consider these issues, the Ontario Confederation of University Faculty Associations (OCUFA) has set up a working group with experts in methodology, research ethics, and human rights. The group has been tasked with developing a deeper understanding of how student questionnaires are currently being used at Ontario’s universities, defining the limitations of these questionnaires, and developing proposals for ensuring that these questionnaires are used appropriately. The working group is expected to release its report and recommendations later this year. What follows is a summary of the group’s findings so far.

Student questionnaire results are skewed by factors outside of faculty control

When completing a questionnaire, students are influenced not just by their impression of their professor’s instruction, but by their more general experiences in the class, program, and institution. So many factors influence the classroom experience that it is very difficult to determine whether ratings are the result of faculty performance or other contributing factors. For instance, students in larger classes, lectures early in the morning, or more difficult upper year courses are more likely to give low ratings than those in smaller classes, mid-day lectures, or easier first-year courses. In fact, in multiple large studies it has been shown that instructors who help students achieve higher outcomes in future learning receive relatively poor ratings compared with instructors of the same course whose students later attained lower academic outcomes.

Student questionnaire results are skewed by systemic discrimination and bias

Systemic discrimination based on gender, skin colour, and accent is a very real issue on Ontario’s campuses, and one of the places it manifests itself is in student questionnaire results. Research conducted in several countries over the past two decades has shown that women, people of colour, and those with accents receive lower evaluation ratings than their white male peers – regardless of ability. This discrimination and bias even plays out on the basis of course content, with classes about gender and racial issues more likely to receive lower ratings.

Student questionnaires facilitate anonymous harassment

Course evaluation questionnaires are composed of mostly multiple-choice questions. As such, they provide a very limited type of feedback. Accordingly, many such questionnaires include room for comments, allowing students to address topics not captured in the multiple-choice section. Unfortunately, as these questionnaires are anonymous, the comment section has become a means by which many faculty are being subjected to racial and sexual harassment. In the absence of effective precautions, moving the questionnaires online only facilitates this kind of threatening behaviour.

Student questionnaire results can compromise educational quality

Student questionnaires can provide important feedback about the student experience in a course, but not necessarily about a faculty member’s teaching performance. Determining whether a professor is conducting class according to student expectations is not the same as assessing how well students are learning in that class nor whether effective instructional methods have been used. Innovation in the classroom often results in lower SQCT scores even when it improves learning outcomes. If the employment status of faculty is tied to the results of these questionnaires, professors are incentivized to gain favour with their students and make course work less rigorous. That compromises the integrity of courses. Contract faculty are especially vulnerable in this scenario, as many have to reapply for their jobs each term, and the results of these questionnaires could be used to determine whether they are hired again.

Student questionnaire results should not be used to determine university funding levels

With the Ontario government’s intention to expand the portion of provincial funding based on performance indicators in later rounds of Strategic Mandate Agreements (SMA), it is important to be clear that student questionnaire ratings have no more place in allocating funding than they do in setting tuition fees, as proposed in the UK. The aggregation of ratings would compound the effect of biases and would not provide reliable indicators of program quality or respect qualitative differences between programs or institutions. It would ultimately penalize universities for achieving faculty diversity, instructional innovation, and true challenge and long-term learning for students.

Conclusion

Student feedback is important, but the purpose of student questionnaires on courses and teaching should be to help faculty develop their teaching, not to undermine their standing as employees, subject them to harassment, or punish them for factors outside of their control.

If you enjoyed this article, please consider sharing it!

6 Responses to “Briefing note: Student questionnaires on courses and teaching”

  1. Anne Savage

    I agree with all of the above. Getting assignments back in good time with comments is another aspect students criticize, not understanding that TAs are in training, and that as an instructor I must look over the papers with their comments on before returning them; or that I have only two TAs for a class of nearly two hundred students; or that I have no TAs for the course.

    I especially disagree with our management’s use of these evaluations as a way of ranking our performance, especially under worsening conditions in the classroom.

    Reply
  2. Helen McDonald

    Thank you for this thoughtful response to a thorny issue. I am an associate prof in the Midwifery Education Program at McMaster University. This is a small program of 30 students. The Program is highly structured, most courses are mandatory leaving tutors of unpopular or difficult courses quite vulnerable. In addition tutor (or preceptor in the clinical setting) may be provided by only a very limited number of students, making any interpretation of the feedback quite difficult.

    Reply
  3. Laurie Kruk

    Thank you for initiating an important discussion of Student Teaching Evaluations (or as they are termed at my institution, Student Opinion Surveys). It is a very important to explore their continuing use in hiring, tenure and promotion processes.

    I was especially struck by this comment from OCUFA’s study:

    “In fact, in multiple large studies it has been shown that instructors who help students achieve higher outcomes in future learning receive relatively poor ratings compared with instructors of the same course whose students later attained lower academic outcomes.” Although I am not entirely sure how to read this, it SOUNDS as if teachers are being ‘marked down’ for actually helping student to develop and learn, now and in the future. Food for thought?

    Sincerely,

    Laurie Kruk, Associate Professor, English Studies
    Nipissing University

    Reply
  4. Zachary Rose

    Very interesting! OUSA noted similar concerns when researching SETs for our policy papers. However, we maintain that student feedback has the potential to be an important and useful source of information. The next step, in our view, is discussing how to design SETs that eliminate harassment, reduce bias, etc.

    Reply
  5. Gavin Moodie

    There would be far less reliance on student questionnaires if there were systematic evaluations of the quality of teaching. Just as it is inconceivable to make important decisions about research performance without peer evaluation, so peer evaluation should be central to teaching.

    Reply
  6. Catherine Stewart

    I was very interested to read about the initial research on evaluations. I would like to suggest three other factors be taken into account: 1) age/gender and race/gender of instructor (i.e. older women as well as women of colour); 2) content of the course, particularly if the course deals with emotional and/or controversial issues, or if the course challenges the status quo in any way by the information/data presented, or if it deals with an unpopular subject like statistics; and 3) whether or not the course is mandatory. I would argue that there is an intersectionality that applies. That is an older woman or woman of colour, or combination of all three, teaching something controversial/emotional or unpopular/difficult and/or mandatory.

    Reply

Leave a Reply