The report of OCUFA the Working Group on Student Questionnaires on Courses and Teaching (SQCT) has now been released. Prompted by increasing reports of the misuse of results and the harassment of faculty through anonymous comments, and by suggestions in the policy community that SQCT scores be used as university “performance” metrics, OCUFA established the working group in 2016 to examine these issues.
Since the working group made its initial observations – that results are skewed by factors outside faculty control, and by endemic bias and systemic discrimination; and that the SQCTs facilitate harassment, compromise educational quality, and are not appropriate for funding allocation – more doubt has been cast on the use and value of SQCTs. The University of Southern California Provost is reported to have said “I’m done. I can’t continue to allow a substantial portion of the faculty to be subject to this kind of bias.” Closer to home, the arbitrator in a case between Ryerson University and the Ryerson Faculty Association accepted expert evidence and found that SQCTs cannot be used to measure teaching effectiveness.
The working group’s mandate was broader than most reviews of SQCTs. In addition to their scrutiny of methodological matters, they also examined the use of student questionnaires in Ontario through research ethics and human rights lenses. Starting from the premise that student learning must be at the centre of why student questionnaires are used in the first place, they found:
Methodology: SQCT scores are affected by endemic bias against women, racialized, and LGBTQ2S+ faculty; the “halo effect” whereby broader impressions of an instructor manifest themselves in responses to specific questions renders any attempts to unscramble the many biasing factors impossible; focus on scores discourages innovation and affects student learning and academic diversity.
Research ethics: SQCTs are not required to pass research ethics review, but should be administered according to similar standards; students need more information to meet standards of active and informed consent; results should be confidential by default; formative SQCTs refocus on teaching and learning and are more conducive to use for the development of courses and teaching.
Human rights: SQCTS are not intentionally biased, but clearly are discriminatory in effect for women, racialized, and LGBTQ2S+ faculty, as well as faculty with disabilities; in addition to the stress and harm it causes, harassment also taints SQCT scores; effects on career prospects and progress limit academic diversity; the best solution is an equity approach limiting use to formative purposes.
Faculty association agreements with universities already articulate and embody a number of principles important for teaching. The working group proposes seven guidelines for refocusing student questionnaires and placing faculty and students, teaching and learning in the foreground. Their adoption must be consistent with faculty agreements and negotiated with faculty associations.
- Limit the use of SQCTs to formative purposes
SQCTS are only suitable for informing faculty about students’ understanding of their learning experience, and most valuable for the further development of courses and teaching. Summative versions for performance evaluation are not equitable and not appropriate for determining pay, tenure, permanency, or promotion for full-time faculty, or appointment and renewal for contract faculty.
- SQCTs should provide useful feedback for instructors
How different the design of formative questionnaires will be from summative end-of-course versions currently in use will vary, but summative questions do not have a place. Nor will a one-size-fits-all model provide instructive feedback if SQCTs are intended to shed light on different iterations of a course. Common questions follow from, rather than guide, the design of formative instruments.
- SQCT results should be confidential except at the instructor’s discretion
Results and scores should not be made public, or shared with anyone other than those whom the instructor chooses. They are dubious guides for students choosing courses. If the questionnaires are formative, the responses should matter to no more than the faculty member, and perhaps those competent to help interpret them and inform teaching strategies. Any departure from this default must be subject to the terms of faculty association agreements.
- SQCTs must seek informed and active consent from students
If harassment is to be challenged wherever it appears, student comments on questionnaires cannot be an exception. Students must be advised of their institution’s policy on harassment, and the scope of confidentiality in the event of an investigation of alleged harassment or threat of violence.
- Surveys for other reviews should be separately administered
To avoid double counting, canvassing respondents not in the relevant population, and tainting results with bias endemic to SQCTs, surveys for program and institutional reviews should be administered separately. Further, no other methods of teaching evaluation should be reduced to numeric scores and used as metrics for program or institutional performance.
- Teaching evaluation requires a suite of tools
If SQCTs are included as part of teaching evaluations, they should be only one tool in a bigger toolkit. The principal methods are the careful examination of teaching dossiers and in-class observation by peers. If SQCT results feature, it is not the scores which are informative but the instructor’s explanation of how the responses figure in the faculty member’s own evaluation and development of their courses.
- Peer evaluation should be the rule
No student graduates with a university credential having taken courses from only one professor: university education is a collective responsibility. Evaluating teaching is a collegial responsibility that should not be contracted out. There is no substitute for peer knowledge of the content, the nature and value of teaching activities outside the classroom, and differences between courses and modes of delivery.
Putting these principles into practice will require resources. A renewed commitment to teaching excellence and academic achievement will require more funding and less focus on metrics from the provincial government. It will also require the willingness of university administrations to allocate resources to support faculty, students, and teaching as vital to the academic mission.
OCUFA thanks the members of the working group for this comprehensive report. The full report can be downloaded here: https://ocufa.on.ca/assets/OCUFA-SQCT-Report.pdf
Peer evaluations are problematic where there are cliques that are supportive for some colleagues and other colleagues are not being properly supported/ treated respectfully for whatever reason (even in some instances a poisoned work environment for some ) In such cases it may be necessary to have an independent review from outside the university in much the same way external reviewers are solicited for consideration of an applicant for promotion to full professor and the applicant him/herself has some say in who the reviewers will be who are considered for the full list from which the reviewers will be selected The objective is unbiased assessment from various reviewers with different perspectives on the matters to be considered
Professors should also have the right to respond to the reviews
In sum peer evaluations are not a panacea and are open to colleague rivalry and various biases There is also an issue of risk to academic freedom where others are perhaps setting demands for teaching that in some way impinge on academic freedom