When it comes to feedback about faculty or courses, one size does not fit all

Considering the Design of Evaluations of Faculty or Courses

I saw yet another item in the POD listserv whether and how to ‘grade’ teaching, student feedback.  Obviously, any such system requires collecting best evidence and in several different ways, "triangulating" from them to get a more valid, reliable sense of what's been going on. 

The fallacy I see in most such discussions is the assumption that one procedure must fit all kinds of courses and faculty.  (The IDEA Center is a noteworthy exception to 'one size fits all.')

Before getting to judging faculty, let's be clear: the most important target for intensive evaluation is the performance of academic programs (e.g., degree programs, general education, writing or other essential learning outcomes as they are mastered across the curriculum).

• What should students have learned by the time they leave (not just in content and capabilities but also in motivation and perspectives)? 
• What did students learn by the time they left? Compared with the goals? compared with students graduating five years earlier?
• Have equity gaps increased or decreased for students who have been in the program?  
• Has the program made responsible, effective uses of its resources including students' time and money?

Now, let's talk about making judgments about faculty. We can learn more, and waste less time and money, by having different inquiries for different situations

	Purpose -Formative	Purpose - Summative	Sample criteria
The top educators  (or courses) (~5%)	Cross-fertilization among most engaged innovators, SoTL practitioners. Forge direction and leadership cadre for future programmatic improvements	Rewards of distinction	In what ways has your work benefited the practice of other faculty and staff, here and at other institutions?
Broad middle (90%)	Provide feedback useful for faculty growth and course improvement	Use evidence of growth or no growth over the years as one of many inputs into personnel decisions	Use highly engaging practices? With desired effects?
Bottom of the scale (~5%)	Identify and then fix problematic teachers/courses if possible	Identify teachers/courses performing at unacceptably low levels, term after term.	Is instructor showing up? Providing feedback in a timely fashion?

The "broad middle" is the most important group because they do most of the teaching and have far more impact, collectively, on students. Because so many faculty are in this group, any central committee or office will be able to spend comparatively little time with each instructor, teaching assistant, and learning assistant.  So these kinds of feedback need to come from standardized processes (e.g., standard data about course performance; standard review from peers who have been educated about what's worth noticing and how to coach).

The bottom of the scale is the most important for a "high touch" approach because of the risks that the intervention might make things worse.  Each case requires sensitivity and a unique approach.

The top of the scale is also worth high touch, to harvest, interpret and disseminate important lessons and new challenges arising from the work done in these very best courses.

Does your institution have a faculty/course improvement program that treats the worst performers differently than the others -uses different criteria to judge them, gathers different data about them?

The Book in Progress

As you might know, I'm working on a book tentatively titled, Quality, Access, and Affordability: Pursuing 3Fold Gains in Higher Education.  About 2/3 of the planned book is in at least first draft shape.  The book describes 5-6 institutions that have been working for years to enhance how well students learn, to make access more equitable, and to control affordability in time and money, for the students, the institutions, and their benefactors. Even more important, these institutions have all been trying to accomplish such gains in ways that are scalable (engage more than boutique numbers of students) and sustainable for many years to come.

If you'd like to hear more, write a comment below or email me (ehrmannsteve gmail).

Progress on quality, access, and affordability in higher education

For much of the last two years I've been researching and writing a book: 

1. To describe how some colleges and universities have been (re)shaping themselves in order to make sustainable gains in educational quality, access, and affordability at scale, and 
2. From their experiences, to suggest a conceptual framework and implementation principles that can be used to pursue such 3fold gains more effectively and efficiently.

There's a lot of research and writing still to do. The book probably won't be published until late 2018 or early 2019.

Recently, the Teaching, Learning, and Technology Group invited me to give a webinar on some of the most important findings and suggestions emerging from the book.  A video of the event is on YouTube at http://bit.ly/3Fold-FRLV-Feb18.  My talk begins at 6:37 into the video, pauses for 20 minutes at 50:16, picks up again at 1:10:48 and concludes at 1:22:15.  In the talk itself lasts less than an hour.  The book analyzes a number of case histories; some points about Georgia State University's recent history and achievements are included in the talk.

Some Tentative Findings:

1. Pursuing three gains (in quality, access, and affordability) as three independent agendas is probably not the best way to actually achieve such improvements in institutional and program outcomes.  
2. Instead create a single constellation of changes over the years that, as a group, cumulatively improves elements of quality, access, and affordability. ("pursuing 3fold gains")
3. There are three major ways to conceive of how how learning should be organized.  Each suggests a different, incompatible way to pursue 3fold gains. 
4. Institutional case histories also suggest that, for 3fold gains to be sustained at scale, the constellation of changes needs to include targeted improvements aligned across: 

• Educational strategies, 
• Organizational foundations (including culture), to better sustain those strategies at scale, and 
• Interactions with the wider world that also influence the sustainability of the institution's educational strategies. 

I hope you'll take a look at my talk.  Post your reactions here or contact me at ehrmannsteve at gmail.