Questions: When good 'teaching' can defeat learning

What is it called when a student sits listening to lectures week after week, thinking, "Yes I understand." But suddenly, in a later lecture, homework assignment, project or test, the student realizes. "I'm completely lost!"  


Is there a name for the student's delusion of understanding? or for the sickening realization that it was indeed a delusion?  It's such a common phenomenon, but I don't know what to call it. 

Ironically, this phenomenon may be most common when the lecturer is "good" in a traditional way: crystal clear in explaining things, visibly caring about students, perhaps charismatic and riveting.  

There's another learning problem, also unnamed, that can afflict students of particularly good professors. I still have it often myself. The lecturer makes a provocative point, and I start thinking about it. A moment or two later I suddenly begin hearing the lecture again and, if I'm unlucky, am completely lost - for a few minutes and sometimes for the remainder of the talk. It happens to me all the time at conferences. I wonder how often it happens to students? Does it happen more often to good students who are really actively listening?   Is there anything a faculty member can do to give students the time needed to process what they've just heard before trying to cram more invaluable information and insight into their short term memories?  Please post a comment or send me an email.

The troubling question of grades

I wasn't in charge of a college course until I was 62.  So please forgive me for yapping about something that many readers of this blog have thought about for years.

As I was grappling with the problem of grading my students ("How much should they be able to do in order to rate a "B+?"), I talked with my old friend Tom Angelo, who reassured me that grading was the shameful secret of teaching. 'No one can really tell you how to do it properly.' For example, faculty will disagree forever on such profound questions as "Are there many courses and circumstances in which it would be appropriate for all students in a large class to earn an A?"  At GW and many other institutions many faculty are upset about 'grade inflation' (such a large fraction of students being given A's or B's.)

Against this backdrop, here are three pieces of information.

The first I encountered years ago: there is virtually no relationship between undergraduate GPA and anything about the student's experience in life after college (except for graduate school grades). Grades don't predict job success, voting behavior, likelihood of winning a Nobel Prize, or anything else.

• 1984      Cohen, Peter, “College Grades and Adult Achievement: A Research Synthesis,” Research in Higher Education, XX:3, 281-293.

I've only seen one study of GPAs in professional schools - it was about a business school - and they didn't find any relationship between an MBA student's grades and later success in business, either.

The second piece of information comes from  Ken Bain's classic book from 2004, What the Best College Teachers Do.   On page 32, he writes (I'm paraphrasing) , "For years, psychologists have studied what happens when someone has a strong intrinsic interest in something, and they're also offered an extrinsic reward to do it (e.g., a high grade, money, etc.).  Would that reward increase their fascination with the subject, leave it unaffected, or decrease it?"

In one set of experiments by Deci and others, students were asked to play an unfamiliar game under supervised conditions.  Some students were paid money for winning, while the others received no rewards for their performance.  When left 'alone,' the students who'd been competing for prizes stopped playing the game. In contrast, students who had not played for any reward often continued playing after the supervisor left.  Variations of this experiment showed that, in order to provide external feedback while doing minimal damage to the student's own interest in the subject, it's best to combine feedback about performance 'Here's what you did right; here's how do it better.') with selective praise for performance ('The way you castled your pieces was quite ingenious!').

Bain reports that decades of research by different investigators indicate that, 'when given a reward for good performance, students' own interests in the subject actually go down and sometimes disappear.'

But why should instructors worry about whether grading is sapping the student's own interest in the subject if students are doing well enough in the course?

If literature students no longer love novels so much, or if engineering students lose their taste for creating things, they're less likely to keep using their academic learning on their own.  We're trying to help students develop their 'thinking muscles.'   If students leave a course (still) loving what they've learned, they're more likely to find ways to use that capability and, in the process, get even better.  (If the receipt of a final grade also stops the use of those thinking skills, on the other hand, it seems possible that those skills will atrophy.)

Those great college faculty that Bain studied do everything possible to increase the students' intrinsic motivation, and tap that energy.  That fits what I've seen over the years.  For example, I've written about Jon Dorbolo's experience as a novice teacher of introductory philosophy.  When his first course went sour, Dorbolo asked his students to rate every assignment anonymously. To his dismay, a majority of his students rated every single assignment useless and boring. However Dorbolo then noticed that every assignment was rated useful and interesting by a minority of students.  Different students liked different sets of assignments. Some students liked only religious philosophers.  Some liked only utilitarians.  Dorbolo realized that his most important goal was to help students learn the rudiments of philosophical thinking. And an implicit goal was to not teach them to hate philosophy.

So, by the next time Dorbolo taught the course, he'd created different, overlapping reading lists. Each set of assignments helped students learn skills of close reading and analysis.  But each student got to choose a set of readings most likely to motivate them to think hard, work hard, and appreciate what they were learning.  Dorbolo also converted the course from mostly lecture (since it made no sense to lecture about one reading when most students in the room hadn't seen it).  Instead class sessions were mainly devoted to small group discussion. Dorbolo saved a lot of time that would otherwise have been devoted to creating and delivering lectures.  Instead he focused on managing the discussions, assessing learning, and studying what helped and hindered student learning.

But what about grades?  Is it possible that grade inflation is a good thing, if it motivates us to provide feedback and documentation of student learning in other, less destructive ways?

Do you know of any faculty who've looked at research on assessing and valuing student performance? motivating learning? Tried any experiments?  If so, please add a comment.

PS. If you have any trouble posting comments on the blog, please email me.

Post slightly revised on May 19, 2012

THATcamp

I just came across some mentions on the ProfHacker site about THATcamp (The Humanities and Technology camp), a plethora of  self-organizing get-togethers around themes. There's one at George Mason next month around the theme of "History and New Media" (registration now capped at 150).  Sounds like folks are staying at a motel and meeting in a research center, not in tents!  Have you ever taken part in, or organized, a THATcamp?

PS. If you try to post a comment on this blog, and have problems doing so, please email me at sehrmann@gwu.edu and let me know. Thanks!

Dysfunctional Illusions of Rigor

Several years ago, Craig Nelson wrote a book chapter attacking 'dysfunctional illusions of rigor.'  There were seven ideas about rigor which he no longer believed:

1. Hard courses weed out weak students. When students fail it is primarily due to inability, weak preparation, or lack of effort.
2. Traditional methods of instruction offer effective ways of teaching content to undergraduates. Modes that pamper students teach less.
3. Massive grade inflation is a corruption of standards. Unusually high average grades are the result of faculty giving unjustified grades.
4. Students should come to us knowing how to read, write, and do essay and multiple-choice questions.
5. Traditional methods of instruction are unbiased and equally fair to a range of diverse students of good ability.
6. It is essential that students hand in papers on time and take exams on time. Giving them flexibility and a second chance is pampering the students.
7. If we cover more content, the students will learn more content.

"Rigor" is a term I've heard used frequently in conversations about college courses, but rarely (until now) defined. Nelson's book chapter, adapted in Tomorrow's Professor 1058 and 1059, explains the problems inherent in each of those illusions, and suggests a way of defining rigor that is more likely to educate students than to repel or reject them.  (For example, he discovered that giving every exam twice and allowing students to use the better of the two grades actually improved student learning because most of them studied twice for each exam.)  Although the extensive data Nelson cites come mainly from science and mathematics education research in colleges, most of his argument applies equally well to other fields. 

A GW Hybrid Master's Program in GSEHD

Patty Dinneen (TLC) and I just attended a master's portfolio defense at the Graduate School of Education and Human Development (GSEHD).  This hybrid program (Early Childhood Special Education) takes advantage of study on-campus, on-site (clinical experience) and online.

The ECSE program is organized around a set of about fourteen capabilities that students develop cumulatively as they take courses (e.g., the ability to use a variety of assessment procedures as they work with children).  Each ECSE course helps students take another step in developing several of those capabilities. And in each course the student uploads projects to their portfolio and writes about how those documents provide evidence of what they can now do.

As the program comes to a climax, the students use their portfolios to describe their personal philosophy of the practice and document their achievements in each of the fourteen areas as they relate to that philosophy.  It was obvious that this particular student, Rebecca Parlakian, was exceptionally good at articulating what she had learned and how she now thought as she worked in the profession; she drew interesting lessons from each of the experiences she had documented.

It's looking more likely that the Innovation Task Force will help us work with GW departments at creating hybrid master's programs.  The ECSE program is just one example of what such a hybrid degree program might look like.

Interdisciplinarity at Stanford

The article linked to the title of this post, and also this Ken Auletta piece from the New Yorker, paint an intriguing picture of an effort at Stanford to develop a more interdisciplinary undergraduate education there. Worth reading!

Assessment - Improving Learning AND Accountability?

In a recent op-ed in the New York Times, "Testing the Teachers," David Brooks contrasted the increasing price of higher education with the evidence that results are getting worse.   For example, he cited studies showing that the reasoning ability of graduating seniors is disappointingly poor, significantly worse than a couple of decades ago. A co-author of one of those studies, Josipa Roksa, spoke at our Teaching Day this past October. (Click here to see a video of her talk.) People have faith that going into years of debt is worth the price. Taxpayers have faith that spending large amounts of tax money on student aid and research is justified. What if that faith begins to falter? Brooks suggests that academics study learning in order to figure out the problems, and fix them, before it's too late.

Fortunately, many faculty (like Nobel Laureate Carl Wieman who spoke here at GW last October) have been working on the same problem because of their own doubts about whether their students are actually mastering even the most elementary ideas and capabilities in their disciplines.

As Wieman showed, the problem is that hard-working faculty can spend enormous effort on teaching with methods that sometimes don't work very well.  (As a rueful Gregory Peck remarked in "Twelve O'Clock High," "I'm choppin' but no chips are flyin'.")

Wieman reported research that compared two sections of physics, each of about 270 students.  In one, students were taught by an experienced faculty member using traditional, respected methods of instruction. The other section was led by an inexperienced GTA who had been trained to use new methods that continually challenged students to think and work together throughout the class rather than silently taking notes.  The worst of the GTA's students learned as much as the best students learning from traditional methods.  Attendance in the GTA's section was substantially higher, too.   Wieman's study appeared in Science on May 13, 2011 (p. 862-864)  (Click here to see a video of Wieman's talk.)  These kinds of results are not unusual, but, when a Nobel Laureate does the research, it tends to draw attention to the new possibilities.

To oversimplify, here are a few ideas, all of which are already being tried by at least a few GW departments:
1. Figure out what capabilities your program is trying to develop in all your students by the time they graduate.  For example, an engineering program might be trying to educate students who can figure out a design that can solve a problem, convert that design into a production plan, produce a product and test it.
2. Don't expect that one course can develop one capability by itself ("We need our graduates to be ethical so we will require them to take an ethics course.")
3. Assign students projects and other activities in course after course: activities that reveal whether those capabilities are actually developing satisfactorily.  Use capstone courses, senior projects, and student portfolios to see whether, by graduation, students have learned what they need to learn.
4. Where many students are having problems making progress, don't get waste energy on the blame game. Instead find different ways to teach them. While you're at it, look for methods that can also save faculty and students time.
5. Of course, each student will also learn many things that beyond those core capabilities. Those same student projects and activities (#3) can help the student and faculty see how each student's distinctive achievements (and problems). Taking a good look at that evidence may suggest ways of improving this aspect of students' education, too.

I heard about an example of these ideas in action a few days ago when I talked with Prof. Jay Shotel, Chair and Prof. of Special Education and Disability Studies in GSEHD.   Jay and his colleagues have identified 14 capabilities that all their master's students in Early Childhood Special Education should master.  In each course, students do projects that help them and faculty see how they're progressing in developing those capabilities. And the students continually write about that learning ("reflection").  As part of those courses,  students are also each developing an online portfolio -- a collection of their projects that, along with those reflections, documents the cumulative development of those 14 capabilities.  They present their portfolios as a requirement for graduation. And they can use that evidence to help get jobs.  (Jay, did I get this right?)

PS. I was also fascinated to see that this GSEHD program is a 'hybrid,' i.e., it combines learning at Foggy Bottom, online learning, and clinical experience at off-campus sites.  If the Innovation Task Force approves our plans, we'll be selecting a couple of faculty teams each year that want to take advantage of the hybrid format in order to develop world class master's and graduate certificate programs.  We will support the winning proposals with grants and expert consulting help.  Let me know if you'd like to learn more about these ideas.

One Reason Why Faculty Resist

I've been enjoying helping Natalie Milman by writing contributing several "Ends and Means" columns for the magazine Distance Learning this year.  Today I'm submitting my most recent contribution (and my last one for a while) called "One Reason Why Faculty Resist."  It begins:

"Have you ever heard the phrase “resistance to technology” used to imply that some faculty are irrational dinosaurs? I have, and I don’t like it. In my experience, most such resistance is quite reasonable.  The following story about online discussion in real time suggests what worries these instructors."

My story's point is that, when the terrain of teaching and learning changes, faculty are quite likely to encounter unexpected problems in their courses.  I don't just mean technical problems.  I mean problems with teaching and learning that are frustrating, embarrassing and sometimes potentially threatening. And, when they encounter such problems, they may well blame themselves. And they may feel that student reaction may put them at risk.

We know all this. But most institutions do little or nothing to prepare faculty for those problems.

So one reason for faculty "resistance" is that they sense that the effort to get them to teach online is a bit of a con game: "Come on in, the water's fine!"  Young technology staff,  leading training workshops, probably aren't aware of the problems. And, when workshops are led by faculty enthusiasts, they often  paint a rosy picture because they discount once-painful problems and don't want to scare their colleagues away.

My column concludes with some suggestions for how to organize self-sustaining, scalable, inter-institutional faculty conversations about teaching a particular course (e.g., "Econometrics 202; ). Their online and face-to-face discussions should be comparatively brief, brisk, relaxed, and helpful enough (trading tips, insights and moral support about what happened last week) that faculty would look forward to next week. That's the theory. Perhaps we can start a few such groups from GW.

If you'd like to learn more, the column should be published in a few months. Or contact me and I'll send you the draft.

Beyond "Comparability"

My colleague Patty Dinneen and I are just finishing up a column for publication in the next issue of "Distance Education."  (In the last issue, Natalie Milman and I wrote a column about ways in which online educational formats make it easier to identify and respond to the different needs of different students in the same course.)

This new column, entitled "Beyond 'Comparability,'  begins:

“Comparability”: an institutional strategy for assuring quality in online and hybrid courses by insisting that the content and, sometimes, the assessments be “comparable” to courses already offered on campus.  

As a standard, “comparability,” sounds reasonable enough.  After all, this sameness makes it possible to compare the quality of learning outcomes without regard to delivery method.  So long as the distant learners get test scores that are comparable to the students on campus, all is well and no further thinking or oversight are needed.

In a similar vein, Richard Clark argued in a classic article that the quality of learning is unrelated to the technology used for teaching.  For example, previous meta-analyses of huge numbers of studies of ‘presentation’ (i.e., information from a single source is made available to many students) – these studies had shown that students who are taught by presentation learn just about as much, no matter what the medium of presentation.   It doesn’t matter whether they get the presentation via live lecture, video tape, streaming video over the internet, or textbook. Clark also pointed out that the activity of self-paced instruction (SPI) produces substantial learning gains over the activity of presentation. But SPI implemented using paper produces almost as much learning as SPI using computers. According to Clark,  using technology for teaching is analogous to a vehicle delivering your groceries to your home.  The quality of milk  is the same, whether the delivery truck is made by GM or Ford.  Technology and quality are completely unrelated, he argued. (Clark, 1983: 445)

But Clark’s analogy is misleading, and his conclusion is the problem with the standard of comparability as well.  It’s true that any teaching/learning activity can be implemented with a variety of technologies or facilities. However, for any particular teaching/learning activity, some facilities or technologies are a better fit than others.  For example, SPI can be done far more easily and inexpensively with digital technology than with paper; that’s why such tutorials have become more common as computers have become more common and why paper versions of SPI have become almost extinct.  The process approach to writing, a pedagogy, spread once computers became common because rewriting is easier with computers.  Course activities involving analysis of video (e.g., video recordings of science experiments in action; film clips) became more common when individual manipulation of video became inexpensive and easy.

Once the medium or tools of learning change, it also becomes easier to change who is involved in the course.  Obviously distance learning makes it possible to involve not only more students but also students with specific kinds of backgrounds or needs.  Equally important, the institution can make different choices about who to use as instructors, or assessors of student work, when those activities can be done online.

Changes in learning spaces and tools can also enable improvements in assessment: self-grading assessments can be administered more readily online, for example.

And the dominos keep falling.  When changes in learning spaces and tools enable improvements in the activities, assessment and people, the content and goals of the course, or course of study, can be improved, too.  In the early 1980s, for example, Prof. Marvin Marcus of the University of California Santa Barbara, was able to use a new computer lab in mathematics to begin offering the math department’s first minor in applied mathematics, consisting of several on-campus courses and an off-campus internship program in which students applied their skills to solving problems faced by community agencies.  A more recent example: the Internet and cheaper international communications have helped Worcester Polytechnic Institute make research abroad become so easy, inexpensive and common that applications research abroad has become a signature activity of that institution.

When universities change technologies and/or facilities (e.g., from campus-bound to hybrid), faculty ought to take a fresh look at learning goals, content, teaching/learning activities, and assessment. The change of facilities will make some goals harder to pursue than before, others easier; some teaching/learning activities easier, others harder; and so on.  The problem with “comparability” as a standard is that it discourages faculty from thinking about how they might take advantage of new learning spaces and tools in order to offer more valuable hybrid or online courses of study. 

Remember the old tale about the tiger that had been caged since birth. It would roam its cage ceaselessly. One morning it awoke. The bars had been removed. But for a long time, the pacing tiger didn’t notice. It continued to pace within the boundaries of its vanished prison."  ....

The column goes on to summarize nine different ways in which online or hybrid formats create the potential for courses that are more valuable and more effective than do campus-bound course formats.

The column ends, in part:  "If “comparability” should not be used to provide a quick and easy method of quality assurance, what should we do instead?  Our answer is simple: we should evaluate online and hybrid offerings in the same way we ought to assess campus-bound offerings:

1. Are we doing the right thing?  Use internal and external points of reference to discuss whether the goals are valuable.  This will almost always involve comparing ‘apples and oranges’ so it’s important to think carefully about what points of reference to use. 
2. Are we doing the thing right? Ask whether there is a good alignment between that goal, the teaching/learning activities proposed, and the facilities and technologies to be used to support those activities.

Do you think that we should abandon ‘comparability’ as a standard for quality assurance for online and hybrid programs? 

Evaluating a pilot test of PBL in a virtual world

I'm an advisor to Bernard Frischer's fascinating, NSF-funded project to create a virtual version of Hadrian's Villa and test a problem-based approach to organizing undergraduate and graduate courses in that world. Students would familiarize themselves with this recreation of what the vast complex of buildings probably looked like in Hadrian's time, and do research to suggest how people might have used the complex.

In our latest advisory committee meeting, we considered how to evaluate the pilot tests.

It seems to me that, in such tests, a key stakeholder group are faculty who are mildly interested in the possibilities and who want information from the evaluation to decide whether or not to invest the time, and take the risk, of trying it themselves. In this case, these faculty would be, for example, classicists and archaeologists, not educational psychologists. Their interests wouldn't be "Did these course activities improve critical thinking?"  And it's hard to imagine what elements of history they might wonder, "Did students learn this in a deeper and more lasting way than if I'd taught my class as I usually do, spending an hour on Hadrian's Villa?"

Instead they might have questions such as these:

1. Does the environment provide a huge number of creative possibilities for assignments and class activities, or are there only a few assignments and activities that would work in this space?
2. When one considers student with different characteristics (e.g,. committed to a major in classics or archaeology? background in multi-player gaming?  interest in or aversion to computers? Gender?) do many kinds of students find these assignments and activities engaging? Or just a small fraction of students?
3. When external referees look at student projects, do they think most students learned valid insights about history and culture? Or were many students deluded into imagining that whatever they created, no matter how far-fetched, was real?
4. Were students attracted by their assignments and activities to engage in sustained, cumulative learning over the term? Or did they most ignore it until a frenzied cramming session near the deadline?
5. Did the student work contribute to the instructor’s research and insight? Is that a likely outcome for future instructors?
6. Was learning to use the system a burden for many students? Was using it annoying to many students? Or was the interface sufficiently transparent?
7. Were any satisfactory rubrics or procedures developed to grade student projects?

However, these are questions I imagined. Better yet, I suggested, find faculty who are mildly interested (not those who'd leap at trying this, no matter what), show them a draft list of questions such as those above, and say, "Now, really, what information from the pilot test could persuade either not to try this, or to try it, depending on what the evidence showed? What are the real questions that would influence your decision to alter your course, or develop a new course?"