Evaluating a pilot test of PBL in a virtual world

I'm an advisor to Bernard Frischer's fascinating, NSF-funded project to create a virtual version of Hadrian's Villa and test a problem-based approach to organizing undergraduate and graduate courses in that world. Students would familiarize themselves with this recreation of what the vast complex of buildings probably looked like in Hadrian's time, and do research to suggest how people might have used the complex.

In our latest advisory committee meeting, we considered how to evaluate the pilot tests.

It seems to me that, in such tests, a key stakeholder group are faculty who are mildly interested in the possibilities and who want information from the evaluation to decide whether or not to invest the time, and take the risk, of trying it themselves. In this case, these faculty would be, for example, classicists and archaeologists, not educational psychologists. Their interests wouldn't be "Did these course activities improve critical thinking?"  And it's hard to imagine what elements of history they might wonder, "Did students learn this in a deeper and more lasting way than if I'd taught my class as I usually do, spending an hour on Hadrian's Villa?"

Instead they might have questions such as these:

1. Does the environment provide a huge number of creative possibilities for assignments and class activities, or are there only a few assignments and activities that would work in this space?
2. When one considers student with different characteristics (e.g,. committed to a major in classics or archaeology? background in multi-player gaming?  interest in or aversion to computers? Gender?) do many kinds of students find these assignments and activities engaging? Or just a small fraction of students?
3. When external referees look at student projects, do they think most students learned valid insights about history and culture? Or were many students deluded into imagining that whatever they created, no matter how far-fetched, was real?
4. Were students attracted by their assignments and activities to engage in sustained, cumulative learning over the term? Or did they most ignore it until a frenzied cramming session near the deadline?
5. Did the student work contribute to the instructor’s research and insight? Is that a likely outcome for future instructors?
6. Was learning to use the system a burden for many students? Was using it annoying to many students? Or was the interface sufficiently transparent?
7. Were any satisfactory rubrics or procedures developed to grade student projects?

However, these are questions I imagined. Better yet, I suggested, find faculty who are mildly interested (not those who'd leap at trying this, no matter what), show them a draft list of questions such as those above, and say, "Now, really, what information from the pilot test could persuade either not to try this, or to try it, depending on what the evidence showed? What are the real questions that would influence your decision to alter your course, or develop a new course?"

How can hybrid (and online) courses be better than campus courses?

Many institutions want to offer hybrid and online courses in order to improve the economic bottom line: to make better use of expensive campus space, increase enrollments, and allow working students to carry bigger loads and graduate sooner. Those aren't bad motivations.

But it's also the case that, in selected ways, good hybrid and online courses can be better than campus courses that rely mainly on lecturing, textbooks, and students doing homework alone.

By "hybrid" I mean courses where the instructor allocates more time to work outside the classroom by shrinking the time that students and the instructor spend on the classroom. The goal: improve learning while not increasing total time spent on the course by students or faculty.

Here are a few ways in which such hybrids and online courses can be superior to traditional classroom formats (especially when classes have more than, say, 30 students). Can you suggest others?

Large campus-bound courses teach one message to all students, a message which may motivate some students more than others, at a pace that may be right for some students, too fast for others, too slow for still others. In contrast, courses that do more (or all) of their work online enable faculty to offer more than one version of lectures, assignments, and even tests. The goal: to excite and engage a larger fraction of all students by giving them challenges that mean something at a pace they can maintain. For example, a class in biology might offer three versions of some lectures or assignments: one version pitched more toward future health care practitioners, another for future engineers, and another for future scientists.  (These different versions need not be created by one faculty member all at once. Faculty might split the effort, or a single faculty member might develop different versions over a period of years.)

Just in time learning: In order to motivate students to come to the class sessions prepared, use online quizzes, discussions, or assignments that are due just before class.  Gathering data just before class can also help instructors create classroom activities that respond to what students have learned, or  what they still misunderstand.

More thoughtful pace of discussion: Traditional classrooms, especially when classes are large, offer few students a chance to answer questions, or discuss issues.  Online discussion (e.g.,, threaded discussions on Blackboard) give all students a chance to talk, with each other, and with faculty. The time-delay gives students a chance to think about what’s been said, and to think about what they want to say. Many faculty have observed that students indeed become more thoughtful when the pace of conversation slows down, and there’s no danger of being interrupted.

Expand the visual dimension: Traditional classroom formats are dominated by words.  But the visual dimension (still, motion) is potentially crucial in many disciplines (e.g., art history, health care, sociology, engineering, ...). For students to develop an educated, intuitive understanding of visual materials, students must be able to (a) call up one or more visual artifacts, (b) discuss those artifact(s) with other students and the instructor while pointing to elements of the visual materials ("Notice the contrast between this and that."). Student discussion of visual materials can often be done more easily and inexpensively online (e.g., using Elluminate, Adobe Connect, Wimba, or other real-time conferencing systems or created narrated presentations using software such as PowerPoint or Prezi.)

Open the conversation: Traditional classrooms have a closed door: discussion is limited to the faculty member and students registered in the class. Online discussions are more easily expanded to include significant others outside that group: outside experts, students who differ from those in class, members of the public.  For an extreme example of this, see my previous post.

Expand the variety of students and classwork settings: Online and some hybrid formats enable a program to recruit a student body of unusual talents and diversity.  For example, some hybrid programs (often masters programs) combine substantial online work with periods when students and faculty gather in one instructional setting (not always on the home campus). For example, Prof. Maida Withers, at George Washington University, leads a new blended (hybrid) master's program in dance. Students applying for admission were required to have identified their settings near their residences where they could practice, choreograph, and present their dances.

Expand the expertise used in offering the program: the more interaction is online, the easier and more natural it is to use instructors and guest experts selected for their fit to the program. Guest experts can speak with a classroom from a distance, but it's even more natural if everyone is online, as natural as a conference call. Some instructors use a 20 minute guest presentation or discussion, involving a different specialist, every week. Assign a reading, and then give students an opportunity to interview the author online.

Increase the specialization of the program: Put all these pieces together: it's possible to offer incredible programs that can be be unusually specialized in order to attract (distant) highly motivated students-- interdisciplinary degrees in HMO management, sustainable recreation, 21st century literature and culture, ...  Such programs might be targeted at small, high-paying, recently created job markets. Such programs could be taught by a combination of local faculty and top experts from around the world. The program design would ideally be hybrid: a mix of online education with gatherings of faculty and students at appropriate sites for intense work and coaching.

Education as Choir. As Distributed Choir?

As people do research together, write together, teach and learn together, the experience can be something like a choir of people singing, the whole more than the sum of the parts.
Do you agree?
If so, take a look at this TED Talk about two choirs of singers, each organized by composer and conductor Eric Whitacre:
http://www.ted.com/talks/eric_whitacre_a_virtual_choir_2_000_voices_strong.html

It's been a truism that, the smaller the class, the better.

So, what kinds of teaching, learning, research, and service in the arts, humanities and social sciences can be become better, more engaging, richer and more effective as the numbers of students and faculty grow? as the participants become more diverse? as they are drawn from across a larger geographic area?

Lecture capture and podcasting

I'm starting a consulting assignment for George Washington University this week. I'll spend the next few months helping their faculty consider the options and problems associated with lecture capture and podcasting, how to get the most educational value from them, and how the university should (or shouldn't) support those activities.  There are really at least two distinctly different activities here: faculty (or their students) who create and upload video or audio materials for use within a course, and faculty who record their entire class sessions and upload the recordings for later use. 

I'm eager to get started.  We're going to run focus groups with faculty who use these techniques and faculty who've stopped, along with some of their friends. We'll videotape some interviews with faculty in various disciplines, and put short clips on the web. We'll see where the sources of 'friction' are for newbies, and figure out how to reduce them.   

Know some good stuff for me to read? People I should contact at other universities? web materials that already exist?

Spreading the word about innovations in teaching

As you may know, I spent almost 20 years funding innovative educational projects, most in undergraduate education, many using technology.  Most (though not all) of those projects were guided by the idea that our job as funders was to help demonstrate what was possible; it was then the job of the educational institutions to learn about that (by attending the right conferences and reading the right journals) and using what would work for them.  "Build it and they will come."  Of course things don't work that way. We also funded a few projects explicitly designed to get ideas and materials into wider use: publishing and advertising materials, for example, and experiments with new approaches to faculty development. But these, too, were limited by the three year maximum to our grants. 

Now I'm working on this problem from the other end. Suppose that, in a decade or two, some of the best, easiest, cheapest, and most educationally important techniques and materials we know of today had become quite widely used by 'mainstream' faculty all over this country. How could that have happened? What would funders, accreditors, university leaders, faculty champions and others need to have done in order for something so odd to have happened? I'm especially interested in improving the dissemination and effective adaptation of teaching techniques and resources in STEM (science, technology, engineering, and mathematics). 

I'll summarize some ideas, and sources, in this blog over coming months. In the meantime, if you have ideas, or suggestions about what I should read, or who I should speak with, please post it or email me at ehrmannsteve@gmail.com

What should I do next?

I've finished my full-time work at Drexel and have shifted to adjunct status (teaching grad courses online).   My most recent article, "Taking the Long View: Ten Recommendations about Time, Money, Technology, and Learning," is getting a lot of attention; it was published in the September 2010 issue of Change Magazine and has already been reprinted in the January 2011 issue of Planning in Higher Education.  Here's a link to the penultimate draft of the Change version.

Now I'm taking a breath (no financial pressure to get right back to full-time work) and considering three main options for what I might do next:

1. Return to being a program officer, something I did pretty well for almost 20 years with FIPSE and with Annenberg/CPB.  I like spotting great ideas, and nurturing them. You can advance lots more good ideas and people as a program officer than if you're doing just your own projects. I'm especially interested in strategies for major increases in the use of proven educational practices and innovations.  Gladwell's The Tipping Point is analogous to the kind of research-based approach to dissemination and adaptation that I'd like to seek out and support.
2. Do just my own projects: research on dissemination and adaptation of academic innovations, building perhaps on my recent research on how faculty adopted open source software from iCampus; evaluation of grant-funded projects; evaluation consulting and training; continuing to teach perhaps one course at a time, online.  Continue to write about patterns that I see in innovations and problems across institutions.  Work with an association? Prof. at an institution?
3. Get a position in a teaching center or faculty support unit on a campus. I'd like to help engage much larger numbers of faculty and promote incremental, continual improvements in teaching.   For example, I'm working on a set of ideas for teaching diverse classes in ways made possible, or made easier, with technology.  As my recent article describes, I'd also like to help focus institutional support for those degree programs that are unusually motivated to make well-documented, influential improvements in their learning outcomes. 

A fourth idea is also bubbling: developing an online certificate program for the academic professionals who work with faculty on the improvement of teaching (e.g., in IT, distance ed, libraries, and teaching centers). I've got a lot of ideas about what such a program might look like, but I'd to be part of a small team to develop it, and find the right kind of institutional or association sponsor the program.

Obviously, I can't do all these things. There are two other things I know that should help me figure things out:

1. I want to be part of a team, and the character, wisdom, and good humor of those people matter to me more than almost anything else.
2. I don't intend to move from Silver Spring, MD, but I like travel and am willing to spend lots of time on the road, either visiting places or, if my institution or association is distant from DC, staying in that city several nights a week.

So my friends, and friends of my friends, any pointers?

Polling & Peer Instruction - A Brief Online Intro

While I was working in the Provost's Office at Drexel, I developed this chunk of the web as a way of:

1. Explaining an easy-to-try improvement in teaching/learning that relied on technology; and
2. Doing that in the minimum possible space
3. Doing it in a way that a mildly interested or mildly skeptical faculty member might find reasons, and the information needed, to actually try the technique.

What do you think? Seen it done better? More briefly?