return to speaker list

FIRST ANNUAL FEDERAL FOCUS ED-MENTOR SYMPOSIUM

DR. KEVIN O'NEILL

Biography

Dr. O'Neill is a Research Associate in the Learning Research and Development Center at the University of Pittsburgh. He received his Ph.D. in Learning Sciences at Northwestern University, and he is currently completing a post-doctorate fellowship from the James S. McDonald Foundation. His research on telementoring began as part of an NSF-sponsored project at Northwestern called Learning through Cooperative Visualization.

Mentoring in the open

Overcoming developmental challenges in telementoring

D. Kevin O'Neill
LRDC, University of Pittsburgh

Presentation

I am going to take a little while to weave my way into the theme I want to talk about today, Mentoring in the Open. What you will see here is a collection of work that has taken place across the three different institutions at which I have been in the past few years. However, to set the scene for talking about the things that I believe are important in this realm. It has been observed many times in the past that when people first get their hands on a new technology, they have a difficult time envisioning all the ways that it might truly improve the nature of what they do. So, what they tend to start with are more or less trivially technologized versions of what they did in the past. This is a growth experience, and it is necessary to go through that stage.

One of the powerful, venerable knowledge is tutoring. I am not up here to knock tutoring. I have benefitted a great deal from tutoring in my own life. However, in this realm, we have to think about ways that we can do better than tutoring. Again, this is the search for ways that technology can improve what we do in a qualitative way. So, there are two problems with traditional tutoring:

• The work that students are engaged in is generally not very authentic. Tutoring generally takes place in a well circumscribed domain in which you can specify very, very clearly what the learning objectives are.
• Students have very little agency. In a traditional tutoring interaction, students are assigned problems by the tutor, the tutor diagnoses their performance, does some just-in-time instruction, and then assigns a new set of problems until the domain is mastered. Obviously, in this way, students do not learn very much about how to take control over their own learning. They do not learn those sorts of higher order, metacognitive processes necessarily.

Models for "Experts" in Education

Tutoring

• Work not very authentic; students have little agency

One-shot expert visits

• e.g. talking head videoconference, science fair judging
• High pressure; little or no opportunity to apply lessons learned from failures; little opportunity for incidental or collateral learning
• Ongoing advice and guidance (mentoring)
• Frequent and routine consultations
• Students have strong agency; opportunities to obtain feedback and apply lessons right away; opportunities for collateral learning about adult careers

There is another powerful model here from which I think we need to differentiate ourselves. I call this guest appearances. Probably the prototypical example is science fair judging. Once again, I want to make sure that no one thinks I am up here to knock science fairs.

Well, the first problem with science fair - and this is what my colleagues and collaborators with whom I have worked in the teaching profession have told me repeatedly - is that it is a very high pressure, one-time event. It is high pressure, in part, because it is one-time. There is little opportunity for students to take the lessons that they learned from these visiting advisors and apply them, and get some feedback on how they applied these lessons. These people do not come back again. In fact, with science fair, you really will not get the opportunity to get feedback on how you applied new lessons unless you go on to the next stage of the judging. And very few students, by definition, do that.

So, when I talk about telementoring, and I am much in agreement with most of the people who have spoken today about this conception of mentoring, I am talking about in a similar way to Judy's work, curriculum-based telementoring, and I am shooting toward the ideal of frequent and routine consultations about subject matter that students are learning. And those frequent and routine consultations take place in a setting in which students have strong agency over the work they are doing. They also have opportunities to apply these lessons and get feedback right away on how they applied those lessons.

Here is a short history of how I got involved in this and how my interests in it have developed over time.

Short history of my research

• 1992-93: Co-designed an educational groupware tool, the Collaboratory Notebook
• 1994-95: Realized I had a lot to learn about telementoring relationships. Went "low-tech", and worked with teachers to orchestrate these relationships over e-mail
• 1997-99: Began orchestrating telementoring relationships in a more richly structured group medium Knowledge Forum

Back in 1992, I was a new graduate student back at Northwestern University working on this Co-Viz (Learning through Cooperative Visualization) project that was mentioned earlier. This was a project run by Roy Pea and Louis Gomez with the intent to make high school and later middle school instruction in science more authentic to the practice of science in the adult world. My first contribution to that work was to build a tool called the Collaboratory Notebook, which was one of the first educational groupware tools. We really intended it to be the kind of software environment in which students and mentors could have the kinds of relationships I am talking about, in which it would be practical to have routine consultations about ongoing learning challenges.

Well, after a couple of years of that, I realized there was a lot I had to learn about telementoring to begin with. I really tried to put the software a little too soon, given what I knew. So, I struck up partnerships with a couple of wonderful teachers, one of whom is a fellow named Rory Wagner, who shows up a great deal in my writings. For the past two years, I have been working on a fellowship from the James S. McDonnell Foundation at the University of Toronto with two researchers there, Marlene Scartimalia and Karl Bereiter, who have been working on these tools for computer-supported collaborative learning for almost ten years now. They would be homes for telementoring activity in which it would be designed in from the start. You would attain all of the educational objectives, and you would not need to redesign projects from scratch every time.

Everything I will talk about revolves around three issues that we have been swirling around all day. The first one is, "What do students get out of telementoring?" There are a variety of ways to address this. I have written some lengthy case studies about successful and unsuccessful telementoring relationships and how they work. I will talk briefly today about analysis I have done on the influences that these telementoring relationships appear to have on the way students think about and write about scientific arguments.

Issues I've explored

What do students get out of the telementoring?

• Interviews, case studies,
• Analyzed influences of telementoring on students' argument strategies in science

What do mentors get out of telementoring?

• Extensive interview research

What technologies would ideally support telementoring?

• CoVis Mentor Database
• Knowledge forum design experiments

Another question we have visited today is, "What do mentors get out of telementoring?" I have been doing a funded project with the Office of Learning Technologies, Human Resources Development, Canada, for the past year or so in which we are doing qualitative coding of intensive interviews with past telementors to try to explore that.

And then, the last theme is, "What technologies would ideally support telementoring?" We have talked about types of backing technologies, sophisticated websites and search engines, and such like. I have developed one of those myself called The Kovia's Mentor Database. Today I will not talk about that. I will talk about these design experiments I have been carrying out with knowledge forum.

Typically, when I have orchestrated telementoring along with my collaborating teachers, we have worked in the context of curriculum-based projects, usually science projects, designed to give students the freedom to define a research agenda from scratch. With younger kids, we may allow them to define a research focus within a larger agenda that the teacher lays out.

Initially, when I started doing this work, the matches would be as small as one student to one volunteer scientist, and now they have grown. I did not do anything larger than five in a team until I got to Toronto. Then I started to play with larger and larger numbers. I have gotten up to ten with some satisfactory results, although it is a very sort of different interaction than you see in the e-mail. And the relationships here last about the length of one project or curriculum unit. They might continue on longer than that if students do another project and they want to work with the same mentor, but everything here is attached to the curriculum.

Orchestrating Telementoring (95/96)

• Projects designed to give students the freedom to define a research agenda, or refine a focus within one defined by the teacher
• Teams of 1-5 students were matched with a volunteer from academia, industry or government
• Relationships lasted at least the length of one project (4 weeks in the middle school, 7-10 weeks in the high school)
• Communication entirely via email

Some of the benefits of telementoring within this curricular context, in which students are trying to do, very often for the first time, some kind of empirical investigation. First of all, they get help focusing their inquiry; they get into a position to think. Often the most valuable thing that a mentor offers here is advice at getting toward manageable investigations, something that is not way over their heads.

They also get help finding and interpreting data. This is something that other sorts of services could accomplish equally well. But, I think here what is going on is that there is a nice intertwining of the advice-giving and inquiry-shaping functions with the information seeking. I think the synergy there is sometimes quite important.

Last of all, and this is harkening back to the theme I initiated earlier, improved feedback. These students are in a classroom environment in which there might be eight or ten quite different investigations taking place at one time. This is a situation that a teacher would find completely unmanageable normally. So, the teachers at work with me are the ones who feel that they have not been able to do justice to their students' curiosities. That students carry curiosities into their classrooms, and what they really want to do, if they can find a feasible way to do it, is to let these students over here study black holes, and those students over there study the extinction of the dinosaurs, and these students over here study something entirely different. This is why they do this; that is the motivator.

Now, the last point I would like to dwell on for a little while is the way in which these telementors can provide an authentic audience for students' ideas. It takes a little while to develop this idea. I mentioned science fair earlier. This, is some ways, is the most authentic experience of scientific investigation that many students ever have in their school careers. They produce a board like this. They will run an investigation, and they will normally carry it out at home with parents who might or might not be helpful to them. Then they bring the board into the school and present it to their peers and teachers. If they do not win that stage of the competition, it is over. The problem with this, when we think about deep scientific literacy would mean, is that after so many years of it, students are still writing things like this. essential lesson about what a methods section is like or what scientific argumentation is like. What I found in my dissertation research was that teams who better sustain their telementoring relationships with their mentors were more likely to do two things:

It is common argument strategy to simply make the strongest argument that you possibly can, the most simply stated claims. But that is not really what wins reward in the scientific domain. These are aspects of scientific literacy that are not easily taught, but I showed - and this is a small-scale study, mind you, in which the teacher had been pursuing telementoring for about 3 years at the time I began making these precise measurements. We did see, in this noisy social science data that we collect, some measurable effects. The careful analysis that I made on students written reports was confirmed by the judgments of volunteer scientists, whom I asked to review the same papers and judge them on three criteria:

• The quality of the paper as a whole,
• The quality of the argument presented, and
• The quality of the research.

One of these score, the one I call the IMRD score, correlated with all three of these. The important thing to point out here is that neither these argument quality scores, nor the holistic ratings that the scientists gave me, correlated significantly with student scores on a traditional content test, that they wrote in the same test. So, it was not the students whom you would expect otherwise to be successful in this class who are gaining the benefit from telementoring.

So, I have a measurable effect in one classroom where there is a well developed practice of telementoring. And, the effect is there for the students who stick it out with their mentors over the long haul. So, you might well ask me, "What about the students who don't stick it out?" I am very concerned about them, because I know from these results here and results from surveys that students complete after they finish each telementoring relationship that if students have a chance to have more than one - in one class, students had 3 telementoring relationships in the same year over three successive science projects - then the kinds of desires students have for their telementoring relationships, and the kinds of things that they can appreciate change with experience.

In the beginning, students are interested in "getting the goods" from their mentors; in this class is was getting the data. However, over time, they began to appreciate that it was important for your mentor to read your messages carefully. Later on still, they were able to see that it was really important for your mentor to ask you useful questions and give you feedback on your progress.

I did a lot of interview work with these students because I was interested in why students threw effort into these relationships and why they did not. I began to think about it this way, as a developmental problem because the degree to which you will have an interest in investing effort in a mentoring relationship is a function of the experience that you bring in. Here we are working with whole classes of students who have a great variety of experience that they bring in. If you have high expectations, as a result of your experience, for what a telementoring relationship might bring you, you will have high motivation to make your work and your thinking visible to your prospective mentor. That is what I call visibility. So, if you have high visibility, your mentor knows just where you are, which is great. If your mentor knows just where you are and where you are trying to go, then he or she can give you some useful advice. If the mentor does not know where you are, then he will give you poor or very little advice. What we face is this developmental Catch-22, in which students who do not have the experience to appreciate what a mentoring relationship might bring them have low expectations. They produce low visibility for their mentors. This, in turn, produces poor or little advice, and their worst suspicions are confirmed and they are caught in a downward spiral. I wanted to see how I could fix that. So, let us be clear about what the limitations are on what I might call telementoring behind the curtain, what I did for years, and may still do again.

The first problem is that every team and every mentoring relationship in this work that I did was an island unto itself. So, there was a lot of valuable experience in the class that was being had about the nature of mentoring relationships and what one might possibly get out of them and how one could do that. The people who most needed that, who most needed to be exposed to it, did not have it. I could walk into a class and interview five students who could tell me, "Oh, yeah. We had a great time. We learned a lot." I could talk to a few of their peers from the same class, who would say, "This thing stinks. Why do you guys do this? It obviously doesn't work." As a result, some participants never saw what a successful mentoring relationship would look like. This was also the case with mentors who were matched with unmotivated students. I began to be concerned about attrition, which is what everyone is concerned about in mentoring programs. We worry that even if someone gets a second chance to do a better job at this, he or she will not take it. We cannot afford that. We know that the number of volunteers out there is not large enough that we can afford to let people down that way. This is just a reiteration of the problem I just mentioned.

So, this is part of what motivated the work I have been doing in Toronto for the past two years with this piece of software called Knowledge Forum. I did not have any part in developing this software, nor do I profit from its sale. It is a commercial product, developed over the past many years by a team at the Ontario Institute for Studies and Education, now part of the University of Toronto. When I went to Toronto to work with Scartamalia and Bereiter and the team there, to see what I could do to bring telementoring into the practices that they have been developing with teachers over a very long time now. Essentially, Knowledge Forum is a multimedia knowledge base that students, teachers and the mentors build together. They begin with an empty database at the beginning of the year. It is completely void. As the students do their research, and it can be of a bewildering variety, they build this knowledge base together to represent what they think they know, what their best understandings are at the time. Those evolve a great deal over the course of the school year.

The important thing to remember about the software is that one can build compartments within these knowledge bases, which can be shared by teams. The compartments are nice for organizing the work of a large group of students and volunteers, but the compartments are accessible to anyone who chooses to look inside them, that is, anyone who has access granted to the database. Here is what a note looks like in one of these Knowledge Forum databases. This one happens to be a text note; notes can also have graphics in them. There are all kinds of nifty referencing features. That is what the little boxes are for. The notes go into this knowledge base. Each little blue box is a note. The notes can live in multiple views. It is a very, very richly structured medium. The views can have all sorts of graphical backdrops to help people organize and understand what they have done.

Now, what I saw happened spontaneously and almost immediately when we started organizing telementoring relationships in this medium is what I have come to characterize as an opportunistic model-seeking behavior. Each mentor and group of students had a view within this public database they were working in that was their own. Without any instruction from us, they began crossing back and forth between these views and seeing what their peers were up to and seeking models for their own behavior as mentors and as students.

Knowledge Forum

Problems - Is it true there is no pattern in blinking?

My theory Judd stated
The eyelids close every couple of seconds, this is called blinking. No person has a pattern for blinking.

My theory is that there is a pattern in blinking. To test my theory, I am going to do a study of five different people and how their eyes clear themselves through blinking.

Low/High Air Pressure - How an Airplane gets lift hector
Good Question arakua - Shape of wings hector
Wind teanlorg - Picture of wing shape teanlorg
Atmosphere signe - Picture mike
Sun's relation to earth alison
The higher you go up alison
Why teanlorg
Look at experiment #2 richard
Should include teanlorg
Picture Barometers mike
Suck or lift richard
Air Pressure pushes teanlorg
Can you be more specific? Duane
Hard to explain teanlorg

Telementoring in Knowledge Forum

How?

• Students and mentors joined research working groups, each of which had its own "view" within the public discourse space
• There were also common views for administrative purposes

Why?

• Team membership could be more flexible
• Publicity of mentoring dialogues helped to defeat the "developmental catch-22"

Opportunistic Model-Seeking

Mentor 1

Student 1      Student 2

Student 3

Mentor 2

Student 4       Student 5

Student 6

Model-seeking by mentors

So then I started...peeking in on some of the other discussions to see what level of assistance was going on and how harsh (a mentor) should be about certain things. Because you want to be encouraging but you also want to say (to students) you're really out of line there, way off in left field. And maybe you should think about this (laughs). Where are you going?

(Jodi, a volunteer mentor, in interview)

Factors underlying students' desires

Factor 1
(Prodding partner)

Offer challenges
Ask questions
Review work
Help interpret data and learning resources
Suggest strategies
Explain scientific ideas

Factor 2
(Inquiry jumpstart)

Pointers to internet resources
Background information
Readings
Help shape project
idea/question

Desires and reading habits Jumpstart Prodding Partner % Notes Read Overall .122 .320* % Own Mentor's Read .022 .294* % Other Mentor's Read .186 .269*

* p. .05
Students who read widely in the database were more likely to want a real inquiry partner than a jumpstart

Reading habits and student satisfaction Satisfaction % Notes Read Overall -.227* % Own Mentor's Read .005 % Other Mentor's Read -.232*

P. 05
Low satisfaction in mentor relationships was associated with reading a lot of other mentors' notes and a lot of notes throughout the database

Audience effects of telementoring

Teams who better sustained their telementoring relationships were more likely to:

Limitations of telementoring "behind the curtain"

Every team, and every mentor relationship, is an island

• Valuable experience is inaccessible to the students who could most benefit from it

Mentors can wind up "stuck" with unresponsive students, or vice-versa

• A waste of enthusiastic volunteers and students

Some participants never see what a successful mentoring relationship looks like

• Even if they get a second chance, many will not choose to take advantage of it

A telementoring relationship

Crisis!

Dear Bruce,
We tried to contact the lady that has all the information relating to lightning and forest fires, but she did not write us back. Our situation is now pretty brutal: we don't really have a specific question (because we can't find any data), our paper is due in a little over a week, and if this lady doesn't talk to us, we have no more leads to follow and we will have to start from scratch. If there is any way possible for you to think of another aspect of lightning to study and write about, we are in desperate need. We especially could use something that won't take a month to gather all the research on. Thanks for all your help and effort.

Good morning,
Did you look at the web site I mentioned in my last message? It contains the 1995 fire statistics, noting cause and state. That's about the best data you can get on-line, I suspect. If you can then find data on either the acres of forest in each state (should be relatively easy to find) or the population density in each state, you can look at some comparison of the % of fires started started by lightning and the population density, or the forested area. Taking the population density issue, you should be able to get state populations from either an atlas or the U.S. Census Bureau (I'm pretty sure they have a web site). State areas should also be easy to find. With this info, and the fire statistics, you could easily calculate 50 (x,y) pairs of coordinates where x and y are the population density and the % of fires started by lightning. (Which of these would be the dependent and which the independent variable?) Did you know about least-squares fit, or linear regression? Do you expect to find that % lightning fires increases as pop. density increases, or should %LF decrease when pop. density increases? Does a linear regression agree with what you expect?

Time Passes: The team commits to a research question

Glad you've got a topic, finally! I checked a couple of sites that I thought would list the acres of forest for the 50 states, but neither panned out. I have three options for you:
1. Call the Forest Inventory and Analysis office of the Forest Service at (202) 205-1343 and ask if the date is on-line. If it is, they should know.
2. Email me again with a fax number and I can fax you some pages that will list "acres protected" of forest and rangeland, which is a column that normally appears next to those on the lightning-data web page.
3. Do population density, not forested area. I suspect there'll be a stronger correlation, and the population of the U.S. states should be quite simple to find (look in a road atlas if you can't find the Census Bureau's web page).

Benefits of Telementoring

Help focusing inquiry

Help finding and interpreting data and other resources

Improved feedback

An authentic audience for students' ideas

Method: My problem with this topic was that all I found was the temperature and precipitation data. I sat at Netscape for hours just cruising through the information endlessly. I even tried Lycos and all of the other searching mechanisms in order to find the rate of photochemical smog. Nobody had it. This time period was quite frustrating. Finally, I posted on a newsgroup. For awhile, I did not hear anything, but finally a very nice person wrote me back. A man on the California Air Resources Board sent me quite a bit of information. As a result, I had to change my topic. I decided to try and find a correlation between the precipitation and temperature and ozone statistics between 1970 and 1979. That is when I could get down to business.

Advantages of "mentoring in the open"

Agendas for the future

One of the things about Knowledge Forum that is really beautiful for a researcher is that it keeps track of who reads every single individual note in the database. So, I started to look at the correspondence between students' desires for these prodding partner functions and jump-start functions and the percentage of notes of various kinds and by various authors that they had read through the course of this experience. I found that students who read widely in a database were more likely to want a real inquiry partner than they were to want a jump start. On the surveys I asked students to rate how satisfied they were with their mentoring relationships. I found that low satisfaction in the mentoring relationships was associated with a lot of other mentors notes, and reading a lot of notes throughout the database. Now, those are just correlational data. We have to unpack that. What does all that mean?

First of all, students whose assigned mentoring relationships were not working out took the opportunity to look elsewhere for guidance. Second of all, peeking in on classmates mentoring relationships taught some other students that their relationships had room for improvement. They became choosier. In either case, access to the mentoring relationships of classmates enabled students to overcome this developmental Catch-22.

We have talked about some of the things that mentors learn from their mentoring experiences, and if we believe in the nature of mentoring as a growth experience for people, then we believe that with more experience one will become more capable as a mentor. I have not met a mentor yet who did not want to become better at what he was doing, and who did not want each experience to serve him or her in that way. So, we can ask ourselves, "What are the developmental paths for adults who want to become the best telementors that they can be?" I have a little bit of data on this, of a similar kind to those prodding partner functions that I showed you. Mentors divide up those mentoring functions in a very different way. You would not be too surprised.

Another thing we need to be concerned about. . .What are the developmental paths for teachers learning to support telementoring? There is a great deal of very important work that teachers do to facilitate this work.

The last thing, which brings me to the project for which I have just received a little bit of seed funding to start a pilot of this year, is this: What would an ideal curricular context for telementoring look like? There are all kinds of ideas, a great variety of ideas as to how telementoring could serve the pursuit many different goals and many different curricular areas. I have an idea for a project that I am calling the Freedom Tales Network, which would operate in the area of history. I think this would be a setting in which we could design an umbrella under which people could come to meet the specified curriculum objectives, and at the same time have rich telementoring experiencing that would not appear in the least way forced in the context of their classroom activity.

With that, I will end. I will invite you all to write me. I do have a website at which you can download some publications. The website is www.netlearn.org.