Analysing Tutor Feedback for e-Assessment Systems
Cloud created by:
8 April 2011
Presentation by Denise Whitelock as part of the learning analytics sympsium at CAL 2011.
One of the challenges of today’s education is that students expect better feedback more frequently and more quickly. Unfortunately, resource pressures are high and feedback is often produced late or under time pressure. This is an unsatisfactory situation for the students who complete the UK National Students Survey and clamour for more timely feedback. Lack of tutor response is affecting university ratings with respect to this survey and producing prompt feedback is on Vice-Chancellors’ agendas in order to improve universities’ rankings. However, does just-in-time feedback improve student learning? Will not the quality of feedback play a role here?
In order to answer these questions, the written feedback given to students at The Open University by their tutors in three different subject domains was analysed and then modelled in order to provide a tool that would support tutor reflection on feedback. The first part of the paper will discuss that analysis and how it influenced the development of ‘Open Mentor’, an open-source tool that guides tutors into giving appropriate feedback.
The second part of the paper will present findings from the analysis of tutor feedback in the History domain and illustrate how it shaped the production of an electronic formative assessment tool for History students, ‘Open Comment’, which provides an automated feedback response.
Open Mentor: The classification system used for Open Mentor was based on that of Bales (1950). Bales’ system was originally devised to study social interaction, especially in collaborating teams; its strength is that it brings out the socio-emotive aspects of dialogue as well as the domain level. Bales’ model provides four main categories of interaction: positive reactions, negative reactions, questions, and answers. These interactional categories illustrate the balance of socio-emotional comments that support the student. We found that tutors use different types of question in different ways, both to stimulate reflection, and to point out problem areas in a supportive way. These results showed that about half of Bales’ interaction categories strongly correlated with grade of assessment in different ways, while others were rarely used in feedback. This evidence of systematic connections between different types of tutor comment and level of attainment in assessment was the basis for the design of Open Mentor.
The classes of Bales model are domain-independent – it proved successful in classifying feedback in a range of different academic disciplines An automatic classification system, therefore, can be used in all fields, without needing a new set of example comments and training for each different discipline.
Open Comment: The second section of the paper reports on the feedback engine and pedagogical principles that drove the development of Open Mentor. The pedagogical rationale for this system was to engage students in a series of electronic formative assessment tasks that would provide more free-text entry but also provide automatic feedback. It built on earlier work on Open Mentor (Whitelock & Watt, 2007).
The guidance text arose from the analysis of expert tutor feedback in several Arts disciplines. A range of techniques was employed to elicit the processes they used to provide appropriate feedback. These ranged from role-play (as a student) to analysing collections of real answers and constructing sample solutions.
A preliminary analysis of 68 History assignments together with 100+ assignments from different disciplines revealed a common pattern of tutor responses. These were clustered around the main categories of praise, advice on structure and presentation, particular misunderstandings, and developing and understanding particular issues.
With the continued pressure on staff resources, making individual learning work is always going to be a challenge. However it is achievable as long as we maintain our empathy with the learner. Tools can help us achieve this through providing frameworks that support reflection on our social interaction in order to provide emotional support as well as the conceptual guidance that learners need.
Bales, R.F. (1950). A set of categories for the analysis of small group interaction. American Sociological Review, 15, 257-63.
Whitelock, D. & Watt, S. (2007). E-Assessment: How can we support tutors with their marking of electronically submitted assignments? Ad-Lib Journal for Continuing Liberal Adult Education. Issue 32, March 2007. ISSN 1361-6323.
Liveblogged by Gill Clough
Denise starts by making the point that tutors want to give supportive feedback, so how can we help them do this.
Open Mentor - born of pedagogical models. Open source mentoring tool. As tutor marks a script, this strips out all the comments and feeds them back to the tutor. OM looks to see if the tutor has given enough support for the mark awarded. Uses Bales categories.
- Positive reactions
- Attempted answers
- Negative reactions
- Teaching comments
OM identified a trend which held true for business school, arts, humanities. Initially surprising, but then when you consider that comments based on good pedagogy, then less surprising.
As a tutor - the feedback you get is in a form of a bar chart. Ideal comments are in red, tutor comments in blue. Can detect if say you have made too many negative comments and not enough teaching comments so tutor can understand what is going on and give better feedback to students.
Open Comment - tries to put OM into practice. Immediate feedback. Detects errors, looks for omissions. Requets clarification, inferencing and checks causality. This model allows you to give feedback to enable the student to move on. Trying to identify what is not there, what is not correct in order that the student can improve.
What have they learnt?
Analysed tutor feedback from 600 plus TMAs. Have produced models of feedback that is open to test. Emphasis on the role of scaffolding in learning so that we can come to support the ideals that the student's union want.
Member of the audience reports having a slightly sick feeling upon seeing the heart taken out of something. Questions where the knowledge is. Is it in the text made by students or teachers or is it made by people. When talking to a uni lecturer, you get fascinating insights. Student may not agree with teacher, but is gaining a more vicarious engagement and experience from that interaction. He feels that it is interesting but that there is something very cold about it.
Denise responds that when you see the comments they gather from the tutors, they are very warm. They are a form of dialogue, and you need to open that dialogue which is warm and encouraging where the tutor has a good voice.
Questionner says that good feedback can be cold or harsh.
Denise: When at a distance that can be problematic.
2nd questionner: feels that this programme suggests that there is a best model for giving feedback but that every student is different. He knows that one student requires him to be critical of their work because they react well to that. He knows another student who needs it to be tempered and be alot more supportive and he therefore challenges the idea that there is a best model of feedback for all students.
Denise: Sees where he is coming from. When you're more critical, you're helping student to move forward. With another student who doesn't like that approach, you are still conveying the same information in a different way. The system doesn't negate what he's just said. If you have a student who isn't doing well, he needs comments to help move forward and improve. You need to explain what you want the student to do. To help the student to move forward.
Questionner 2: Concerned that senior management would see software like this and immediately require every tutor to mark to that model. As a tutor, you have more knowledge than this of individual students to ensure that they get the best out of the feedback. He does verbal feedback and he can't see how this would fit into that model.
Questionner 1: If you see a school report you will see what they're talking about. The comments are totally meaningless.
Questionner 3: Does this have a use for students to give peer to peer feedback.
Denise: Haven't tried that yet. They've used it for tutors in the OU thus far.
Questionner 3: But it could be used with students to develop.
Denise: yes it could.
13:21 on 14 April 2011 (Edited 13:32 on 14 April 2011)