Thinking about assessment practices

Associate lecturers' National Conference,

Cloud created by:

Gráinne Conole
21 November 2009

This cloud has been set up to support the assessment session at the Open University associate lecturers' national conference on the 21st November 2009.

Questions

  • 1. Do we over assess? From a student perspective, from a tutor perspective?
  • 2. How do we build in variability? Alternatives to 3000-word essay
  • 3. Right ratio of TMAs to ECAs to exams?
  • 4. What is the role of self-assessment, formative & summative assessment? 
  • 5. How do we assess beyond the individual?
  • 6. Can we move to move standard guidelines for marking criteria/what are appropriate marking criteria
  • 7. Should we move to awards-based assessment? If so, how?

Extra content

Some definitions:

  • TMA Tutor Marked Assignment
  • ECA End of Course Assesssment
  • NSS National Student Survey
  • CETL Centre Exellence Teaching Leaning
  • S104 is one of our first year Science courses.

 

Gráinne Conole
12:30 on 21 November 2009

The following are some notes from the plenary general discussion, around the 7 questions posed ahead.

  1. Do we over assess? What is assessment for? To get the students to improve based on feedback – if they read it!!! Focus on the mark!! What would be the impact of delaying the time between giving feedback to students and the actual mark? Would this then result in  students having to engage with the feedback more? There is a role for more frequent assessment
  2. How to build in variability. There is already a lot of variability, not many of us teach courses where it is only 3000 word essay and where it is that it was the right assessment for that course. We felt overall the assessments were correct for the course. It’s up to the course teams role in terms of the way they design it.  Maybe we should incorporate ALs more in the design process? We felt that not all course teams do test the learning outcomes out enough… In terms of course design, it might be useful to consider designing course, which had a bank of learning objects from which individual tutors could then choose a selection. Alternatively the choice could be made by the students themselves, this would be potentially a way of giving students more choice and flexibility
  3. What are the roles of TMA? There are a range of issues about the TMA deadline and the impact this has on the students and their motivation. It’s important to recognise that TMAs and deadlines do motivate students and help to keep them on track with the course. Openings courses have a different model (they have two formative TMAs) and they work well for students at that level. There was some discussions about iCMAs and what they can do in place of TMAs. There was some discussion about the balance of lots of little assessment compared with fewer larger assessments and the pros and cons of each. Can we find more ways to give students more opportunities to go back and engage in reflective learning. For example what if the next TMA was around rewriting the essay from a previous TMA based on what you know now (and based on the feedback received), concentrating on what would make it better. Exams: general consensus was that we are fans of exams in maths and science. Worried about fear of exams and issues around hand writing. Balance of TMAs and ECAs/exams – you can get high marks on TMAs but can be slashed in a bad exam mark. There was a feeling that students tend to ignore study guides to concentrate on TMAs. On the main the group was in favour of projects. The Science course S104 has a lot of short iCMAs, which works well there but would it work in all subjects, for example where you might want to build up a bigger picture first. It was felt that P/G level is different and therefore the types of TMAs are likely to be different.
  4. Self assessment is good on the whole.  What about the role of peer assessment. Will students do formative assessment? Self-reflection elements in the assessment questions would be useful. One participant gave a nice example of a peer assessment activity, it is an online exercise students in pairs and then pass over and comment on each others’ work – helps develop their critical abilities. Also made them realise how difficult it is to mark and to not give negative comments
  5. Assessing beyond the individual: The group discussed the issues around trying to assess both the team work and the quality of the final product. We did feel that the assessment should include a self-assessment, stating what they had contributing.  The group discussed the difficulty of awarding individual makers around group tasks, how do you differentiate? There are also issues in terms of tutor workload for this in terms of facilitating the group and a new skills. If it is made a learning outcome around collaboration is this an issue.
  6. Marking criteria even with tight criteria there will be differences – it’s about professional attitudes. One idea was that at P/G student project negotiation can be very powerful. If extensive guidelines are given to students, do the tutors actually need any additional guidelines and isn’t there a merit in making the guidelines as transparent as possible anyway? Self-assessment at p/g level is possibility even more important, looking beyond the course
  7. Awards based – there is a case for overarching assessment for some qualifications – particularly in professionally-based/vocational courses. Did feel that each course should be assessed on its own merit. Should there be an overall examination type assessment, if contributing courses were only ECA based.

Gráinne Conole
15:52 on 21 November 2009

Indiviudal group summary notes:

  • Do we over assess? What's assessment for? Student improvment based on feedback (if they read it!); pacing students through the course, making them learn/
  • Could we do it by formative assessment? Will they do it?
  • How do we build in variability? Do we need it? That is something that is traditionally done by the course team not the ALs. How do we build in working collaboratively? Important to note that the 3000 word eassy is appropriate for some courses.
  • Assessment must be of the appropriate length and number for the tutor/student/course.
  • Assessing beyond the indiviudal - group tasks, wiki shows who's done which work, assessment should include self-assessment by each student of how they've contributed. Importance of knowing who has done what. Need to be able to assess both the teamworking and the quality of the final product.
  • Marking criteria - even with tighte criteria there'll still be differences between markers.
  • Students often just look at the mark they received and don't read the comments. What can we do to encourage them to take more note of these?
  • Awards-based assessment - each course should be assessed on its own, there may be a case for overarching assessment for some qualifications. Should there be an overall exam if the courses are all assessed by ECA?

Gráinne Conole
15:59 on 21 November 2009

Group discussion notes

– Do we over assess?

a) from a student workload perspective?

  • We can over assess for example a 60 point course might have 7 TMAs, a number of iCMAs, a collaborative project and an end of course assessment.
  • The purpose of assessment is to get students to improve based on the feedback they receive. But it was felt students don’t engage with feedback as fully as they might.

2 – Given the range of learning outcomes that we are addressing, the pedagogies we are using and the learning activities that students undertake how do we build in appropriate variability in the types of assessment? What are the alternatives to the 3000 word essay and how should we be using them?

  • There is variability – if it was on the course then right assessment. Part of design of the course. ALs should be used to input to course design especially around assessment.
  • Course design – provide more learning objects – to offer more student choice.

3- Have we the right ratio of TMAs to ECAs to exams? What would be the arguments for increasing or decreasing the current ratios?

  • The only motivation for students is TMA deadline, self or formative assessment wouldn’t provide this motivation. However on Openings courses the two TMAs are formative (students have to submit one to enable them to take the ECA). And they have no fixed deadline. So tailoring assessment to level and skill of student is important. MCT use early assessment to encourage students to get to engage with the course material. Teaching is in lock step with assessment so opportunities to change assessment are limited. There could be an adverse effect on retention if number of TMAs were reduced. CMAs have some similar properties in engagement, pacing, providing feedback so we don’t need to rely solely on TMAs. There are differences between subject areas – eg iCMAs might work better in Science.  If students do not submit a TMA it is often too late to pick up the problem. Looking at online engagement to pick up student progress might be useful. There is a balance  to be made between lots of little assessment points vs fewer larger assessment points. How students study (eg cramming method) plays into the design and development of effective assessment strategies. The group felt that students are unwilling to read the assessment guidelines. Students want to submit drafts – should we facilitate this and if so how? Assignments where students have the opportunity to engage in reflective leaning would be valuable, as evidenced in AA100.
  • There was acceptance of the value of exams in maths as a defence against plagiarism/ false identity. Some courses require an exam as they are part of externally accredited degree. Fear of exams is strong: handwriting is a problem, time pressure, poor memory, the surprise question, luck of the draw what I have revised, fear of/ history of failure. Providing opportunities to engage in mock exams (eg in tutorials) may reduce fear. Design exams to overcome these problems eg open book. ECAs and exams test different skills. It was noted that GCSEs and  A levels are going back to exams.

5 – Should we move towards award based assessment, and if so how?

  • For some qualifications it would be useful to check achievement. Each course is currently assessed on its own. There should be an overall exam if all courses in award had ECAs.

4- What is role of self assessment, peer assessment ; formative vs summative assessment

  • For science, maths, computing iCMAs can be useful. They can be useful in other disciplines such as social work if students need to understand factual data.
  • iCMAs can be fun.
  • There was a general perception that students don’t want to know about formative assessment. Though openings courses provide an exception to this.
  • Adding self reflection elements in assessment questions would be useful, but we need to convince students of the value of this.
  • Peer assessment with students commenting on others work provides a good exercise and develops students’ critical abilities. It also shows students how difficult it is to mark and come up with positive comments.

5 – how do you assess team-based activity/ beyond the individual?

  • How do you award an individual mark based on taking part – is it all or nothing? What does assessing the quality of an individual’s contribution mean? There are difficulties in putting collaborative work in the first assignment eg with delays in some students getting equipment (esp FAF and LEQ). If teamwork is requires who puts students into teams and on what basis? It was felt that students were reluctant  to engage in group work. For students in prisons there were additional difficulties in collaborative activities though some examples were given of how this works. There was some concern about the amount of tutor time taken to make it group work a success. Course teams need to consider how hard it is to run/ mark group work. Including group work was considered a challenge at level 1. Students were concerned about the perception of the level of ICT skills required. The solution might be to introduce an introductory ICT course, this already existed as an extension to an openings course. There is a bootstrap problem with ICT skills if we expect students to go online (eg through openlearn) to pick up these skills.
  • Self assessment could be included on a student’s contribution to the group.

6 – Can we move to standard guidelines for marking criteria/ what are appropriate marking criteria

  • At PG level student project negotiation is important. Self assessment at PG level is more important especially as students will be expected to look outside the course material.

Peter Wilson
16:40 on 21 November 2009 (Edited 08:49 on 22 November 2009)

Group discussiion notes

  • Idea of having a student project negotiation
  • What is the purpose of the exam?
  • Balance between TMA and ECA/Exams
  • In theory low TMA grades could give low overall course grade but other way round is much more likely
  • Need to ensure that the TMAs are right to cover the course material
  • Comparision of projects to exams to check on the succe of the understanding of the course
  • Need exam questions that make students think rather than memorise
  • Often there is 2 months between TMAs - is this too much of a gap? Could more frequent iCMAs be the answer? Also can help tutos to check student progress
  • S104 has 9iCMAs 7 TMAs and and ECA - feedback from the students is positive
  • Issue is that if it is summative then students dont get immediate feedback
  • What about the possibility of have some form of substitution assignments where there is alot of assessment to help in terms of alleviating the pressure many students face in terms of keeping up with the course and doing regular study
  • Many subject areas use drip feeding to assist their increasing knowledge and undestanding of the course
  • At p/g level students are looking at themselves often and working with othes to acheive success
  • In the Business school use of self-assessment and finding materials online

Gráinne Conole
18:18 on 21 November 2009

Embedded Content

Assessment notes

Assessment notes

added by Gráinne Conole

Assessment presentation

Assessment presentation

added by Gráinne Conole

Plenary summary

Plenary summary

added by Gráinne Conole

Contribute

Gráinne Conole
10:15am 21 November 2009


Feel free to use this space to discuss the questions posed or any other issues related to assessment. I have included links to the pedagogy profile, the REAP website (which is an excellent set of resources for assessment) and to the design tips assessment cloud.

Gráinne Conole
10:17am 21 November 2009


If you are new to cloudworks, have a look at this short 3 minute video about the site or here for further help and advice.

Michelle A. Hoyle
1:00pm 21 November 2009 (Edited 6:40pm 21 November 2009)


What I find interesting is that modular, self-contained courses are the norm where I come from (Canada).  So I don't think that in itself is a problem necessarily.

My undergraduate degree required 120 credit hours generally done across 4 years, at 3 credit points per course (University of Regina, 2009a).  So that was at least 40 courses.  Granted, some of those aren't completely self-contained.  For example, you are not allowed to take the course on integral calculus before you've completed the differential calculus course.  Regardless, each course had at least one midterm (most had two), a final exam or project, and a variable number of assignments.  My experience at two other Canadian universities was similar.

I think what was more obvious, however, was how you progressed through the degree.  A certain number of the courses were required and then you had some free choice within your major area (like computer science), plus some free choices within the schiool (science for me, as that was where computer science belonged), plus some free choice in arts and languages to round you out.  What was different was the number of possible degrees/pathways.  I think there were far fewer.  Basically, you were either doing a B.Sc. in computer science, a combined maths/computer science, or software system development.  3 basic choices (plus honours variants of those, which consisted of some additional requirements in place of free choices in computer science).  There was also a 2-year certificate available (University of Regina, 2009b).

Contrast that with the Open University.  I counted 10 bachelor degrees, 1 foundation degree (whatever that is!), 2 dipliomas, and 2 certificates (Open University, 2009) for computing.  One of the degrees was an "Open Degree", a kind of "choose your own path" degree. What we have here is too much choice!  That's probably why associate lecturers get so many questions from students about what courses they should do next and why it's so hard to advise.  When there's too much choice, you get uncertainty, second-guessing, or even an inability to advance.  Barry Schwartz (2004; 2006) has done some research on how too much choice can be a bad thing.

Are we over-assessing?  Maybe compared to other universities in the UK that have integrated programs.  Are we over-assessing when compared to other higher education institutions with modular degrees?  I don't think so.  That's not to say all modular institutions aren't over-assessing, but that we're not over-assessing when comparing like degree methodologies within the same area.  I'll address assessment issues in a separate comment.

 

 

References

Gráinne Conole
4:03pm 21 November 2009


HI Michelle thanks so much for contributing and for adding these links. You raise a host of very important issues. I think there are distinctions between the OU courses and ones elsewhere. The issue of progression as you say is critical. We talked alot today in our session about the relationship between assessment and student motivation. Also on how we can encourage more self-refective approaches so that assessment helps in terms of students taking their understanding forward.

Glenys
3:29am 22 November 2009


Hello Grainne,

Our school is currently grappling with assessment.  The comment in the thread about giving students the comments before they get their grade is an interesting idea.  I wonder if this would enhance the consideration of the comments, students are so conditioned to look at their mark and once they know that the rest does not matter. 

Many teacher comments are wasted if we do not provide the opportunity for students to improve their work.

Interesting ideas to consider,

Glenys

Gráinne Conole
8:46am 22 November 2009


Hi Gienys

your comments resonate very much with what we were talking about yesterday, i.e. that often students don't take enough notice of the very rich feedback they get. There is a tendency for them to focus on the mark rather than really internalise the feedback and self-reflect on how this can help them improve. Given this what else might we do to improve feedback? Giving feedback first and then the actual mark later is one idea. Another is to give the feedback as a podcast, so the student has to listen to the comments first. We also talked yesterday about how peer assessment can be very valuable, not least because it makes students realise how difficult it is to give feedback and to get the right 'tone' in terms of practical advice and encouragement rather than focusing on the negative aspects of the student's work!

Glenys
3:09pm 22 November 2009


Hi Grainne,

I am coming from a Grade six perspective but I think my comments are important in all levels.  First I think that our feedback can be overwhelming.  If I wrote a paper for Graduate studies and if I received comments as if it was a peer-reviewed document for publication, I think I would be overwhelmed.  We as educators need to nudge our learners to the next step not to our level of perfection.  I think rubrics have the ability to do this, as do peer and self evaluation.  I use all with my grade six students.  They need guidance on moving to the next level on the rubric.  They also need to be guided on what would the categories in peer and self assessment would look likes  at the various levels. These young students are very perceptive and maybe 5% need to defend their selections with me.  What I have discovered with those 5% is that they really can not differentiate what makes something better than something else.  They lack the ability to judge. If they can not do this, then we truly have wasted comments on these students, since they will not know how to improve based on them.  There are always a few students, when we do this the first time, that choose to give high marks all around.  If I ask these students if they were honest in their assessment, they smile and say no, and ask for it back so they can redo it.  I know they were just testing the process of self/peer evaluation.

Glenys

Gráinne Conole
3:47pm 22 November 2009


Hi Glenys

I totally agree with you - I think these are issues across the board and not contained to one educational sector. I also agree that we need to motivate not demotivate students - to my mind the purpose of feedback is to support them and to help them develop their understanding. I also agree that peer and self-evaluation can be very powerful and we perhaps don't use them enough. In our discussion yesterday we talking about how peer assessment can help learners realise how difficult giving good feedback is! And of course we all know the old saying 'if you want to understand  something teach it' maybe we need to go a step further 'if you want to really really understnad somethiing assess it'. ;-)

Derek Jones
8:45pm 15 June 2010


Should we move to awards-based assessment? If so, how?

One of the saddest pieces of research I have read recently was the felt tip pens study (I know it's not called that - it's just the image that comes into my head - real reference attached up top).

Incentivising learning through externalised reward does not work (unless you are teaching manual skills or want to foster individuals who are failure averse and who will in turn be far less likely to try new things in case of failure). Pretty animation and the obligatory TED talk.

In fact, my impression is that the OU actively encourages alternative motivations for (self) assessment  in my experience (as student and tutor).

On U101, our assessments are extremely varied in content, scope and media (it's fair to say we are lucky in this respect due to the nature of the course). We use text, maps, sketching, photography, diaries, etc. and interwoven through all of this is the constant of self evaluation (which has really been approached well by students so far).

The number of times I have said "Your score is a number - it doesn't tell anyone what you have learned, how many late nights you had, or what you would do differently the next time you were to come up against an apparently impossible problem. "

I wish I could find the reference to the MIT (or was it Harvard Business School) experiment (1930's?) with removing grades completely and relying entirely on feedback / self assessment. In fact I think there is a Level 3 OU course trying this some time soon? 

PS - we need forum type cascading for some of these discussions :) 

Contribute to the discussion

Please log in to post a comment. Register here if you haven't signed up yet.