So important to have sound evaluation processes when prototyping
Cloud created by:
13 February 2013
I was involved in the evaluation of prototypes for a national multimedia development of online information literacy resources. The evaluation reports of the different phases (including the evaluation survey questions and observation questions and expert review material) that we used can be accessed via the link. We went through several stages.
1. Storyboard of prototype - We were careful to storyboard the original concepts for each of the 12 online modules. This was done after discussions with the subject experts about what they thought should be in the modules. We did this using paper BEFORE any multimedia production began. Several people were asked to comment on the design - subject experts and multimedia experts as well as people involved in the project.
2. Usability testing - this was carried out on the prototypes in their multimedia format. This involved observations of students and academic staff using the prototypes.
3. Formative evaluation of the prototypes - this involved evaluating the navigation, instructional design, effectiveness for learning, content and activities etc. Students, subject experts and academic staff from various disciplines were asked to use a checklist with likert-type scales to rate the modules.
4. Expert review - this occurred early before the formative evaluation phase. Two experts, one technical and the other a content expert used heuristic evaluation questions to critique samples of the resources.
The process took time but it was well planned and reported comprehensively, and this gave the project team confidence that the modules were of high quality when they were released. Educators from around the world have accessed and used and some have modified these modules for use in their organisations.
I am now involved ina development of a new tertiary teaching qualification. This is going online to an open global market, and yet the prototyping phase has been bypassed and the evaluation process totally ignored until now. Time is cited as the prohibitive factor but this is precisely why evaluation needs to be carefully planned so that it is integrated from beginning to end. From my perspective and also in the view of expert evaluators - prototyping and usability testing, in fact most types of evaluation are ignored. It is at the peril of an organisation to do this I believe because it actually costs money and time when you have to redo everything later on because it is poorly designed.
Why is everyone in such a rush to get the product out there and in doing so rush or ignore probably what is the most important aspect of any design project? I am interested in others' experiences of this.