Crosby's evaluation experiences

Cloud created by:

Tiffany Crosby
22 February 2013

I've used per reviews and expert reviews quite extensively in work that I've done. I may also do some beta testing if the learning design lends itself to it. When I use this approach, I often do so through pilot workshops. This also allows me to test learner reactions to different features. A series of pilots allows continued improvement while speeding time to market. With all this said, I haven't traditionally operated with a formal written evaluation plan. I do think through the types of information / decision that I would like feedback on but I haven't then taken that next step to formulate a rigorous evaluation plan to obtain that feedback. So I'm looking forward to working through an evaluation plan this week as I continue to build out the prototype.

Extra content

More information on the peer review process that I used: 1. Identifying reviewers - I used my network to identify individuals with sufficient content knowledge that they could question the content but that also were business practitioners. It was important to me that they were not in an academic setting because I wanted to make sure that they've had to apply the concepts being taught. I also tried to get coverage generationally to see how differently the training was perceived. The last factor was years of supervisory experience. 2. Establishing guidelines. Each reviewer was provided a checklist that I reviewed with them prior to their review. The checklist wasn't extensive. It focused on these main points ( in order of importance) . Given the planned content and instructional plan, would the defined learned outcome being achieved. . Would a learner that completed this course walk away with at least 1-2 action items that they could start to do immediately, 1-2 that they could act upon in the short term, and 3-5 medium and long-term. In essence, could they construct a comprehensive development plan that continued to enhance their learning and improve their skills. . Was the content interactive enough within the constraints imposed by the certification standard setting body or hosting environment (these constraints were provided) . What elements of the design appealed to them . What elements of the design didn't they like . What would they like to see more of (content wise) . What would they like to see less of . Are their other features that should be included or that they think really enhance learning based on their other experiences 3. Evaluating feedback - if there were easy changes, I made them. If they were going to be more extensive then I did a cost-benefit That was about it. There was lots of opportunity for free form feedback

Tiffany Crosby
14:48 on 22 February 2013

Embedded Content


Yishay Mor
10:11am 22 February 2013 (Edited 1:16pm 22 February 2013)

Hi Tiffany,

Can you tell us about one peer / expert review experience in detail? What were you evaluating? How did you choose the evaluators? What instructions did you give them? What where the results? How did this effect the final product? Thanks, Yishay

Tom Reeves
12:46pm 22 February 2013

Hi Tiffany, Thank you for sharing your evaluation experience. I think most of us do our evaluations informally in the manner you describe. Planning an evaluation ahead of time, especially figuring out the decisions the evaluation should inform, the questions that need to be answered to provide the basis for better decision- making, and then coming up with the appropriate evaluation methods for answering the questions takes time and effort that few of us have in most everyday learning design. But when the stakes are high, it is definitely a worthwhile investment. I look forward to seeing your planning efforts as they move forward. - Tom Reeves

Kelly Edmonds
11:10pm 28 February 2013

Good point on conductng reviews informally and mentally investigating and collecting feedback. However, for those of us trained in formal research methodology we know the pitfalls of not collecting enough or the right data. A little knowledge could be quite dangerous.

Tiffany Crosby
11:31am 1 March 2013

Kelly Edmonds, There is certainly a time and place for using formal research methods but they also have limitations. Subjecting an innovative idea to formal research methods doesn't always work. In fact, many of the tools that we now rely upon would never have been developed if that approach was used. I don't believe that learning formal research methods is difficult. To me, the far harder skill is knowing when to use them and when not to use them. That takes wisdom and vision and no text book can teach that.

Contribute to the discussion

Please log in to post a comment. Register here if you haven't signed up yet.