What does quality mean in OER?

This session is intended to explore both technical and social dimensions to assessing, raising, and sharing "quality" metrics for OER. The session will be interactive and investigative, the better to capitalize on the expertise and interest in this issue manifest in the attendees to the OER meeting. Gary Lopez and Ahrash Bissell will provide some background and framing to start, and then will moderate the discussion.

Background

It is widely agreed that there is no de facto reason why OER cannot be of comparable quality to proprietary educational resources. Indeed, many in the OER movement would argue that OER are likely to be of higher quality, if not immediately, then through adaptation and improvement over time. The challenge for quality metrics for OER is two-fold. First, given that OER are, by definition, subject to derivation and re-mixing, how can we know whether any given OER has been properly evaluated or meets specific quality metrics? Even if a specific OER has passed certain quality-control criteria, how can we be sure that those criteria are associated with that exact version of the resource alone? Second, what are the metrics that we might use, and how can we obtain them? Given the global nature of the OER movement, we should expect that any metrics will be highly variable, though we would also be surprised if some shared criteria did not emerge. In addition, the situation is complicated by the fact that such metrics have to be provided and obtained in a manner that does not fundamentally undermine the openness of the OER.

We will consider the technical dimensions of "OER quality" separately from the social (evaluative) dimensions. On the technical side, we would like to gather and share information regarding needs and can also explore some of the new technical developments for web-based information which are providing possible (if partial) solutions to these issues. On the social side, we would like to begin gathering ideas regarding the types of metrics that would be of value for different types of OER, as well as insights for how such metrics might be produced and then applied across the OER landscape.

As with all sessions at this meeting, all of the data collected during this session will be available to everyone afterward, and we will be seeking interested partners for taking next steps towards implementing an OER-quality infrastructure.

Roberta - Quality is:

- accuracy

- age-appropriate

How do we actually vet resources to obtain these measures? Does not believe that stand-alone quality-control orgs can do this. Need to go to the community.

Flagged Revisions is a Wikimedia Foundation tech project which directly addresses the quality question. Flagged Revs has  been successfully deployed on the German Wikipedia and is currently being considered for other Wikipedias; it is one possible way to addresses the the concurrent need for accuracy and immediacy on-wik.

See http://meta.wikimedia.org/wiki/FlaggedRevs

David Wiley -

Quality is not an independent characteristic of OER. It is dependent on the actual person using the resource. In other words, "quality" should probably not be tied to the resource, but rather is a conjoined property of the resource and person.

Candice -

Would also add that the "quality" issue depends on the intended outcomes.

Andy -

"Technical quality" is another dimension...

Others -

Seems like context is the key. How do we match the specific context to the quality metrics that are provided?

Can we build standard query/ratings from students, who theoretically can provide the most menaingful context, and can themselves be contextualized?

Caution that we are not likely to generate a grand system of quality evaluation... perhaps specific, self-identified communities would do a better job, and of course would automatically be working within a particular context.

Can we capitalize on the searching and evaluative activities that people are already doing every day?

Connexions view - "lenses". Quality and ratings are a facet of specific communities of practice, so how we get those domains of expertise to track with the content? Can we imagine a system that allows the ratings in one system to travel to other systems?

EUN view - explicitly focused on the capacities of resources to "travel well". They need more info than a digest rating... so the social info related to the ratings is also requested and incorporated.

Video (open Yale courses) -

Feels that quality considerations flow first from the intended outcomes... what are the goals, and how will your materials get you there?

Gary comment - How do you match the product with the person actually using it? This gets into aspects of publishing practices, web optimization, etc.

Andy - Is quality a measure of product, or of process? Perhaps a focus on the product in use, as a measure of its success in a specific context, is a better way of thinking about getting valid quality metrics...?

Is it "quality" if it cannot be used (well) or does not improve learning/engagement/etc?

 

A comment specific to collaboratively created educational content: Community consensus is a key quality assessment process, tool, _and_ product.

Issue specific to OER...

How do you square the pre-production quality-control aspects with the fact that OER are, by definition, subject to adaptation by anyone... what can we do to manage evaluative processes for derivative works?

In theory, a focus on process, separate from the actual materials at hand, may absolve this problem, since all materials can only be honestly evaluated  in a specific context.

There is a companion issue of re-using content in a new context... does this make process-evaluation more or less sensible?

What does "quality" mean to a self-directed learner? Are these really different audiences? Should we assume that the quality measurements that would apply should be different?

There is a tension that we need to recognize that "quality" control, coupled with open licenses, doesn't really make a lot of sense... otherwise what is the logic of encouraging people to improve materials?

What responsibility do we have to all of the diversity of learners?

Kathi - We need a permanent versioning system, so that users can track back to see the "bona fide", or original, version. How do we build a system that makes it possible for people to know what exact versions exist and which have been evaluated?

David - Re-use across contexts has always been problematic... traditionally, all resources were somewhat acontextual, since otherwise it was hard to bang a learning environment into the correct shape. This race to the bottom (removal of all context) is bad. Does OER offer a solution? Yes, if the capacity for local innovation is there. In theory, we should be able to produce all resources as locally relevant, knowing full well that the resource MUST be adapted to meet a new context. This is far preferable to the prior situation, since materials can now be (legally) adapted to fit the needed context, rather than trying to alter the context to fit the materials. This still begs the question of how we then affix the different contexts to the resouyrces so that the quality-control measures are sensible.

It would seem from the conversation that there might be at least two ratings types in the OER quality discussion - might be termed "ding an sich" and "ding fur mich." The "an sich" issue could fit into the traditional acadmic model of peer review (out of context???). The other matter is more like a popularity count: how many people have accessed or downloaded or how the users rate it, etc.

The E in OER does make quality a context dependent charactersitic but also multi-dimensional i.e, technical (interoperability); extensability . i.e, modifiability; Educational  design; appropriateness for outcomes, etc.). This suggests:

-that proxies for quality such ast he  reputation of source  or the recommenders are useful proxies

- that we need a scorecard of quality indicators

- Most importantly  that Quality is not absolute but a reasonable indicator to help the selection of an intial set of OER's and incrsse the likelihood o its usefulness for different educational purposes and conexts

There is a significant "trust" factor we need to remember too. How do we retain that crucial feature of the evaluative process?

Important point that "trust" does not mean "local", necessarily. Relationships are much more complex than that.

What Does Quality Mean in OER? : A Conversation

Ahrash Bissell and Gary Lopez

How do we make quality useful to the users? What are the top issues in quality?

 

• Quality is about accuracy, and making material age appropriate.
• Quality is not an independent characteristic of an OER. Quality is a conjoint property between the OER and the user. Resources are not either quality or not. 
• The outcome is also important.
• Technical quality, the form in which the actual resource exists makes it more or less usefull for the person to engage is also a factor. 
• It's context,context, context! 
• Quality will change from different areas and locations depending on their needs and requirements.
• Ratings may also be related to context...where they are, who they are teaching, etc.
• Asking studetns how useful the information was. The students used the resources, and the data will capture who they are.
• So many factors influence one's preference. Blogs within certain genres around similar contexts might allow individuals to form communities around topics and thoughts. Students might be closer to the context of the educator.
• Quality and lenses are community specific. The challenge is to get the lenses to follow the content. How do we make that work as a bigger global OER?
• Ratings from individuals from different areas will inform about what is useful for a wider audience.
• Always keep your outcomes in mind and start your quality from that point. 
• Assuming we match the product with the person using it, how do we bring quality to those products.
• It's about process not product. How do you determine the quality of the product in use?  It is about determining what the quality processes are for the people engaging in use of the product.
• In terms of the educational experience, in post-publication there is more contextualization...with either the self-learner or the mediator (professor, tutor, etc.)
• How do we make it quality post-publication if it is outside of its initial context?
• Resources need to have some permanant versioning system to tie quality and ratings to a particular version. Then, there would be a way to compare changes between different versions.
• The quality trust factor is earned over many years.
• The trust factor is the difference between the global and the local.

So, on the "how do we do this...technically" dimension...one might think about the need for experiments and recommended practice in "recommender system" interoperability. That is being able to share the ratings and recommendations (that include the context, rating, and so on) across systems, that may be linked to individual versions, that may have additional provenance linkages, and so on.

...

From a whitepaper we have on a possible project...

"Recent years have shown an explosion in the amount of easily accessible education content.  While this content might be useful in a particular teaching or learning context, it is not easy to know which content is suitable, effective, and licensed appropriately.  Online communities and services have emerged to capture different authorities’ measures of quality and fitness for a particular purpose.  Some challenges remain, including:

•    Not everyone’s opinion is of equal interest to a particular potential consumer
•    Different recommendation services use different measures and expressions of suitability, effectiveness, rights, etc.
•    Various recommendation services use different technical channels to coordinate providers of recommendations, managers of recommendation data, and consumers of recommendations data."