The web-site is now in readonly mode. Login and registration are disabled. (28 June 2019)

What kind of research is necessary to demonstrate OER can help create better learning outcomes?

Cloud created by:

Kathy Nicholson
2 April 2010

Extra content

  • What metrics should we be measuring?
  • How do we gather impact data in an open environment?
  • Who are the audiences for this research?

Kathy Nicholson
19:47 on 4 April 2010

Dialogue Map created in Compendium (Hewlett Fndn funded tool)

Simon Buckingham Shum
02:53 on 11 April 2010 (Edited 03:00 on 11 April 2010)

Embedded Content


Richard Rowe
6:26pm 5 April 2010

The history of education reform is repleate with "faith-based" approaches about which there is very little, if any, evidence.  It is interesting to speculate why that is the case however the results have been repeated waxing and waining of different approaches to improving education with very little sustained and systemtic improvement. 

It is time for much greater emphasis upon "evidence-based" practice at all levels of education.  OER should not be exempted.  

That is not to say this is an easy task.  We can never measure directly what we are most interested in but must rely upon proxies that seem reasonable.  

And the variables that affect the outcomes we are most interested in (curiosity, knowledge, creativity, agency, persistence) are affected by many more things than those that occur in school such as nutrition, health, security, family community. 

To be most useful we should address Kathy Nicholson's three questions by first differentiating the level and kind of learning we are measuring.    Pre-school, elementary, secondary, tertiary and professional learning are each quite different, requiring different metrics, methods and audiences.  

The Open Learning Exchange is addressing these questions for the k-12 area.  While we find many similarities across regions and nations, the differences can be substantial. 



Patrick McAndrew
10:25am 9 April 2010

Hi Richard,

Great points - gathering the evidence about what works is tricky and as you rightly point out the impact of the factors we are looking at can only be part of the equation. And may well depend on the context and not be transferable. However the experiences of those who are gathered here, like yourself, do give an indication of how working with OER can be an agent for change and an opportunity for individuals. So as well as some hard evidence there seems to be a need to gather some collective wisdom about how OER can operate.

Or is it that OLE has found that there is too much difference to gain from pooling of "results"?

Richard Rowe
10:52am 9 April 2010

It is complicated. 

It seems almost impossible for us, as humans, to suspend judgment.  We "sense" effectiveness" or "ineffectiveness" almost without realizing it.  So it is worthwhile understanding the basis of those judgments.  

"Science" --systematically employing so-called empirical evidence -- is one method we have devised.  It can be quite effective under certain circumstances.  "Intuition" is another and it too, can be quite effective.  

Most efforts to gather "evidence" about the effectiveness of a social intervention, such as "education" find little or no effect from the intervention.   That can be because of someting aobut the intervention or something about the measures.  Either one, or both, can lead to the "no effect" finding.  

The intervention can be too weak, or irrelevant, lacking another dependent variable. or simply slow to show up.  Head Start evaluations are one example of the latter. It took years for the big effects of Head Start to show.  Early reports of little effect were wrong in the longer term. 

The measures can be insensitive, unrelated to the actual outcomes. or poor proxies of what we are really interested in.  The important things tend to be very difficult to measure and the things that are easy to measure are often not important. 

Notwithstanding all of this, relying primarily upon intuition and "feel" about what works, or should work, in helping people learn doesnt work very well. That has been our primary mode of assessment to date and little progress has been made.  The stakes are too high for us to give up on trying to develop more evidence-based practices.  The field of medicine is going well in moving in that direction   It is time for the field of education to follow their lead.

Richard Rowe


Giota Alevizou
8:58am 10 April 2010 (Edited 9:01am 10 April 2010)

Fascinating points and discussion. Richard, I am a researcher at Olent and Patrick’s colleague. I couldn’t agree more with these ideas: "And the variables that affect the outcomes we are most interested in (curiosity, knowledge, creativity, agency, persistence) are affected by many more things than those that occur in school such as nutrition, health, security, family community." It would be very interesting connecting variables and outcomes in specific contexts, and nuances of literacy (information, critical, media, multimodal) are also relevant here.

I think that we can only talk develop robust evidence-base only if researchers manage to contextualise several such dimensions, both cultural and demographic better. There are perhaps a few more ideas about context that we can think about mediated, open learning or social environments, the nature of Open Resources as social media and learning inscriptions, or intentionality that they may contain.  Some of us at Olnet, are very much interested in fleshing out, and gathering evidence on the formal – informal learning duo; informal learning occurs in media/resources/sites, that were not necessarily indented for education. So yes, definitely, notions/typologies of users, audiences, stakeholders, learners are relevant...

 Is it then a question of developing frameworks that are transferrable in other contexts? Or a common understanding of these dimensions that can inform transnational and comparative social research?

David Kernohan
1:41pm 10 April 2010

Writing this during the session - my point was misrepresented!

I was suggesting that we can measure the effects on learning of access to more material, of staff spending less time creating materials and more time with students, of students spending less on materials.

And we can easily measure the way OER effects these variables. So we can make a second order measurement of how OER improves learning, but we cannot make a first order measurement.

Patrick McAndrew
2:10pm 10 April 2010

@david - whoops sorry. I did understand the point that you were making was about learning - but it prompted me to recall how measurements can cover other parts of university sector.

Patrick McAndrew
2:16pm 10 April 2010

Feedback from this session - presented in Compendium. Issues raised - the metrics for Deeper Learning are a challenge and we should mix in the metrics for such things a use but need to think about the bigger aspects of learning and of change!

Richard Rowe
2:20pm 10 April 2010

Partick and Simon,  Can you give us the link to Compendium?  Great tool!    Richard

Karen Cropper
5:39pm 10 April 2010 (Edited 5:42pm 10 April 2010)

Simon, I cannot make the embedded Compendium map legible by zooming in either here or direct on the flickr, so I am guessing others won't either.  If you have chance, at some point, please could you add it in another format so that the information can be examined by people who were there (and of course those where were not too)  Thanks. Karen

Contribute to the discussion

Please log in to post a comment. Register here if you haven't signed up yet.