The web-site is now in readonly mode. Login and registration are disabled. (28 June 2019)

Learning Analytics Expert Workshop (Amsterdam)

Cloud created by:

Rebecca Ferguson
11 March 2016

Day 1 – Tuesday 15 March 2016

Networking coffee is available from 10:00 onwards

10:30        Registration, coffee and Ice-breaker session

11:00        Setting the policy context: Riina Vuorikari, JRC-IPTS 

11:05        Invited Expert introductions (1 slide, 4 minutes each)

12:30        Lunch

13:30        Introduction: LAEP aims, research questions and findings so far: Rebecca Ferguson and Bart Rienties, The Open University, UK

14:00        Learning analytics projects/tools: lightning presentations on European-funded projects

14:30        Introduction to State-of-the-Art Exercise: Identifying key learning analytics tools, policies and practices

15:00        Coffee and discussion

15:20        State-of-the-Art Exercise: Identifying key learning analytics tools, policies and practices

16:20        Plenary session: key tools, policies and practicesIncluding validation of State-of-the-Art Exercise

17:00        End of formal sessions, Day 1

 

Day 2 – Wednesday 16 March 2016

09:00        Arrival, coffee and Day 2 Ice-breaker session

09:30        Visions of the future of learning analytics  LACE – Learning Analytics Community Exchange

10:00        Learning Analytics Planning Exercise: How participants see their work in this area developing

10.45        Coffee break

11:00        Introduction to Foresight Exercise:  Ways of thinking about and planning for the future

12:30        Lunch

13:30        Foresight Exercise: Using learner and educator profiles to envisage the future

15:15        Coffee break

15:30        Validation of Foresight Exercise

16:00        Plenary session: Potential for European policy related to learning analytics

17:00        End of workshop

 

                   

 

Extra content

Amsterdam live blog: Visions of the Future session

Presentation by Doug Clow (The Open University, UK)

Slides at http://www.slideshare.net/dougclow/laep-visions-of-the-future-of-learning-analytics

Visions of the future of learning analytics, a link to that document is available on the LACE website.

http://www.laceproject.eu/visions-of-the-future-of-learning-analytics/

A long document. This is the distilled headline version.

From yesterday’s conversation you have the same sort of ideas. We will talk today about taking it further and the policy implications.

A lot of work about understanding what the future might be. This year’s Horizon Report has learning analytics in the ‘one year to happening’ category. The OU produces ‘Innovating Pedagogy’ reports and they also have learning analytics as an important element.

http://www.open.ac.uk/blogs/innovating/

http://www.nmc.org/publication/nmc-horizon-report-2016-higher-education-edition/

Our exercise today is about looking at different sorts of future and what the implications are for policy.

The LACE study used a Policy Delphi methodology, which provides a menu of options about what might happen. Which of the potential futures do we see as desirable and feasible? What should happen to support the good and prevent the bad?

We drew up eight plausible futures, iteratively within consortium and contacts. Then carried out online survey, with over 130 people participating. About half of those were invited experts, the other half came from calls at invited events.

Also, at open events, we canvased people’s opinions of the visions, for example at BETT.

The Visions

Most of you have these ideas in your head already

1. The phycisal environment. Lots of us have sensors with us. Phones these days include tools like a barometer. Sensors are cheap and widely available. You can use webcams to monitor gaze. We can project how these might be used extensively in education.

2. Personal data tracking and quantified self and biochemical and individualized data tracking.

3. Takes things in the opposite direction. Privacy, ethics, people overstep the mark and then step back. Then we might move away from analytics

4. Learners having control of their information and data. They grant permission but control remains with them.

5. Open learning analytics, standardization and interoperability all come together and works. Everything many of us have been working on.

6. Analytics become much more essential tools. Extrapolating that predictive modeling will become more successful and widely used.

7. Decision making gets better than humans

8. We are so good at developing fine-grained analytics to computers that we can delegate much of the teaching to computers and we can move towards a dream of personalized learning.

Asked people to rate for desirability and feasibility. People broadly positive except for the scenario where we don’t use analytics. Also seen as broadly feasible.

From the free-text responses. Great potential for learning analytics, but not a confident sense that that potential will be realized. Not a strong consensus. We are not sure whether that will happen.

Need for policies and infrastructure. Arguments ranged from UN-policed approach, right down to the smaller classroom, group-related context. Privacy and ethics and standards came up again and again.

Large, bold and centre is pedagogy. Learning analytics needs to be focused on the teaching, in support of what we want from our education system. Need to capture what we value in education. Must take these into account when we optimize towards measures.

Lots about ownership, power and ethics. Who is control. Whether analytics are done to you or whether they support you. How they support and challenge existing power structures. I f we have information that can help learners we have a responsibility to use it to do that.

Looked across schools, universities, workplaces and informal settings. We did find differences but these were often smaller than we expected. Still ethics and pedagogy.

Little sense that the technology was going to be a barrier. Sometimes said we can do this already. Major barriers are social, structure and policy.

AlphaGo has just beaten Lee Sedol at the game of Go – beating the world’s best human 4-1 at the moment. Our capabilities in machine learning are moving forward very rapidly. Unlikely to be the main barriers to whether the visions of learning analytics are achieved or not

Thanks to everyone involved, and to the commission as our funders.

Do these visions make sense to those in the room? Is that our collective view of how things are?

 

Questions

Jeroen: I Like the stress on pedagogy. How do learning analytics change the pedagogy, this goes both ways?

Doug: Yes, this emerged. Teaching practices can and must change.

Alan: This is a European focus?

Doug: We went for a global perspective. The brief was to look globally. Fewer responses from the US that were concerned with privacy and ethics. Lot of European respondents.

Alan: Teaching and practices here are better but technology tends to be better in America. Dislocation between continents

Yves: Like the visions. What sorts of problems do they solve? Dropouts, inequality, access?

Doug: Didn’t ask about that. Covered most possibilities to cover a breadth of things. Weren’t looking at what should happen.

Bart: That’s what we shall do today

Daniel: What about adoption? Business tracks employees, risk of misbehaving. Every part of our life is tied up with analytics and intelligence, but we have a gut response when it comes to education

Doug: Yes, this is happening in a lot of other areas, though some people did argue that education is and should be different and separate from commercial analytics. Degree to which people feel it is choice and they are getting something of value back. Shouldn’t have to be tracked to get an education

Ed: Final destination makes quite a big difference. Each framework looks different if analytics are used in complex or simple way

Doug: yes, need to get more fine-grained

Speaker: Does teacher have a right? Mediating through a technology. Doing the same things as before, but just mediating it through technology. Ethical issues. No one should be able to steal the data. It is not only the data you collect, it is also the context and the use. I can watch students in my classroom already.

Andrew: Yes, you can do it now in class, but if you used it to humiliate students that would be over the line.

 

Rebecca Ferguson
09:12 on 16 March 2016

Amsterdam live blog: Group discussion

Group discussion on what will happen in the next years with learning analytics - seven groups of around five people.

Group 1

Europe has shown that privacy is important

The definition of personal data is going to be larger in future, covering more data

Not only anonymous data but also pseudonymous data

Data controller has moral obligations.

Schools and parents and users need to understand who is responsible for what

Organisations that are privacy conscious will be slow and cautious. Others will be faster moving, so in the worst-case scenario may end up with lots of companies that are not privacy compliant

Comment – we already have this in the US, adaptive, smart systems with no data protection. This helps them to be successful. We must not have regulations that force us to fall too far behind.

Competitive disadvantage if we do everything by the book (but, of course, the book isn’t clear)

Group 2

Three things:

  1. Bringing people on board – going out to teachers and to staff. How do we use it, how do we use it safely, what are the opportunities?
  2. Legislation – creating rules and guidelines, but also navigating existing legislation. Are we already compliant?
  3. Resource creation – building resources and developing them for different contexts and settings, and using what we already have to gain benefit.

Key word is ‘quality’.

Group Three

Also had three things

  1. Institutional support – lot of teachers are experimenting, but when they want to scale up they need institutional support to do this. More time for teachers to spend on these issues.
  2. Empirical evidence. More opportunity to experiment would lead to best practices and more evidence of what works, and the sharing of best practices. Explicit part of grant proposals – these should all include an evaluation phase, where the results are shared with the community. Grant should always include at least two people who spend more than 10% of their time on the project. Make the grant guidelines inline with pushing the field forwards
  3. Grant – require a lot of paperwork and elaborate planning that takes some time for you to start and avoids the gaining of fast results.

Comment – SoLAR had the Open Learning Analytics framework idea, and that sort of framework approach is currently missing. Unless we put grants within a framework we may all end up doing the same thing across different projects.

solaresearch.org/OpenLearningAnalytics.pdf

Comment – more and more requirements from the Commission for things that are developed with EC money to be open. Also academics are interested in developing things but not in going on to exploit them

Who is in charge? Who is there to assess the quality? Governments? External experts? Education authorities?

Group four

Three years is one generation of students and about 0.1 generation of teachers, so three years time is the students who have already applied and, by and large, the students are already here.

People are building silos to protect themselves and their data.

What are the sentiments that are going to be coming in from the outside world? Reference to black-box society – black box analytics are already making decisions for us

http://www.hup.harvard.edu/catalog.php?isbn=9780674368279

We are not Facebook, Google, we are not the unaccountable algorithm. The wave of sentiment created by them could sweep us away.

Comment: very often universities not concerned with the pedagogy but only with academic publications and with ISI ranking – not necessarily seen as the responsibility of academics

Group five

Agree with points about financial resources, quality and resource creation.

At some point, our project ends – what then?

Already involved with the head of our teaching department. Have to realize that this electronic portfolio system – fully developed and implemented - will no longer be funded. Developed and specified – but what next?

What do we want in learning analytics? We have cognitive aspects, but what about social aspects and emotional aspects?

Thinking about workplace learning.

Comment: is there a role for the EC in kick-starting and talent spotting?

Comment: Discussion of business models. If the model works, can we put it on KickStarter?

Group six

Discussed motivational and policy effects at many different levels.

IT department having control over data due to institutional policy and not allowing faculties access to all student data – up to institutional policies on continuous assessment.

Taking into account government and agency policy considerations. How different levels of completion affect student funding.

The group had some examples from Norway of the introduction of continuous assessment.

Different learning models, such as problem-based learning

Persuading institutions to adopt teaching and learning models associated with motivation and with analytics

Touched upon the theme of quality control

Managing expectations. Who is going to ensure the quality?

Comment: A lot of resistance against any educational innovation. Higher education continues to be focused on lectures. Commercial vendors have tools on offer, but there is no quality control. Quality controls take time, and often customers want tools that are faster to use. Consultancies can come in and make a lot of money by selling snake oil.

Group seven

Considered things to do with individual settings.

It is important when thinking about technology and the future to consider the danger of getting carried away and doing too much too soon.

Thinking about the strengths, and the things that are going to help us to expand take-up of learning analytics.

Might want broad but relatively shallow coverage with learning analytics rather than just some institutions going deep

Tone of learning analytics – how do learners react to automated messages.

Views of parents and teachers are also important.

How would you feel if you had your essay marked by a machine – would it miss the nuances?

Doing learning analytics at a low level – the possibilities are already there

 

 

Rebecca Ferguson
10:25 on 16 March 2016

Morning group exercise

Feedback from morning’s group exercise, working with future scenarios and the implications for learning analytics

Scenario 1: Jack Wood is a teenager who cannot easily do his homework because it requires technology his family cannot afford

Learning analytics cannot help here – it is not a methodology that can be used to overcome inequalities.

The scenario was considered to be overwhelmingly grim, but also quite realistic.

Went quite surreal – bringing in government spying agencies.

So many core fundamentals are wrong.

Highlights the inequality – it is unfair.

Learning analytics can identify that the student does not do as well with this homework. Might adapt the homework. Might look at whether there is too much homework

Is access to digital devices as core as access to heat and food? Is it becoming a human right?

Scenario 1: Jane Phillips – a teacher in Jack’s school. Her role has been limited to doing what the computer cannot do

Group got really depressed by this scenario

Role of a teacher is reduced – why would you want to be a teacher in these circumstances?

Learning analytics should support what you are already doing

If teachers don’t understand why they are being asked to do things, they are likely to try to subvert or gain the system

If it could be like that in 2025, we need to start thinking about it now and avoiding that scenario

We are currently behind the curve on influencing people, including policy makers. We need to be aware of the persuasive cases made by those marketing tools.

 

Scenario 2: Tayla Ozdemir, a trained social worker with temporary refugee statuus

An opportunity for a use of learning analytics to help a country to repair past political mistakes. A country is short of social workers, a social worker from another country does not have the language or the certification that would allow her to work there.

Could get her to work through scenarios, compare her with benchmarks, and identify specific areas where she needs training and development.

Using stealth assessment.

Assessment is not the goal of education.

Use a virtual buddy who can act as a coach, and can supply mock cases that Tayla needs to deal with. If she can deal with them, then that suggests she already has the qualifications and experience that she claims to have.

Language is a way of thinking – it changes how you think about things. Not necessary good if you don’t have to learn new languages in the future.

Scenario 2: Jan Zoetermelk, potential employer for Tayla

Talking about brain implants. How different are they from the smart phones we have at the moment? Would being required to have an implant be a physical violation?

If we don’t regulate, society will make its own mind up.

People are switching off their phones when they go into shopping centres because of unregulated tracking.

If society has to decide whether to trust something you often get intense responses.

Speaker’s niece and nephew required to have an iPhone or iPad at their school.

Ethics is important here.

Language is not just about the language, it is about how people think and act in that language – how to approach people.

Rebecca Ferguson
11:16 on 16 March 2016

Embedded Content

Contribute

Rebecca Ferguson
2:49pm 16 March 2016


Group work on future visions: Amsterdam afternoon

Vision 3 learner: Thomas Müller has a depressing life. There is no longer a demand for his role as a taxi driver – he wants to retrain as a microbrewer

People with the expensive, top-of-the-range devices are going to be the gamers.

Smart cars will remove employment prospects from people working in transport and haulage industries.

Draw on the idea of giving the data back to the learner to support them in their career trajectories. Could be matched with career and qualification data relating to other people in order to show what similar people have done, and the outcomes.

LEA’s Box project has been reflecting on what learning analytics would look like for different career trajectories – painters, chefs, guitar players... Check the link on ‘The Box’ at http://www.leas-box.eu/

How do we avoid the satnav problem of ‘The road is blocked – all go down that small road instead’?

So many people are not on a ‘standard’ career track. Analytics might suggest what most people have done – but the most interesting results are from those people who have followed non-standard routes.

Vision 3 educator: Maria Kock designs virtual reality training programmes, including one for micro-brewers – she is now designing training for midwives.

Could use learning analytics to look at knowledge and aptitude of learners.

Broadly feasible, though not the right thing to do, would be to use monitoring and video capture of experts in their practice.

In this case, capturing expert behaviour might redress a lack of expert knowledge by the ‘educator’.

Need to think of the roles played by the virtual in this sort of circumstance – what is the role for technology here, what is the role for more traditional pedagogical approaches

Barrier might be the use of lawsuits – experts might feel the intensive monitoring could be used against them in court if something goes wrong.

Need to make sure training is quality controlled and appropriately monitored, and that what is being labeled as expert activity is indeed expert activity.

There are lots of issues around quality assurance, domain expertise, and trying to hit impossible deadlines.

 

Vision 4 learner: Maria wants to shift from painting towards fusion cuisine with the use of 3D printers

Analytics might figure out what her existing skills set is.

Development of a trademarked LABlocker – hire the identity of someone (essentially wearing a mask) in order to complete training without being identified. Pay for this as a premium service.

Should learners be able to pay extra to gain access for higher quality learning analytics?

Different elements are involved when you are studying for personal interest rather than because you need a certificate. Should you have to hand data over in order to be able to learn?

Should you be able to remove your data from the analytics?

Vision 4 teacher: Remy Depardieu teaches online cookery online

Analytics should pick up on changes that have occurred after an organisational change. Could employ analytics to pick up on problems in this area

Problem of cultural context. One approach is to avoid it and apply a one-size-fits-all solution. Another approach is to take cultural context into account, and learning analytics could help teachers by indicating cultural dimensions and ambiguities.

What if teachers who worked for private companies, and without trade unions protecting them, were chipped in order to monitor their behaviour?

Could the computer be smart enough to understand the cultural context of a learner? Could it tell an American student from a European student. Could it distinguish between different types of American student?

You need to know the learning outcome before you

Vision 5 learner: Asa Anderson: Too much use of learning analytics

There has obviously been too many people selling whizzy systems – we need more pedagogic understanding of what systems can do.

How do we deal with social media – what does cheating mean in the age of social media?

Vision 5 educator: Kristofer Palm is making a great deal of use of technology and analytics

This is what marketing companies are trying to sell us.

We are antagonistic to this vision.

This seems a very bleached-white-wood antiseptic scenario

How do you maintain creativity in this sort of environment

We know teachers, particularly new teachers, are interested in having more support.

Erosion of trust. Isn’t it a role of a teacher to trust learners and shouldn’t they trust their colleagues?

How do students learn to trust the system?

Should they learn to game the system?

Vision 6 learner: Natalia Eglitis is a student who wants to go abroad to have better chances on the job market

We are still not clear about the ethics of this

Where do learning analytics infringe upon personal choice?

Vision 6 employer: Lian Xian in China is looking to take on low-paid European staff if they can learn to speak Chinese

Would people need to learn different languages?

Vision 7 learner: Giovanni Zanardi lives in an environment where there is little use of technology and analytics

Vision 7: Laura Botticelli teaches in an environment where there is little use of technology and analytics

This profile puts forward a lot of the views that teachers have towards technology

Educators carry out analysis of rich data from human interaction as they have always done.

We would want data at the European level to see what the results are of this approach. Do we see inward or outward mobility to these schools, and do we see their graduates being employed or not?

Would the majority of teachers love this approach?

What about informal use of technology – would students start to use the technology outside school and shift their learning elsewhere?

Analysing data is only one part of learning analytics. We need to be cautious about what we can do with this analysis.

Vision 8 Learner: Elena Fechter Access to data

Vision 8 Educator: Ernst Bild

Are we talking too much about élites, where parents want access to the data of young adults? Do we feel disconnected from the people in the scenarios?

Not everyone can be above the average. Big data are charming. We have them in MOOC settings and in university settings and in commercial perspectives, which are not like learning settings. We sit in front of data sets and we just try things out.

These scenarios are scary – as if they are from a bad science fiction movie.

It is a dystopia if we don’t get it right.

This is a system that isn’t meet the needs of either the learner or the educator, so they are both finding ways to work around it.

If you don’t have a minimal subset of data you can’t use the data.

What if we empower students to deny people access to data?

Vision 9 Learner: Juan Hernandez-Santiago learners are making use of a massive open access environment. Satisfaction of students and professionals has never been so high

Vision 9 Educator: Marianne Salome-Hernandez

Barrier is that even though there is a large open system, it doesn’t talk to other proprietary systems

If 99.97% of people are satisfied, then you are doing something wrong, because if you are learning you need to make mistakes and find things difficult. Students need to fail to a certain degree to learn. Maybe you are measuring the wrong thing.

Contribute to the discussion

Please log in to post a comment. Register here if you haven't signed up yet.