Learning Analytics Expert Workshop (Amsterdam)
Cloud created by:
11 March 2016
Day 1 – Tuesday 15 March 2016
Networking coffee is available from 10:00 onwards
10:30 Registration, coffee and Ice-breaker session
11:00 Setting the policy context: Riina Vuorikari, JRC-IPTS
11:05 Invited Expert introductions (1 slide, 4 minutes each)
13:30 Introduction: LAEP aims, research questions and findings so far: Rebecca Ferguson and Bart Rienties, The Open University, UK
14:00 Learning analytics projects/tools: lightning presentations on European-funded projects
14:30 Introduction to State-of-the-Art Exercise: Identifying key learning analytics tools, policies and practices
15:00 Coffee and discussion
15:20 State-of-the-Art Exercise: Identifying key learning analytics tools, policies and practices
16:20 Plenary session: key tools, policies and practicesIncluding validation of State-of-the-Art Exercise
17:00 End of formal sessions, Day 1
Day 2 – Wednesday 16 March 2016
09:00 Arrival, coffee and Day 2 Ice-breaker session
09:30 Visions of the future of learning analytics LACE – Learning Analytics Community Exchange
10:00 Learning Analytics Planning Exercise: How participants see their work in this area developing
10.45 Coffee break
11:00 Introduction to Foresight Exercise: Ways of thinking about and planning for the future
13:30 Foresight Exercise: Using learner and educator profiles to envisage the future
15:15 Coffee break
15:30 Validation of Foresight Exercise
16:00 Plenary session: Potential for European policy related to learning analytics
17:00 End of workshop
Amsterdam live blog: Visions of the Future session
Presentation by Doug Clow (The Open University, UK)
Visions of the future of learning analytics, a link to that document is available on the LACE website.
A long document. This is the distilled headline version.
From yesterday’s conversation you have the same sort of ideas. We will talk today about taking it further and the policy implications.
A lot of work about understanding what the future might be. This year’s Horizon Report has learning analytics in the ‘one year to happening’ category. The OU produces ‘Innovating Pedagogy’ reports and they also have learning analytics as an important element.
Our exercise today is about looking at different sorts of future and what the implications are for policy.
The LACE study used a Policy Delphi methodology, which provides a menu of options about what might happen. Which of the potential futures do we see as desirable and feasible? What should happen to support the good and prevent the bad?
We drew up eight plausible futures, iteratively within consortium and contacts. Then carried out online survey, with over 130 people participating. About half of those were invited experts, the other half came from calls at invited events.
Also, at open events, we canvased people’s opinions of the visions, for example at BETT.
Most of you have these ideas in your head already
1. The phycisal environment. Lots of us have sensors with us. Phones these days include tools like a barometer. Sensors are cheap and widely available. You can use webcams to monitor gaze. We can project how these might be used extensively in education.
2. Personal data tracking and quantified self and biochemical and individualized data tracking.
3. Takes things in the opposite direction. Privacy, ethics, people overstep the mark and then step back. Then we might move away from analytics
4. Learners having control of their information and data. They grant permission but control remains with them.
5. Open learning analytics, standardization and interoperability all come together and works. Everything many of us have been working on.
6. Analytics become much more essential tools. Extrapolating that predictive modeling will become more successful and widely used.
7. Decision making gets better than humans
8. We are so good at developing fine-grained analytics to computers that we can delegate much of the teaching to computers and we can move towards a dream of personalized learning.
Asked people to rate for desirability and feasibility. People broadly positive except for the scenario where we don’t use analytics. Also seen as broadly feasible.
From the free-text responses. Great potential for learning analytics, but not a confident sense that that potential will be realized. Not a strong consensus. We are not sure whether that will happen.
Need for policies and infrastructure. Arguments ranged from UN-policed approach, right down to the smaller classroom, group-related context. Privacy and ethics and standards came up again and again.
Large, bold and centre is pedagogy. Learning analytics needs to be focused on the teaching, in support of what we want from our education system. Need to capture what we value in education. Must take these into account when we optimize towards measures.
Lots about ownership, power and ethics. Who is control. Whether analytics are done to you or whether they support you. How they support and challenge existing power structures. I f we have information that can help learners we have a responsibility to use it to do that.
Looked across schools, universities, workplaces and informal settings. We did find differences but these were often smaller than we expected. Still ethics and pedagogy.
Little sense that the technology was going to be a barrier. Sometimes said we can do this already. Major barriers are social, structure and policy.
AlphaGo has just beaten Lee Sedol at the game of Go – beating the world’s best human 4-1 at the moment. Our capabilities in machine learning are moving forward very rapidly. Unlikely to be the main barriers to whether the visions of learning analytics are achieved or not
Thanks to everyone involved, and to the commission as our funders.
Do these visions make sense to those in the room? Is that our collective view of how things are?
Jeroen: I Like the stress on pedagogy. How do learning analytics change the pedagogy, this goes both ways?
Doug: Yes, this emerged. Teaching practices can and must change.
Alan: This is a European focus?
Doug: We went for a global perspective. The brief was to look globally. Fewer responses from the US that were concerned with privacy and ethics. Lot of European respondents.
Alan: Teaching and practices here are better but technology tends to be better in America. Dislocation between continents
Yves: Like the visions. What sorts of problems do they solve? Dropouts, inequality, access?
Doug: Didn’t ask about that. Covered most possibilities to cover a breadth of things. Weren’t looking at what should happen.
Bart: That’s what we shall do today
Daniel: What about adoption? Business tracks employees, risk of misbehaving. Every part of our life is tied up with analytics and intelligence, but we have a gut response when it comes to education
Doug: Yes, this is happening in a lot of other areas, though some people did argue that education is and should be different and separate from commercial analytics. Degree to which people feel it is choice and they are getting something of value back. Shouldn’t have to be tracked to get an education
Ed: Final destination makes quite a big difference. Each framework looks different if analytics are used in complex or simple way
Doug: yes, need to get more fine-grained
Speaker: Does teacher have a right? Mediating through a technology. Doing the same things as before, but just mediating it through technology. Ethical issues. No one should be able to steal the data. It is not only the data you collect, it is also the context and the use. I can watch students in my classroom already.
Andrew: Yes, you can do it now in class, but if you used it to humiliate students that would be over the line.
09:12 on 16 March 2016
Amsterdam live blog: Group discussion
Group discussion on what will happen in the next years with learning analytics - seven groups of around five people.
Europe has shown that privacy is important
The definition of personal data is going to be larger in future, covering more data
Not only anonymous data but also pseudonymous data
Data controller has moral obligations.
Schools and parents and users need to understand who is responsible for what
Organisations that are privacy conscious will be slow and cautious. Others will be faster moving, so in the worst-case scenario may end up with lots of companies that are not privacy compliant
Comment – we already have this in the US, adaptive, smart systems with no data protection. This helps them to be successful. We must not have regulations that force us to fall too far behind.
Competitive disadvantage if we do everything by the book (but, of course, the book isn’t clear)
- Bringing people on board – going out to teachers and to staff. How do we use it, how do we use it safely, what are the opportunities?
- Legislation – creating rules and guidelines, but also navigating existing legislation. Are we already compliant?
- Resource creation – building resources and developing them for different contexts and settings, and using what we already have to gain benefit.
Key word is ‘quality’.
Also had three things
- Institutional support – lot of teachers are experimenting, but when they want to scale up they need institutional support to do this. More time for teachers to spend on these issues.
- Empirical evidence. More opportunity to experiment would lead to best practices and more evidence of what works, and the sharing of best practices. Explicit part of grant proposals – these should all include an evaluation phase, where the results are shared with the community. Grant should always include at least two people who spend more than 10% of their time on the project. Make the grant guidelines inline with pushing the field forwards
- Grant – require a lot of paperwork and elaborate planning that takes some time for you to start and avoids the gaining of fast results.
Comment – SoLAR had the Open Learning Analytics framework idea, and that sort of framework approach is currently missing. Unless we put grants within a framework we may all end up doing the same thing across different projects.
Comment – more and more requirements from the Commission for things that are developed with EC money to be open. Also academics are interested in developing things but not in going on to exploit them
Who is in charge? Who is there to assess the quality? Governments? External experts? Education authorities?
Three years is one generation of students and about 0.1 generation of teachers, so three years time is the students who have already applied and, by and large, the students are already here.
People are building silos to protect themselves and their data.
What are the sentiments that are going to be coming in from the outside world? Reference to black-box society – black box analytics are already making decisions for us
We are not Facebook, Google, we are not the unaccountable algorithm. The wave of sentiment created by them could sweep us away.
Comment: very often universities not concerned with the pedagogy but only with academic publications and with ISI ranking – not necessarily seen as the responsibility of academics
Agree with points about financial resources, quality and resource creation.
At some point, our project ends – what then?
Already involved with the head of our teaching department. Have to realize that this electronic portfolio system – fully developed and implemented - will no longer be funded. Developed and specified – but what next?
What do we want in learning analytics? We have cognitive aspects, but what about social aspects and emotional aspects?
Thinking about workplace learning.
Comment: is there a role for the EC in kick-starting and talent spotting?
Comment: Discussion of business models. If the model works, can we put it on KickStarter?
Discussed motivational and policy effects at many different levels.
IT department having control over data due to institutional policy and not allowing faculties access to all student data – up to institutional policies on continuous assessment.
Taking into account government and agency policy considerations. How different levels of completion affect student funding.
The group had some examples from Norway of the introduction of continuous assessment.
Different learning models, such as problem-based learning
Persuading institutions to adopt teaching and learning models associated with motivation and with analytics
Touched upon the theme of quality control
Managing expectations. Who is going to ensure the quality?
Comment: A lot of resistance against any educational innovation. Higher education continues to be focused on lectures. Commercial vendors have tools on offer, but there is no quality control. Quality controls take time, and often customers want tools that are faster to use. Consultancies can come in and make a lot of money by selling snake oil.
Considered things to do with individual settings.
It is important when thinking about technology and the future to consider the danger of getting carried away and doing too much too soon.
Thinking about the strengths, and the things that are going to help us to expand take-up of learning analytics.
Might want broad but relatively shallow coverage with learning analytics rather than just some institutions going deep
Tone of learning analytics – how do learners react to automated messages.
Views of parents and teachers are also important.
How would you feel if you had your essay marked by a machine – would it miss the nuances?
Doing learning analytics at a low level – the possibilities are already there
10:25 on 16 March 2016
Morning group exercise
Feedback from morning’s group exercise, working with future scenarios and the implications for learning analytics
Scenario 1: Jack Wood is a teenager who cannot easily do his homework because it requires technology his family cannot afford
Learning analytics cannot help here – it is not a methodology that can be used to overcome inequalities.
The scenario was considered to be overwhelmingly grim, but also quite realistic.
Went quite surreal – bringing in government spying agencies.
So many core fundamentals are wrong.
Highlights the inequality – it is unfair.
Learning analytics can identify that the student does not do as well with this homework. Might adapt the homework. Might look at whether there is too much homework
Is access to digital devices as core as access to heat and food? Is it becoming a human right?
Scenario 1: Jane Phillips – a teacher in Jack’s school. Her role has been limited to doing what the computer cannot do
Group got really depressed by this scenario
Role of a teacher is reduced – why would you want to be a teacher in these circumstances?
Learning analytics should support what you are already doing
If teachers don’t understand why they are being asked to do things, they are likely to try to subvert or gain the system
If it could be like that in 2025, we need to start thinking about it now and avoiding that scenario
We are currently behind the curve on influencing people, including policy makers. We need to be aware of the persuasive cases made by those marketing tools.
Scenario 2: Tayla Ozdemir, a trained social worker with temporary refugee statuus
An opportunity for a use of learning analytics to help a country to repair past political mistakes. A country is short of social workers, a social worker from another country does not have the language or the certification that would allow her to work there.
Could get her to work through scenarios, compare her with benchmarks, and identify specific areas where she needs training and development.
Using stealth assessment.
Assessment is not the goal of education.
Use a virtual buddy who can act as a coach, and can supply mock cases that Tayla needs to deal with. If she can deal with them, then that suggests she already has the qualifications and experience that she claims to have.
Language is a way of thinking – it changes how you think about things. Not necessary good if you don’t have to learn new languages in the future.
Scenario 2: Jan Zoetermelk, potential employer for Tayla
Talking about brain implants. How different are they from the smart phones we have at the moment? Would being required to have an implant be a physical violation?
If we don’t regulate, society will make its own mind up.
People are switching off their phones when they go into shopping centres because of unregulated tracking.
If society has to decide whether to trust something you often get intense responses.
Speaker’s niece and nephew required to have an iPhone or iPad at their school.
Ethics is important here.
Language is not just about the language, it is about how people think and act in that language – how to approach people.
11:16 on 16 March 2016