Digital Assess – adaptive comparative judgement (LAEP Inventory)

Cloud created by:

Rebecca Ferguson
11 February 2016

Synopsis

Digital Assess provides support for workflow around assessment of coursework or other evidence-based assessment scenarios. The system can be used for conventional assessor marking or for peer-learner assessment. Learning analytics are used to drive a process known as adaptive comparative judgement, which increases the reliability of the assessment.

 

Adaptive comparative judgement is a development of the assessment approach in which pairs of work by students are compared, using some defined dimensions of quality. Learning analytics drives the adaptivity by automatically determining which pairs to present to which individuals undertaking the assessment, in order that the increase in the reliability of the grading is maximised in each round of comparison. Over several rounds of comparative judgement, reliability statistics are computed, as well as statistics that identifies student work that is problematical. The process can also support year-on-year standardisation. The method is particularly applicable to cases where a detailed marking scheme is ill-suited to the object of assessment – for example for creative subjects or ‘soft skills’ – or would be excessively time-consuming, or where peer assessment has a pedagogic role.

 

Research undertaken by academics and high-stakes awarding bodies has demonstrated that adaptive comparative judgement is a reliable method, exceeding the inter-rater reliability typical of conventional essay marking.

Classification

Inventory type:

analytics for assessment

Role of analytics:

statistical inference

adaptation

Data sources:

uses own data

Keywords:

adaptive comparative judgement, peer assessment

Tool in Context

Learning:

school/VET/post-compulsory/informal

Supply model:

shared service model

Origin:

Digital Assess: TEL vendor

Ethics and privacy:

The Digital Assess system is designed to support secure high-stakes assessment. Peer assessment is undertaken anonymously, but any free-form assessment has some risk of re-identification.

Languages:

English

Maturity and Evidence of Utility

The tool has been rigorously evaluated by an awarding body (responsible for high-stakes public assessment), and piloted at scale at Edinburgh University. In 2015, Digital Assess reported raising $3million in new investment.

Further Information

Tool provider’s website: http://digitalassess.com

Outline of adaptive comparative judgement component: http://bit.ly/DA-ACJ

Story on new investment: http://bit.ly/DA-3m

Example(s) of use:

  • Case study, adaptive comparative judgement at Edinburgh University - http://bit.ly/DA-UoE
  • Report of an exploratory study undertaken by the Centre for Education Research and Policy of the Assessment and Qualifications Alliance (a UK awarding body) - http://bit.ly/DA-AQA

Research: Alastair Pollitt(2012): The method of Adaptive Comparative Judgement, Assessment in Education: Principles, Policy & Practice,DOI:10.1080/0969594X.2012.665354

Extra content

Embedded Content

Contribute

Contribute to the discussion

Please log in to post a comment. Register here if you haven't signed up yet.