د/ايمان زغلول قاسم

استاذ تكنولوجيا التعليم المشارك بكلية التربية بالزلفي

resaerch 3

Effective Practice with e-Assessment
An overview of technologies, policies and practice
in further and higher education
1
The Joint Information Systems Committee (JISC) supports UK post-16
and higher education and research by providing leadership in the use
of Information and Communications Technology in support of learning,
teaching, research and administration. JISC is funded by all the UK
post-16 and higher education funding councils.
www.jisc.ac.uk
Effective Practice
with e-Assessment
Further information
For further information about this publication or about the
work of JISC on e-assessment:
Email: [email protected]
Tel: 0117 954 5083
URL: www.jisc.ac.uk/assessment.html
To order further copies of this publication:
Email: [email protected]
For further information about the JISC e-Learning
Programme, visit the JISC website:
www.jisc.ac.uk/elearning
For news, features and further discussion of the Programme,
visit the JISC e-Learning Focus website:
www.jisc.ac.uk/elearningfocus
Disclaimer
Please note that the content of the case studies in this publication
reflect the views and opinions of the institutions concerned and are
not necessarily indicative of JISC policy. Products, websites and listed
resources are provided for guidance only and do not constitute an
endorsement of such resources. The contributors and JISC can accept
no liability for loss or damage arising from the reliance upon any advice
provided in this publication.
Copyright
The copyright for the content of this work is held by the Higher Education
Funding Council for England (HEFCE) on behalf of JISC. This publication
may be copied in its entirety without charge for use in an educational
context. However, extracts may not be quoted without acknowledgement
of the source, and material from this publication may not be sold or used
by way of trade without the express permission of the copyright holder.
© Copyright, 2007. HEFCE
1
Contents
Introduction 5
How to use this publication 5
Introducing key terms 6
Summary of key findings 7
Where we are now 10
The learner’s view 12
Barriers and enablers in
computer-based assessment 14
Case study: Collaborating for change – the COLA project 15
Challenges and benefits from
e-assessment tools and systems 16
Case study: Designing for high-stakes
computer-based assessment – University of Derby 17
Waymarking institutional practice 18
Case study: A tool for quality improvement –
South East Essex College 18
Case study: Managing the virtual and physical
environment – Farnborough Sixth Form College 21
Case study: Developing strategies to support
e-assessment – University of Dundee 22
Waymarking academic practice 24
Case study: Ensuring the quality of feedback –
The Open University 24
Case study: Raising the stakes with confidence-based
marking – University College London 27
Case study: Preparing for summative computer-based
assessment – University of Manchester 28
Waymarking learner-focused practice 30
Case study: Merging formative and summative
assessment – Coleg Sir Gâr 30
Case study: The e-portfolio route to accreditation –
Ultraversity, Anglia Ruskin University 33
Case study: Supporting learners through
diagnostic assessment – The Oldham College 34
Looking to the horizon 36
Case study: Understanding the role of assessment
in learning – the REAP project, University of Strathclyde 38
Case study: Embedding an institutional policy
for e-assessment – Loughborough University 39
Working towards effective practice 40
Glossary 43
Further information 45
‘The variety of applications of e-assessment reported and their innovation
and general effectiveness indicate the potential of e-assessment to
significantly enhance the learning environment and the outcomes for
students in a wide range of disciplines and applications.’
Roadmap for e-Assessment Report for JISC (Open University, 2006)
Introduction
Effective Practice with e-Assessment is the third in a series of
JISC publications on the skillful use of e-learning in 21st
century practice in a technology-rich context. In contrast to
the preceding guides in the series – Effective Practice with
e-Learning and Innovative Practice with e-Learning – the focus
of this publication is on practice in a broader institutional
sense, including the potential impact of e-assessment on
learning and teaching.
Through a sequence of case studies, Effective Practice with
e-Assessment provides an overview of current e-assessment
activity in further and higher education in the UK. In doing so,
it does not cover in any depth the technical developments that
underpin e-assessment. Its aim is to illustrate how the work
of researchers, tools and technical standards developers
has combined with initiatives from individual colleges and
university departments to advance our understanding of
what is effective practice in relation to e-assessment.
Its audience is those who manage, deliver or support
educational practice in a technology-rich context –
departmental heads, educational developers and learning
technologists, e-learning managers, examinations officers
and administrators, as well as classroom practitioners
seeking to know more about how e-assessment can add
value to teaching and learning.
Effective Practice with e-Assessment draws on a range of JISC
projects and studies, notably the e-Assessment Glossary, First
edition (JISC/QCA, 2006), the Roadmap for e-Assessment Report
for JISC (Open University, 2006) and case studies researched
for JISC by The Open University and the University of Derby.
Links to further information about these resources can be
found in the final section of this publication.
How to use this publication
The case studies provide illustrations of practice to
accompany an ongoing discussion of the issues and benefits
relating to e-assessment. These may be read in sequence or
be approached individually through three broadly based
thematic groupings: institutional, academic and learnerfocused
practice.
The full sequence of case studies provides a benchmark
against which to assess the progress and barriers in any one
institution, offering an opportunity to explore how others have
tackled issues, the lessons they have learnt, and the models
of effective practice that have been developed. For those
who set strategic targets or are involved in research and
development projects, these examples can also help to
identify the challenges still to be met.
A tool for implementing effective practice with e-assessment
is included to assist institutions in reviewing their current
practice and procedures or preparing for wider use of
e-assessment. The tool can also be downloaded from the
JISC website [www.jisc.ac.uk/assessment.html] for further
development and adaption in individual contexts.
As the terminology used may not be familiar to all readers,
terms defined in the glossary section have been highlighted
on first occurrence.
5
Introducing key terms
e-Assessment is used in this publication as a generic term
covering all uses of computers in assessment. This and some
closely related terms are explored here. Other key terms are
defined in the glossary section, using definitions given in the
JISC/QCA e-Assessment Glossary.
The term e-assessment is a broadly-based one, covering a
range of activities in which digital technologies are used in
assessment. Such activities include the designing and delivery
of assessments, marking – by computers, or humans assisted
by scanners and online tools – and all processes of reporting,
storing and transferring of data associated with public and
internal assessments.
JISC/QCA definition of e-assessment
e-Assessment is the end-to-end electronic assessment
processes where ICT is used for the presentation of
assessment activity, and the recording of responses. This
includes the end-to-end assessment process from the
perspective of learners, tutors, learning establishments,
awarding bodies and regulators, and the general public.
To differentiate between different types of activity,
computer-based assessment (CBA) is used in this publication
to refer to assessments delivered and marked by computer,
and computer-assisted assessment (CAA) to refer to practice
that relies in part on computers – for example, use of online
discussion forums for peer-assessment, audience response
systems in group work, completion and submission of work
electronically, or storage of work in an e-portfolio. However,
it should be noted that these terms are often viewed as
interchangeable.
Some form of e-assessment may be used at each of the three
stages at which a learner’s attainment and progress come
under review:
Diagnostic – assessment of a learner’s knowledge and skills
at the outset of a course.
Formative – assessment that provides developmental
feedback to a learner on his or her current understanding
and skills. Formative assessment can also be described as
‘assessment for learning’ since an assessment that is entered
into voluntarily, and on which no final qualification depends,
can prompt learners to adjust their own performance.
Summative – the final assessment of a learner’s
achievement, usually leading to a formal qualification or
certification of a skill. Summative assessment is also referred
to as assessment of learning.
Assessment of any kind can be referred to as low, medium or
high stakes. A low-stakes assessment is usually formative,
with results recorded locally. A medium-stakes assessment
is one in which results may be recorded locally and nationally,
but is not life changing. A high-stakes assessment, however,
is one in which the outcomes are of high importance to both
centre and candidates, affecting progression to subsequent
roles and activities.
A closely related concept is the e-portfolio, which can be
used to document in digital form the outcomes of each stage
in a learner’s journey. The distinction between an e-portfolio
as a record of achievement and an e-portfolio as a tool for
assessment becomes blurred when the outcomes of
assessments, including self- or peer-assessments in the form
of diaries, blogs or wikis, are included. An e-portfolio may also
be the means by which some qualifications are assessed.
6
Summary of key findings
Assessment is central to learning and teaching. What is
assessed defines what is taught and how it is learnt. The
process of assessment, in turn, shapes institutional practice
and affects a learner’s view of the value of engaging in
learning. Getting assessment ‘right’ is essential to the wellbeing
of learners and institutions, and instrumental to the
achievement of national strategies for widening participation
and e-learning.
Given the potential importance of e-assessment, developing a
clearer understanding of what constitutes effective practice in
relation to its use is now of key importance.
The potential of e-assessment
This review of current practice suggests technology can add
value to assessment practice in a variety of ways. If used with
skill and imagination, e-assessment – defined in its broadest
sense to refer to both computer-assisted and computerbased
assessments – can increase the range of what is
tested. It can provide evidence of both cognitive and skillsbased
achievements in ways that are durable and
transferable. It can enhance the validity of assessment
systems and encourage deeper learning.
e-Assessment in fact is much more than just an alternative
way of doing what we already do. A growing body of evidence,1
some illustrated in this publication, indicates that welldesigned
and well-deployed diagnostic and formative
assessments can foster more effective learning for a wider
diversity of learners. Assessment is perhaps the best way of
identifying the support needs of learners and can instil a
desire to progress further if linked to appropriate resources,
good quality, timely feedback, and to challenging but
stimulating ways of demonstrating understanding and skills.
Effective use of technology can make significant
contributions here.
e-Assessment can support personalisation. Any time,
anywhere assessments benefit learners for whom a
traditional assessment regime presents difficulties due
to distance, disability, illness, or work commitments.
On-demand summative assessments, when available,
increase participation in learning by enabling learners to
progress at a pace and in a way appropriate to them.
Looking to the future, e-assessment can offer a broader
palette of tools for awarding bodies, developers and academic
staff to work with. Some significant projects in 14-19 and
higher education indicate that more valid and imaginative
assessment and learning experiences could ensue. Which
technologies are selected and how they are deployed form
part of the challenges and skills inherent in ‘designing for
learning’, a concept explored further in a companion guide,
Effective Practice with e-Learning (JISC, 2004).2
1 Nicol, D J (2006) Increasing success in first year courses: assessment
re-design, self-regulation and learning technologies, paper presented
at ASCILITE Conference, Sydney, 3-6 December, 2006, www.ascilite.
org.au/conferences/sydney06/proceeding/onlineIndex.html
Also Sharpe, R, Benfield, G, Roberts, G and Francis, R (2006)
The undergraduate experience of blended e-learning: a review of UK
literature and practice undertaken for the Higher Education Academy,
www.heacademy.ac.uk/4884.htm
2 Effective Practice with e-Learning (JISC, 2004),
www.jisc.ac.uk/elp_practice.html
7
Enhancing assessment practice
The case studies in this publication have highlighted the
following as important considerations when rethinking
assessment practice:
• Appropriateness
• Timeliness
• Relevance
• Accessibility
• Validity
• Quality of supporting systems
These criteria do not apply solely to e-assessment, but are
significant here because the application of technology has
either proved beneficial, or has prompted a reassessment of
institutional, organisational or academic processes in
the following respects:
One way in which the introduction of e-assessment may
enhance the quality of the learner’s experience is through the
closer alignment of assessment with the pedagogic approach
used – as options for modes of delivery broaden to include
blended and online models, assessment in a similar form
becomes increasingly appropriate and relevant.
However, e-assessment should not be viewed in isolation
from the practice surrounding its use – for example, the
timeliness of assessments and the quality of the feedback
provided are key to learners’ progress. Technology can assist
by supporting on-demand delivery of tests to large numbers
of learners, but the real advantage may lie in the immediacy
of feedback and opportunities for further learning – for
example, the any time, anywhere availability of resources.
Increasingly, a range of methods, both computer-based and
computer-assisted, is occurring in higher education. Online
tasks involving peer- and self-assessment and increasing use
of e-portfolios and assessment tools within virtual learning
environments (VLEs) indicate the diversity of approaches that
has been established in this sector. Equally important is the
relevance and accessibility of the assessment for the learner:
evidence suggests e-assessments can provide assessment
experiences that are more authentic – through the use of
e-portfolios, reflective diaries, blogs or virtual world scenarios,
for example. When objective tests are the most appropriate
method, interactive elements and multimedia, or confidencebased
marking, can make assessments more valid,
accessible and engaging.
Quality assurance procedures and staff training in the design
and delivery of e-assessments need to respond to these new
developments. For example, in preparation for increasing
use of e-assessment in further education, the regulatory
authorities in England, Wales and Northern Ireland are
producing regulatory principles and guidance for
e-assessment.3 These will be supported by the development
of accredited qualifications for staff in conjunction with
Lifelong Learning UK (LLUK).
Implementation of computer-based assessment on a wide
scale, however, presents a range of organisational
challenges, including management of the IT infrastructure
and accommodation in response to fluctuating periods of
demand, and provision of dedicated technical support staff. As
a result, the use of such assessments is still uneven, although
highly effective in some areas. Examination regulations need
to be reassessed to encompass new practices and procedures,
3 www.qca.org.uk/downloads/qca-06-3006_e-assessmentweb.pdf
8
9
to address issues of authentication and accessibility, for
example, and institutions or departments involved in the
delivery of medium- to high-stakes tests will require
contingency plans against computer failure and loss of data.
As e-assessment becomes more widely adopted, institutions
may prefer to place in the hands of specialists many of the
issues associated with computer-based assessment, such as
accessibility and intellectual property rights (IPR) in item
bank development, or the maintenance of a suitable IT
infrastructure for assessment delivery. Test design and item
bank development may be outsourced and scanning bureaux,
optical mark reading units or commercial e-assessment
centres used. Some universities have set up specialist units
for e-assessment, taking responsibility for assessment
development away from individual departments and changing
the scope of the practitioner’s role.
The use of any form of e-assessment has increased concerns
over plagiarism and malpractice, but some case studies in
this review have revealed how technology can enhance the
validity and security of assessments. Currently, e-assessment
is more frequently found in low- to medium-stakes than in
high-profile, high-stakes contexts. But this is changing, as
the major awarding and regulatory bodies and institutional
departments prepare for wider use of e-assessment in
the second decade of the 21st century, and as a more
sophisticated understanding develops of how learning and
assessment can be effectively integrated into a flexible and
supportive learning environment, enabled by technology.
Figure 1 shows how the integration of assessments, feedback, learning resources and e-portfolio records into one
technology-supported learning environment can provide the basis for effective learner progression.
Figure 1 The relationship between assessment and effective learning, adapted from a model by Roads, M (2006)
Learning module
(e-learning or
blended learning)
Assessment Successful
Feedback to student
Information rich to
support next element
of learning
Next learning
module
(e-learning or
blended learning)
Unsuccessful
Feedback to
student
Information rich to
support revision
Revision
module
Bank module
towards accredited
qualification – use of
portfolio
Where we are now
e-Assessment can play a significant part in a more flexible
and personalised environment for learning as illustrated in
this review, but the extent to which awarding bodies and
institutions across the UK are prepared for its wider
adoption still varies. There are a number of reasons for this
quiet prelude to what could be the next technology-based
revolution in the classroom.
Further education: summative assessment
All four regulatory authorities in the UK are committed to a
future in which e-assessment plays a more significant role.
The Qualifications and Curriculum Authority (QCA)’s vision for
awarding bodies and learning providers is that existing GCSE,
AS and A2 examinations will be available on screen, all new
qualifications will include an on-screen option, and the first
on-demand assessments will be introduced by 2009.
However, the fastest uptake has been in those areas of the
curriculum that have a strong rationale for on-demand
objective testing – use of online City & Guilds tests has
increased sharply since 2002 (see Figure 2). Elsewhere, the
financial risk inherent in systems development and concerns
over public acceptance of computer-based assessment (CBA)
in high-stakes examinations have acted as a brake on the
introduction of summative e-assessment in England.
Nonetheless, the foundations are being laid. Edexcel and the
Assessment and Qualifications Alliance (AQA) now offer online
components in GCSE general science. Modern foreign
languages and design and technology could soon follow suit.
In 2006, Scottish students used computers to sit a multiplechoice
examination in biotechnology for the first time. A pilot
for on-screen Key Stage 3 ICT tests is to be completed in
2007, although this test will not be statutory.
These examples have provided test beds for high-volume
application of CBA. New qualifications, especially those
associated with ICT – for example, the online GCE Moving
Image Arts from the Council for Curriculum Examinations
and Assessment (CCEA) in Northern Ireland and Edexcel’s
Diploma in Digital Applications (DiDA) – have offered further
opportunities to test the robustness and security of systems
and procedures and the appropriateness of online
assessment for learners, staff and institutions.
Further education: an integrated approach
Integration of e-learning and e-assessment into the fabric of
the curriculum is vital to wider-scale implementation of
computer-based and computer-assisted assessments in
schools and further education.
Scotland has made significant inroads into establishing an
integrated e-learning environment through initiatives such
as Glow, a national schools learning platform funded by
the Scottish Executive, through the development of online
question banks for Scottish Qualifications Authority (SQA)
unit assessments and through the SCHOLAR virtual college
resources and assessments. In Northern Ireland, the
Department for Education and Learning (DEL) has proposed
that colleges include e-assessment in their ILT strategies –
another major step towards the integration of e-assessment
into an ICT-rich vision for learning.
2002/3 2003/4 2004/5 2005/6
Financial year
0
100000
200000
300000
400000
500000
600000
700000
800000
Figure 2 City & Guilds Global Online Assessment (GOLA) system growth
in test numbers since 2002
10
‘The assessment industry is undergoing a fundamental transformation.
A decade ago, the phrase ‘e-assessment’ was not even in use.’
Martin Ripley (2004) Educational Consultant and former Head of e-Strategy Unit, QCA
Higher education: formative e-assessment
Increased student numbers and reduced student-to-staff
ratios have focused attention in higher education on ways in
which e-assessment can support learners starting out with
varying levels of knowledge and ability.
New pedagogic approaches are opening up, challenging
perceptions that e-assessment is only suited to testing
surface knowledge.
Some innovative examples of assessment for learning include
virtual world scenarios for professional training, and
web-based tools, such as e-portfolios, blogs and wikis, to
develop skills in reflection and self-assessment. Online
tasks involving peer- and self-assessment, which can
capture evidence of achievement in collaborative activities,
and increasing use of the assessment tools within VLEs,
indicate the diversity of approaches that has been established
in this sector.
Higher education: summative assessment
Longer experience of e-assessment in higher education has
demonstrated the challenges and benefits in low- to mediumstakes
contexts. Objective tests – either computer based or on
optical mark reader sheets – are well-established in subject
disciplines that depend on accurate recall of knowledge. With
effective integration of interactivity and multimedia, these
have established a valid role and purpose, but are still not
used in all disciplines.
Issues of plagiarism and authentication of candidates’ work
present considerable challenges. Nonetheless, applications of
computer-based assessment in high-stakes contexts are
being piloted and innovative developments from these trials
can be transferable to other subject areas.
Key steps towards e-assessment
g Over 80,000 e-assessments were taken at
Loughborough University during 2006.
g The City & Guilds Institute has delivered 1.3 million
tests since 2003, 650,000 of which were in the
academic year 2005-06.
g The CCEA GCE Moving Image Arts completes its
pilot phase in 2007-08 to become the first AS/A2
qualification to be assessed entirely online.
g Awarding bodies in England offer an online
GCSE science in 2006.
g In 2006, awarding bodies and TechDis collaborate
on the production of guidelines for accessibility in
e-assessment.
g A pilot online multiple-choice examination in
biotechnology at Intermediate 2 and Higher Level
was offered by SQA for the first time in 2006.
g In 2007, the Phase 2 report became available
from Project e-Scape on e-portfolio-based
assessment of GCSE design and technology
coursework.
g SQA now provides online National Qualification
Unit assessments in computing and Higher
mathematics. Pupils with additional assessment
requirements at Standard Grade are able to word
process their responses.
g Following initial trials in 2001, on-demand basic and
key skills tests in England, Wales and Northern
Ireland can be taken online at levels 1 and 2.
g Availability of VLEs and their use by UK educational
providers grows year on year.
11
The learner’s view
Learners are increasingly more likely to experience
technology-based assessment directly, or to be assessed
within a system that is supported by online resources.
Figure 3 gives an insight into how this could occur and
suggests the experiences and needs learners could acquire
at the different stages of their learning journey.
As today’s school pupils progress on to post compulsory and
higher education, their experience of gaming in the social
and personal dimensions of their lives, and of interactive
formative assessments in the educational dimension of their
lives, means that they are increasingly confident about
taking computer-based assessments.
5 – 11 11 – 14 14 – 16 Access to computers and
learning resources out of school
High-quality web-based
assessments for learning
Innovative games-based
assessments for learning
A unique learner number
Opportunities to personalise
their learning, by taking
some assessments online
when ready
Learners need:
Online internet safety and
proficiency training
Web-based interactive
multimedia learning
resources and games
Drill and skill quizzes
direct to mobile phones
Statutory teacher assessment
based on online assessment
materials (National
Assessments – Scotland)
Statutory teacher
assessment based on
online exemplars (National
Curriculum – England)
e-Profiling of early
years development
(Optional –England)
BBC Mobile Bitesize quizzes
for learning on the move
Online interactive resources
for personal, social and
health education
Gaming
Online banks of National
Assessment materials
(Scotland)
Key Stage 3 ICT
non-statutory on-screen
assessments (England)
Curriculum Online
multimedia resources for
learning (England)
DiDA – a paperless
qualification in digital media
Interactive web-based
BBC Bitesize resources
for revision
Online learning and
assessment materials for
gifted and talented children
Virtual world simulations
testing skills in context (ICT
and science)
Online Standard Grade
practice resources (Scotland)
Computer-based components
in GCSE science
examinations (England)
Figure 3 An overview of e-assessment opportunities at different ages and stages
12
16 – 19 19 – 25 25+
e-Portfolios that are transferable
between institutions
Personal online space for
e-portfolio development
Online peer and tutor support
High-quality feedback on online
formative assessments
Personal online space for
e-portfolio development
High-quality feedback on
online formative assessments
Formative and summative
e-assessments in some
degree and diploma courses
Online on-demand
City & Guilds assessments
Online diagnostic assessment
of basic and key skills
Driving theory test
Online basic and key
skills at levels 1 and 2
(England, Wales and
Northern Ireland)
Continuing professional
development online
Work submitted online for
Open University courses
Formative and summative
e-assessments in some
degree and diploma courses
Online learndirect courses
Online on-demand
vocational qualifications
(eg City & Guilds)
Online on-demand City &
Guilds assessments
Online diagnostic
assessment of basic
and key skills
Driving theory test
Online National Unit
assessments in Higher
mathematics (Scotland)
Multiple-choice online
component in Intermediate
2 and Higher biotechnology
(Scotland)
Online basic and key skills
at levels 1 and 2
(England, Wales and
Northern Ireland)
Many older learners are also finding benefit in e-assessment
due to the respite this provides from traditional written
examinations, the convenience of on-demand testing and the
greater likelihood that results will be based on objective and
accurate marking. However, the full value from the learner’s
perspective can only be obtained where access to computers
is widely available and adequately supported with guidance.
Learners also need rich and timely feedback on formative
assessments and a personal online space for recording and
evidencing attainment in an e-portfolio.
‘I feel that assessment on a computer is better than writing because usually
you forget what you are going to write…before you write it. The brain works
faster than your handwriting.’
Student, University of Derby
13
Barriers and enablers in computer-based
assessment
Much of the innovative work in computer-based assessment
(CBA) in higher education has been in institutions where
systems and tools development has been seeded by
project funding. Even then, use is often limited to a few
departments. Scaling up the use of e-assessment requires
financial commitment and senior management support.
Where these exist, many of the case studies show that there
is a greater confidence in the capability of e-assessments to
deliver a return on investment, and a wider acceptance of
the benefits of change amongst staff.
Effective e-assessment for formative purposes can be set
up relatively simply by using the assessment tool within the
institution’s VLE, but a sustainable programme of technical
and pedagogic support for academic staff is still needed. In
most cases, wider-scale use has only been achieved when a
central support unit has been set up. Medium- to high-stakes
e-assessments demand a dedicated team whose role is to
ensure the reliability of systems, provide technical support
to academic staff and facilitate interoperability with other
systems in the institution. The next stage of development is to
create item banks that can be shared between departments,
or even between institutions. However, mutually compatible
interfaces adhering to universal technical standards have to
be in place to make systems fully interoperable. These
complex and costly issues have been barriers to vigorous
growth in computer-based assessment.
In the schools and further education sectors in England, the
cost of development of secure systems for use across a range
of qualifications is currently borne by awarding bodies, and
that investment is only made when there is a sound business
case. As a result, despite valuable and successful pilot
schemes, it may be some time still before online public
examinations are available across the curriculum. A
prevailing public mood of conservatism over changes to the
formal examination system is a further curb to innovation.
A different picture emerges, however, when a close-knit
community of institutions, supported by funding from
central government, and a single awarding/regulatory body
collaborate on an integrated e-learning and e-assessment
environment, as has happened in Scotland. Two initiatives –
SCHOLAR and Pass-IT – show how a partnership approach
between a university (Heriot-Watt), subject specialists, key
national agencies, colleges and schools has opened up
opportunities to empower research and development in
e-assessment.
The COLA project, however, illustrates some of the technical
barriers that had to be addressed to facilitate partnership
approaches to item bank creation and highlights the
importance of JISC work on systems design based on web
services and a service-oriented approach to facilitate the
exchange of data.
14
Around 500 online formative tests have been written and shared
since 2005.
Collaborating for change
The COLA project
Background
The COLEG Online Assessment project (COLA) is run
by Scotland’s Colleges Open Learning Exchange Group
(COLEG) – an association of further education colleges in
Scotland formed to develop and share learning resources,
including e-learning content.
With funding from the Scottish Further Education Funding
Council and subsequently the Scottish Funding Council, the
aim of COLA has been to create a bank of online formative
assessments across a wide range of courses and levels in
support of SQA qualifications. COLA assessments are
objective tests containing up to 20 questions available in 17
subject areas from Access to HND level, or Scottish Credit
and Qualifications Framework (SCQF) levels 1-8.
Technologies, systems and policies
The project generated online assessments for use in the
four VLEs used in Scottish colleges – Blackboard®, Moodle™,
Serco (Teknical) Virtual Campus™ and WebCT® – with the
assistance of subject specialists from the colleges. To
ensure that question writers had a good understanding of
objective testing and online delivery, they attended
workshops to develop best practice techniques.
This ambitious project has not been without problems,
however. The final databank was delivered in IMS Question
and Test Interoperability (QTI) standard compliant form,
but none of the VLEs complied with the IMS QTI standard
satisfactorily. The solution was that the databank should
hold only the QTI version of each question but that certain
conversion tools for different platforms (for example,
Respondus®) should be provided alongside.
At the outset it was decided to restrict the question types
to multiple choice, multiple response, true or false, and
match and fill-the-blank questions, but because of
difficulties in rendering the fill-the-blank questions,
these were excluded from the item bank. There were
also problems with user tracking in certain VLEs and
with the way in which some VLEs handled the scoring.
The accessibility of the questions could also be affected
by the way in which the individual VLE rendered them.
Rethinking assessment practice
Solutions to many of the initial problems have been
found, although the way that VLEs handle scoring is
still an issue. The experience of COLA has also
influenced some colleges to upgrade or change
their VLE.
In time, the value of the online formative tests will
outweigh the cost of resolving these issues. Already,
around 500 online formative tests have been written and
shared since 2005 and practitioners and learners have
found use of the assessments beneficial. Notably, the
project has raised the profile of e-learning in Scotland.
It has also highlighted the value of a nationally funded
drive for innovation and focused attention on the urgent
need for interoperability between systems.
15
Challenges and benefits from e-assessment tools
and systems
Adopting e-assessment has more radical implications than
just changing the mode of assessment – it can challenge
assumptions about the way the education system is
structured and change patterns of work for staff and
students.
If examinations can be taken on demand or when a tutor feels
the learner is ready – as is the case with many professional
and vocational qualifications – individuals can be fast tracked
when they excel in certain aspects of the curriculum. This
represents a considerable challenge to the formal education
system, but may increase the motivation of more young
people to stay within it. When interactive online tests are
available for formative assessment, there is evidence that
learners retake tests voluntarily and that availability of these
any time, anywhere, can help to establish more regular
patterns of study.
Interactive tests are a valuable means of engaging learners,
as has been shown by innovative work with formative
assessments on mobile devices designed for disaffected
learners4 or those studying in workplace environments.
Assessments built around virtual worlds can remove the
tedium associated with traditional examinations for all types
and age groups of learners. For many, including those with
cognitive and some physical disabilities, e-assessment can
offer a richer, more engaging, and a potentially more
valid assessment experience than paper-based testing.
The challenge is to make more use of this motivational
potential in high-stakes examinations.
4 Attewell, J (2005) Mobile technologies and learning,
www.lsda.org.uk/files/PDF/041923RS.pdf
To ensure a fully inclusive experience, objective tests must
provide built-in accessibility features that are customisable
for each candidate in advance – a challenge that will increase
in importance as e-assessment is more commonly used. This
is especially the case when teaching staff are responsible for
question authoring. Questions in item banks also need to be
assessed and indexed so that each candidate experiences a
test of equal measure to their ability, regardless of the
combination of questions presented to them in a randomly
generated test. While computer-based assessment may offer
a reduction in the time spent marking, it will for these
reasons shift the focus of effort for staff to before, rather than
after, the examination period. However, a greater variety of
methods used in assessment will undoubtedly benefit those
disadvantaged by pen and paper tests.
What is clear is that understanding of the potential of
e-assessment tools and systems needs to be matched by
awareness of their pedagogic benefits and pitfalls. Perhaps
the greatest challenge to be faced over the next decade is
choosing the best and most appropriate ways of using
these tools.
16
‘Something I always try to point out is that it is intellectually a lot more
stimulating to design an online or e-assessment than it is to sit there
marking 300 scripts.’
Professor Don Mackenzie, Senior e-Assessment Manager, Innovation 4 Learning, University of Derby
Designing for high-stakes
computer-based assessment
University of Derby
Background
A multimedia interactive e-assessment system – the
Tripartite Interactive Assessment Delivery System (TRIADS)
– was developed in house at the University of Derby in 1992
and developed further in partnership with the University of
Liverpool and The Open University. Unusually, most tests
delivered by TRIADS are medium or high stakes with a
smaller but increasing proportion of purely formative
assessments. In 2005-2006, TRIADS delivered mediumto
high-stakes assessments to 10,000 students at Derby,
measured on a single student access per assessment basis.
Technologies, systems and policies
With a Macromedia® Authorware® base, TRIADS provides
a flexible design structure with the option to build in
simulations and multimedia. The wide variety of question
designs required by different subjects has been resolved by
producing code templates for generic question styles that
may be used singly or in combination. The templates
contain full error-trapping routines and can facilitate the
award of partial credit, ie the awarding of marks for
method or for stages in an answer, as well as for final
answers. The screen layout has been developed over
many years for clarity and ease of use and in accordance
with current guidelines on accessibility.
TRIADS can also support randomised selection and
sequencing of questions and may be delivered on the
web, local area network (LAN) or CD. Detailed reporting
of candidate performance is also provided by the system,
helping to pinpoint atypical results that suggest
malpractice has occurred.
The Centre for Interactive Assessment Development
(CIAD) at the University of Derby provides support for
staff in all aspects of test production, including question
design, quality assurance, monitoring of delivery and
reporting on results.
Rethinking assessment practice
Despite the sophistication offered by TRIADS, 70% of use
occurs in the first year of courses at Derby, tailing off to 20%
in the second and 10% in the third years. So a further role
for CIAD is in widening understanding of how e-assessment
can foster the development of higher order skills and reduce
the workload for staff. ‘Something I always try to point out is
that it is intellectually a lot more stimulating to design an
online or e-assessment than it is to sit there marking 300
scripts,’ points out Professor Don Mackenzie, Senior
e-Assessment Manager. He recognises, however, that staff
may face a double burden of developing e-assessments
while still undertaking marking generated by traditional
practice, and for this reason, a centralised support unit,
CIAD, has been set up.
Measuring the frequency and timing of access to tests has
revealed interesting information on how and when students
study. This unexpected insight, derived from tracking data,
revealed that many students retake the multimedia
interactive TRIADS tests, enjoying the challenge of
competing to improve on their previous score, as in
computer games. Evidence from Derby also suggests that
students are more likely to test themselves on a regular
basis than when taking pen and paper examinations.
A TRIADS assessment. The student is required to plot a graph of the
cumulative frequency of graduate earnings, using the data supplied in
the table. The student plots the graph by clicking on each point in turn
and the system scores its position.
17
Waymarking institutional practice
At the institutional level, the impact of
e-assessment will be felt in:
• staff training and support
• quality assurance and quality enhancement
• IT infrastructure and estates
• management of candidate authentication
and plagiarism
Staff training and support
Providing guidance for academic staff in the most effective
use of authoring software and in pedagogically sound
redesign of course assessments is an especially important
consideration for higher education institutions, where
assessments are generated within individual departments.
It is worth noting that adoption of e-assessment also creates
skills development needs for those staff who support the
assessment process: learning technologists, examination
officers, invigilators and IT support staff. This, and other
support issues arising from e-assessment and e-learning,
can generate a reassessment of models of support and staff
development to clarify and realign roles and responsibilities.
Some universities have found it essential to set up a
centralised production team alongside, or as part of, the
learning and teaching enhancement unit to establish a
repository of good pedagogic practice and to mitigate the risk
of errors in delivering tests and recording and managing data,
as at the University of Derby, where a commercial unit for
e-learning and e-assessment – Innovation 4 Learning –
has also been set up in response to growing external as
well as internal demand for design consultancy and
production services.
Other models of staff support include the development of
accredited routes to staff development at the University of
Dundee, and full staff ownership of the process after initial
compulsory training at Loughborough University.
A tool for quality improvement
South East Essex College
The Technologies for Online Interoperable Assessment
(TOIA) project was funded by JISC as part of the Exchange
for Learning (X4L) Programme. Its aim was to provide
practitioners and institutions with a robust online
assessment management system that is web based,
optimised for standards compliance, customisable
and available to all UK higher and further
education institutions.
At South East Essex College, the TOIA assessment tool has
been used to improve performance in key skills at levels 1
and 2 in communication, application of number and IT by
means of rapid diagnosis of learners’ strengths and
weaknesses.
Past years’ key skills papers are uploaded to TOIA to
provide mock tests with instant marking, feedback and
reporting facilities. Tutors can access reports for each
group of learners from the staff homepage on the college
intranet, seeNet, and identify at a glance the types of
errors each learner has made. With scores immediately
available, TOIA has also reduced the marking burden
for staff.
Piloted in 2004-2005, the use of TOIA was rolled out for
college-wide use in 2005-2006, when a total of 1,769 key
skills students were added to the TOIA database. Key skills
tutors have reported increased levels of engagement and
concentration when learners have the opportunity to
complete tests online and receive immediate feedback.
18
‘e-Assessment must not be seen as
a solution looking for a problem.
It is a chance to improve the quality
of assessment for young people.’
Dr John Guy, Principal, Farnborough Sixth Form College
Quality assurance and quality enhancement
Institutions or departments involved in new approaches to
assessment need to develop appropriate quality assurance
procedures and provide codes of practice for learners and
practitioners. Examination regulations also need to be revised
for high-stakes e-assessments.
Nonetheless, e-assessment, whether diagnostic, formative
or summative, can form part of an institution’s strategy for
quality enhancement, as shown in the case study on The
Oldham College on page 34, where diagnostic assessments
are linked to online learning resources to provide a
personalised learning environment.
The case studies on Farnborough Sixth Form College and
South East Essex College in this section reveal further
ways in which computer-based – and computer-assisted –
assessments are improving the standard of current practice
and may even reveal the beginnings of a trend – one in which
well-managed e-assessment is viewed as a means of
improving quality, rather than diminishing it.
When combined with a management information system
(MIS), computer-based assessments also generate rapid,
reliable data about the progress learners are making on a
course. Given that the first step in tackling problems with
retention and achievement is to identify those who are failing
to engage, the use of data from online formative assessments
at critical stages on a course can indicate which learners are
at risk and provide prompts for remedial action.
e-Assessment, whether diagnostic,
formative and summative, can form
part of an institution’s strategy for
improving quality.
IT infrastructure and estates
Because of the high level of resources required, preparing
for e-assessment on any scale requires full managerial
and financial support and a strategy for sustainability,
including the upgrading and replacement of computer stock.
Logistically, high-volume computer-based assessment can
present challenges for institutions. Some have responded by
setting up and equipping dedicated e-assessment centres.
Often these are repurposed spaces, but an institution may be
prepared to signal its preparedness for technology-rich
learning and assessment of learning by investing in new
purpose-designed estate.
The British Standard BS7988 (now the International Standard
ISO/IEC 23988:2007 – Code of practice for the use of
information technology in the delivery of assessments)
provides guidance on implementing e-assessment for the
first time.
What are the challenges for institutions?
g Developing an e-assessment strategy that
incorporates pedagogic and technical aims
g Introducing sustainable ways of supporting
computer-based assessments
g Setting up a viable physical and IT infrastructure to
support e-assessment to the required scale
g Instituting policies and procedures to ensure the
validity of e-assessments
19
Assuring the security and validity of e-assessments has often been cited as
a concern for departmental teams and awarding bodies alike.
Management of candidate authentication
and plagiarism
Assuring the security and validity of e-assessments has often
been cited as a concern for departmental teams and awarding
bodies alike. In the further education and schools sectors,
awarding bodies’ fears of technical failure and loss of data
have decreased as the technology has become more robust.
However, the authentication of candidates taking online
assessments is still recorded as a risk, according to a 2006
survey report by Thomson Prometric on the acceptance and
usage of e-assessment for UK awarding bodies.5
It could be argued, however, that authentication is not a
problem exclusive to online examinations – a variety of ways
of verifying a candidate’s identity is likely to be needed,
regardless of the mode of assessment. Individual student
logins, backed up by photographic and other forms of
identification – including smart cards, additional codes and
passwords or biometrics, where appropriate – together with
training for e-invigilators, have proved at least as effective in
validating the identity of candidates as systems used in
traditional paper-based examinations. Cheating by digital
means during examinations – for example, by accessing the
internet or sending text messages via mobile phones – is a
further issue not confined to e-assessment, for which a
general review of examination procedures is needed.6
To authenticate the originality of work against electronic
sources, the JISC plagiarism detection software provides an
online tool which can be accessed via a standard web
browser. This is backed up by guidance on implementing
plagiarism policies from the JISC Plagiarism Advisory Service.
Electronic submission of work and routine deployment of
anti-plagiarism policies have provided some safeguards.
However, another – and arguably the most effective – method
of ensuring the integrity of submitted work is to adapt the
design of course assessments. An example is illustrated in
the case study on the BTEC Award in IT Skills for Business at
Coleg Sir Gâr on page 30, where assessment is assimilated
into the content of the course, occurring as a sequence of
short tasks which are also discussed online with tutors and
peers to provide an evidence trail.
To ensure the validity of a test, it is also important to note that
monitoring the quality of question designs and assessment
scoring methods is equally as important as the regulation of
student conduct – poor question construction, ineffective
grading of tasks and questions, or assessments that are
inaccessible to some candidates, can equally undermine the
validity of results. Effective procedures for e-assessment
begin by ensuring the most appropriate assessment design,
revising examination regulations and information given to
candidates, and then providing appropriate training for all
staff involved in invigilation and technical support.
5 Thomson (2006) Acceptance and usage of e-assessment for UK awarding bodies, tinyurl.com/yl2jfu
6 Underwood, J (2006) Digital technologies and dishonesty in examinations and tests, www.qca.org.uk/17712.html
20
Managing the virtual and physical
environment
Farnborough Sixth Form College
Background
With around 2,900 students, Farnborough Sixth Form
College is one of the largest sixth form colleges in England.
Its principal, Dr John Guy, is an advocate for the modular
16-19 curriculum, and welcomes changes to the assessment
system where they offer opportunities for improvement.
The first purpose-built e-assessment centre in an English
further education college was opened at Farnborough in
2005. The design of the facilities has been combined with
programming developed in house to ensure that computers
can be used for both teaching and learning and summative
assessment, as required, without prejudice to either activity.
Technologies, systems and policies
Purpose-built e-assessment centres represent a
considerable investment and often need to double as
learning centres. This poses challenges for network
managers who need to support routine learning and
teaching, as well as provide a secure environment for
high-stakes examinations.
Dr Mike Docker, Director of ICLT at Farnborough, rejected
options of setting up separate secured networks, or locking
down access to the standard range of software during
examination periods. His solution to maintaining flexibility
was to write a programme, Invigilator Pro, to provide a
secure environment for external examinations on any
computer in the college.
Invigilator Pro takes regular screenshots, records
all network traffic and logs every keystroke during
examinations that are invigilated by staff in the normal
way. The information is then compressed and moved
to a location on the network where it can be stored
securely and analysed in the event of suspected cheating.
In practice, however, the presence of the software
has acted as a deterrent.
Privacy screen filters have also been used to increase the
flexibility of the centre. Designed to provide privacy for
executives on journeys, the filters prevent students’ work
from being overlooked during examinations by making
screens viewed at an angle appear black. Easily added or
removed, these devices enable computers to be arranged
in collaborative oval-shaped islands, rather than in
back-to-back rows.
Rethinking assessment practice
Staff at Farnborough are also upgrading standards in
coursework using a computer-assisted model. For A-levels
in philosophy and in critical and contextual studies in art,
learners complete work on computers in supervised
timetabled classes. Copies of their work are taken by
Invigilator Pro at the start and close of each session.
The end product, however, is still printed out and posted
to the awarding body in line with current regulations.
With these initiatives, Farnborough Sixth Form College
has set a standard for the successful management of
online assessments, including the prevention of cheating
and accidental loss of data, while still maintaining normal
use of the college’s computer stock.
21
Developing strategies to support e-assessment
University of Dundee
Background
The University of Dundee has around 15,000 students
supported by around 1,000 academic staff. 150-200 of these
use formative and summative e-assessments, mostly in
multiple-choice format, but some are now incorporating
sound and video files. e-Assessment is applied most
extensively in education, law, medicine, life sciences,
nursing and languages, but is evident throughout all 15
schools in the university.
The policy for e-assessment at Dundee is supported in three
ways: departmental e-assessment performance indicators,
a five-week staff development course in online assessment,
and a centralised learning enhancement unit. A wide range
of strategies, such as online submission, peer-assessment
and plagiarism checks, are used to ensure the quality of
students’ work and their experience of e-assessment.
Technologies, systems and policies
The principal tool used for e-assessment at Dundee is
Questionmark™ Perception™. For formative assessments,
this is integrated with the VLE, Blackboard®, which itself
contains a facility for quiz production. Summative
assessments are delivered via a dedicated server running
Questionmark Perception through a secure browser.
A number of the assessments use randomised question
selection. Additionally, there is support for some ad-hoc
approaches using JavaScript™ and Perl™.
e-Portfolios are also used for assessment purposes in
some schools, and with ongoing exploration of online
communication tools, wikis, blogs, and a free text reading
tool, a broad spectrum of practice is emerging. Other
developments include an integrated system for self- and
peer-marking of work submitted online, using an in-house
system for assessment of textual answers.7 Results are
sent by email; alternatively, they may be sent by text
message to a mobile phone.
To encourage staff to acquire new skills, the university has
developed a course in online assessment which forms part
of the university’s Postgraduate Certificate in Teaching in
Higher Education and is accredited under the Scottish Credit
Accumulation and Transfer Scheme (SCOTCATS). As it is not
just the mechanics of question production that matter, but
also the value-added effect of the assessment, the course
focuses on the pedagogy of online assessment with topics
such as the effective design of questions and feedback,
strategies for integrating online assessment into face-to-face
practice, and the management of e-portfolios. The course is
delivered online and is optional, but has proved its value by
attracting applicants from outside the university.
All academics receive a copy of the university’s e-assessment
policy and procedures and the administration of summative
assessments is managed by the Director of e-Learning in the
Centre for Learning and Teaching and by staff in the Learning
Enhancement Unit (LEU). The LEU delivers training on the
design and administration of e-assessment, checks tests
before uploading to a server and reinforces the university’s
policy on e-assessment – for example, it is recommended
that any summative assessment is preceded by practice
exercises of a similar type.
Rethinking assessment practice
Changes resulting from an institution-wide e-assessment
policy can be considerable and support staff play a key role
facilitating those changes. Those at the LEU believe that
working from the ground up pays dividends – starting with
systems and servers that are fit for purpose then focusing
on support and staff training, finally ensuring the physical
environment is capable of supporting the demand. LEU staff
stress the need to work collaboratively with classroom
practitioners to successfully manage change – it is not
possible to drive through a policy, only to support its
acceptance.
22 case study
‘People often talk about the cost of e-learning, which is the
wrong attitude. It is about value and what it brings to the
institution. The people cost is substantial, but a good
investment,’ points out Dr Richard Parsons, Director of
e-Learning at Dundee.
For lecturers, e-assessment is becoming a vital asset in
managing learning in large-group contexts, but there is also
considerable value in online assessments for students. With
content available on a learning platform, lectures can be
used to refine understanding rather than to introduce a topic.
e-Assessment can then be used to confirm their grasp of the
detail without adding to staff workload.
The use of formative e-assessment at Dundee has been
instrumental in upgrading attendance and performance.
Significant improvements have been recorded in second-year
life science chemistry, for example, where self-paced
e-assessments have turned a course from one on which half
the students were failing to one on which most are obtaining
at least satisfactory results.
What are the challenges illustrated by this
case study?
g Encouraging staff to acquire new skills
g Using accredited schemes to enhance the value of
staff development in e-assessment
g Identifying appropriate uses of formative
e-assessment
study 7 www.dundee.ac.uk/learning/leu/ilt/selfpeer.htm
‘People often talk about the cost
of e-learning, which is the wrong
attitude. It is about value and what
it brings to the institution. The
people cost is substantial, but a
good investment.’
Dr Richard Parsons, Director of e-Learning,
University of Dundee
23
Waymarking academic practice
From the perspective of academic staff, key issues
arising from e-assessment include:
• impact on course structure and practice
• impact on workload
• culture change
• fitness for purpose
Impact on course structure and practice
For academic staff, e-assessment must add value to learning
and teaching to warrant the change from traditional methods
– e-assessment must enable learning objectives to be
more fully achieved, enhance learning and assessment
experiences for learners, and provide opportunities to
deepen their understanding.
Some transformative course designs have demonstrated the
potential of e-assessment in promoting more effective
learning, benefiting both staff and learners alike. For
example, the REAP project pilot at the University of
Strathclyde, described on page 38, reveals how redesigning
assessment can have a profound impact on course structure,
in this case moving the focus from transmission of knowledge
to a process more explicitly owned and managed by students,
who complete individual and group assignments using an
online discussion board. This also reduced the time academic
staff spent marking.
In a further innovative example, the case study on the BA
(Hons.) Learning, Technology and Research degree from
Ultraversity on page 33 indicates how use of an e-portfolio
as the mode of assessment can fundamentally change
approaches to learning and assessment, and with this, the
relationship between tutor and learner.
One unforeseen consequence of e-assessment may be to shift
the pedagogic focus towards aspects of practice which have
previously been less well-developed – the case study below
on the role of OpenMentor at The Open University in
improving the consistency of tutors’ feedback is an example.
Nonetheless, these case studies also illustrate how
Ensuring the quality of feedback
The Open University
The Open University (OU) is the UK’s leading distance
learning university with over 600 courses supporting
approximately 150,000 undergraduate and more than
30,000 postgraduate students. Nearly all students study
part time, with around 70% of undergraduate students in
full-time employment.
Funded by JISC, OpenMentor is an open source tool which
analyses tutors’ grading and feedback and is the result of a
collaborative project between the OU and The Robert
Gordon University.
A tutor survey identified the need to ensure that consistent
feedback is given on assignments submitted electronically.
OpenMentor uses Bales’ (1970) categories for interaction
process analysis8 to check that the comments made by
24
tutors support the marks awarded. With as many as 5000
students enrolled on an OU course, it is essential that
every student gets feedback that is appropriate to the grade
given, with a balanced combination of socio-emotive and
cognitive support.
The tool has also been used for training tutors – a display
shows the most appropriate mark for the feedback they
have given in training exercises, ensuring consistent
and balanced feedback no matter who has marked an
assignment.
OpenMentor has been tested with 100 tutors. This research
and development project is unique to The Open University
and highlights the importance of good quality feedback to
student retention and performance.
8 Bales’ Interaction Process Analysis (1970),
www.cultsock.ndirect.co.uk/MUHome/cshtml/groups/groups4.html
‘The role of pedagogically sound,
imaginative design for e-assessment
on the part of tutors is often a
significant factor in its success.’
Roadmap for e-Assessment Report for JISC
(Open University, 2006)
e-assessment has enthused practitioners, many finding new
approaches to assessment a welcome opportunity to rethink
course objectives and pedagogic models.
Impact on workload
It is well documented that the degree of effort normally
required by academic and support staff in the postassessment
period shifts with e-assessment to the
preparatory phase, when test development, establishing
equivalence with paper-based elements, test runs, and trials
of ancillary systems are a significant undertaking. The
development of effective e-assessments and associated
practices means that staff will require additional support
during a transitional phase in which they manage traditional
and new methods of assessment simultaneously.
Some institutions fund departmental e-assessment
champions to provide peer support. One such scheme has
successfully promoted e-learning (ILT) in the further
education sector, but these initiatives require mediumto
long-term rather than short-term funding. The adoption
For academic staff, e-assessment
must add value to learning and
teaching to warrant the change
from traditional methods.
of wide-scale e-assessment will undoubtedly bring an
increased workload for support staff and investment in a
centralised unit to support e-assessment development is a
likely outcome.
Culture change
Few higher education departments use e-assessment at a
summative level. This could reflect a failure to imagine what
can be achieved, as much as any real risk posed by technical
systems, now considered by many to be robust enough for
the task. Resistance to changes in cultural practice may,
however, be as potent a restriction on innovation as the
reliability of technology.
The impact of innovative assessment practice is still emerging
and will support much ongoing research. To support this
process of change, the sharing of successes and solutions to
problems is beneficial. Using case studies to review and
develop current practice is recommended.
What are the challenges for practitioners?
g Allocating time for e-assessment-related
skills development
g Ensuring the accessibility of e-assessments for
a diversity of learners
g Achieving a best fit between e-assessment design,
course objectives and the needs of learners
g Developing confidence and expertise in a full range
of e-assessment approaches
g Assimilating changes in working practices
25
Fitness for purpose
The nature of teaching and learning is changing across all
parts of the sector and assessment practices have come
under scrutiny as a result. Traditional pen and paper
examinations often fail to illuminate the existence of critical
thinking, effective decision-making, collaborative skills, or
the ability to apply to solve practical problems. The use of
technology in assessment may offer opportunities to correct
these deficiencies. However, the application must be
pedagogically led and appropriately selected – or justifiably
rejected in some cases when, for example, the course
content, the context of assessment or the ICT skills of
potential candidates places its appropriateness in doubt.
For many academic staff, the appeal of e-assessment lies in
its ability to capture aspects of learning previously considered
impossible to assess. One example is confidence-based
marking (CBM) which has shown how objective tests in higher
education – often thought to be appropriate only for low-level
knowledge testing – can assess higher order thinking in a
consistently rigorous way. CBM also promotes a deeper level
of learning by challenging students to evaluate how certain
they are of their answer, so that they address the gaps they
discover in their knowledge.
Testing knowledge acquisition through short, self-paced tests
in subjects in which the emphasis falls on accuracy of recall –
such as pharmacy, medicine and law – is an almost
guaranteed role for e-assessment. But subject disciplines
differ in terms of learning objectives and in appropriate
assessment methodologies. Computer-assisted assessment,
such as the use of bulletin board and discussion tools to
foster peer- and self-assessment, may be more suitable
applications of e-assessment in subjects in which assessment
criteria focus on the candidate’s ability to evaluate and
demonstrate conceptual thinking, rather than on the recall
of factual knowledge.
In general, e-assessment may be best used to free staff
for tasks that humans do best. Administrative functions
performed by the computer – scoring, recording, and transfer
of data – will ensure greater accuracy and objectivity than is
achieved by humans. However, a uniquely valuable attribute
that e-assessment can offer to learners and practitioners is
an authentic assessment experience. Computer-based
assessments can be taken in the workplace, acquiring an
immediate relevance. They may also replicate an authentic
context through simulations, virtual worlds or use of audio
or image files, so that the activity seems more real and
purposeful for the candidate. The use of gaming software
as a preparation for, or as part of, a summative assessment,
increases the stakes in this respect, and, where development
funding is available, has the potential to revolutionise
assessment practice.
26
‘CBM encourages students to think: “How would I justify this answer? Is
there something else that supports it out or casts doubt on it?”’
Professor Tony Gardner-Medwin, University College London
Raising the stakes with
confidence-based marking
University College London
Background
Confidence- or certainty-based marking (CBM) has been
used for ten years in the medical school at University
College London (UCL) to assist good study techniques
and to encourage deeper reflection and understanding.
It requires students to state the degree of certainty they
have for each answer in an objective test and can be used
with any type of right/wrong question and with existing
item banks.
Technologies, systems and practice
Students rate their certainty on a scale of 1-3. They gain
1, 2 or 3 marks if correct, but 0, minus 2 or minus 6 marks if
wrong, depending on the degree of certainty expressed.
The CBM questions are stored on a server, and when a
student engages with a test, the files are downloaded
from the server onto the student’s computer. As a result,
formative CBM assessments can be taken anywhere and at
any time. Summative assessments, however, are completed
under examination conditions on paper using specially
designed optical mark reader sheets.
Rethinking assessment practice
One of the perceived problems for tutors with computerbased
assessment (CBA) is that they have only the student’s
answer – written responses, online discussion or face-toface
contact may reveal the basis for the answer and
whether a student can justify it, or was merely guessing.
CBM can provide a better indication of what the student
knows, or does not know. Reliance on rote-learning is
countered because students need to accurately assess their
confidence in an answer to get good marks. In this way, CBM
can stimulate students to link ideas and to challenge their
assumptions. It can also enable examinations to gain
statistical reliability.
Students have found CBM easy to use and the immediacy of
feedback beneficial, as instant loss of marks prompts
further study where knowledge was sketchy or incorrect.
The approach can also stimulate deeper levels of thinking
even when the answer was correct, rather than merely
rewarding what could be the results of rote learning. ‘CBM
encourages students to think: “How would I justify this
answer? Is there something else that supports it out or
casts doubt on it?”’ says Professor Tony Gardner-Medwin,
who developed the system at University College London.
CBM is well suited to follow-up exercises after classes.
Mixing easy questions, for which confident answers should
be expected, with difficult questions can stimulate deeper
and more specialised lines of thought. CBM, however,
is intended to complement rather than replace other
assessment methodologies and has particular value in
subjects such as law and medicine, in which a broad
spectrum of knowledge is important.
27
case study
Preparing for summative computer-based
assessment
University of Manchester
Background
The School of Pharmacy and Pharmaceutical Sciences at
the University of Manchester had several years’ experience
in diagnostic and formative computer-based assessment
(CBA) before exploring its use for summative purposes.
Pharmacy – a factual subject with large student numbers
and an emphasis on accuracy – is a discipline that lends
itself to CBA, but a lack of established examination
protocols, the possibility of computer failure and fears for
the security of the assessments presented concerns at the
start of the six-month pilot scheme.
A total of 240 students, some with disabilities, taking first
year modules in cell biology and biochemistry were involved
in the pilot. The outcomes, published in 2006 in Pharmacy
Education,9 indicate that not only can CBA be valid and
reliable, but it can also offer advantages over traditional
methods when applied to the pharmaceutical sciences.
Technologies, systems and policies
The pilot scheme was run according to the existing
University of Manchester framework for examinations, but
consideration was also given to the SQA guidelines on online
assessment for further education.10 The assessment tool in
WebCT® – the VLE of choice at Manchester – was used to
deliver the assessments, since data from the university
student record system could be fed directly into the VLE and
used in setting up assessments. Examination questions
written by academic staff in Microsoft® Word were imported
into WebCT using Respondus® 3.0, a tool for creating tests
offline which uses a Windows® interface.
The online assessment team have since investigated
additional tools to increase the efficiency of this process –
experience has shown that import software does not always
offer a total solution at this level. For example, the use of
decimal points, commonly used in the writing of numeric
questions, caused question import filters to stumble in ways
that are difficult to predict. Hence, some final editing had to
be carried out within the VLE.
Student identity was authenticated by students logging on to
the VLE using their normal university username and
password, and was backed up by the invigilator checking that
students taking the examination could be identified from
photos on their university ID cards. Timed release was used
to ensure that examinations could be accessed only during
the timetabled period and, for additional security, a testspecific
password was issued. This was given to candidates
only when the examination commenced.
The possibility of a technical failure on the day of the
examination remained a concern. To prepare for this
eventuality, paper versions of the tests were produced as a
backup and, as a further fail-safe mechanism, candidates
were asked to enter their responses online and on an optical
mark reader sheet. This prevented the examination from
being a fully computer-based one, in which random ordering
of questions could take place, but nonetheless enabled a
useful comparison between different methodologies during
the pilot. The process relied on the assistance of a support
team, a backup server operated throughout the examination
and computers were booked at 90% capacity to allow for the
breakdown of more than one machine.
Rethinking assessment practice
The pilot study revealed that some adaptations were
necessary. A disadvantage of the WebCT marking tool,
for example, was its inability to interpret the range of
inaccuracies in spelling that could occur in otherwise correct
answers. Results from preliminary practice assessments
showed that small errors, such as the inclusion of a hyphen,
could be marked as incorrect by the computer, even though
allowed by a human marker.
28
study
As a result, a subject-specific word list of approximately
1,500 correctly spelt key terms was added as an HTML file in
WebCT. A link to the list was provided next to each question,
so candidates could cut and paste the correct spelling into
their answer box. Using this word list to ensure consistency
of spelling reduced the discrepancy between hand marking
and computer-based marking to less than 1%.
Comparisons between the methods used in the pilot indicate
that, with adaptations, computer-based summative tests can
prove more accurate than the substantially slower traditional
method in which scripts are marked manually. CBA also
proved beneficial for students with dyslexia – cascading style
sheets allowed them to customise the display using
accessibility options provided by the browser.
What are the challenges illustrated by this
case study?
g Building confidence in the potential of CBA to provide
secure and valid assessments
g Developing effective protocols for the administration
and invigilation of CBA for summative purposes
g Developing and modifying existing tools and systems
g Providing institutional support for running and
evaluating pilot schemes in summative e-assessment
9 Aojula, H, Barber, J, Cullen, R and Andrews, J (2006) Computer-based,
online summative assessment in undergraduate pharmacy teaching:
the Manchester experience, Pharmacy Education, 6(4), 229-236.
10 SQA (2003) SQA Guidelines on Online Assessment for Further Education,
www.sqa.org.uk/files_ccc/GuidelinesForOnlineAssessment(Web).pdf
Not only can CBA be valid and
reliable, but it can also offer
advantages over traditional
methods when applied to the
pharmaceutical sciences.
29
Waymarking learner-focused practice
A number of wider developments can help to
establish a more learner-centred approach to
assessment. These include:
• a technology-rich learning environment
• on-demand testing and item banks
• interoperability of systems
• e-portfolios as assessment tools
A technology-rich learning environment
Some aspects of technology are now well embedded in
classroom practice – for example, interactive whiteboards in
schools and colleges and digital resources on a VLE in higher
education institutions. Effective pedagogic models for online,
work-based and distance learning are also more fully
developed. However, a learning environment that harnesses
the potential in emerging technologies can offer much more
than this. Virtual world scenarios, simulations, mobile and
games technologies are providing new dimensions to the
concept of e-assessment.
The Altered Learning concept illustrates this groundbreaking
use of emerging technologies in assessment – in an
adaptation of the Bioware® and Atari® computer game,
Neverwinter Nights™, by tutors at West Nottinghamshire
College, players have to improve their numeracy and literacy
skills to progress. Completed tasks are then banked in an
e-portfolio for assessment, with significantly improved
success rates in basic and key skills assessments.
The portability and speed of mobile technologies can also
introduce a new perspective to assessment practice. An
example is the use by tutors of an HP iPAQ Pocket PC at
West Suffolk College to record stills, sound and video files
of trainee chefs at work. Such evidence provides a more
authentic alternative to written work in an e-portfolio, and
can also be transmitted to the tutor’s laptop via Bluetooth®
technology to illuminate discussions in theory classes.
On-demand testing and item banks
For learners in further education, a testing-when-ready policy
offers an opportunity to manage the burden of assessments,
Merging formative and summative assessment
Coleg Sir Gâr
Coleg Sir Gâr has developed an online BTEC Intermediate
and Advanced Award in IT skills for learners in small
businesses accredited by Edexcel.
Delivered via the internet with tutor support online, this is a
skills-based course which allows learners to progress at
their own pace and has no formal examinations.
Assessment is ongoing and assimilated into the structure
and content of the course – as learners complete tasks,
these provide evidence of their achievement. At the end of
each assignment and each unit, a self-assessment exercise
can be completed to obtain formative feedback.
Every possible aspect of the assessment process is
automated. Elements in an assignment are marked as
correct or incorrect – the self-assessment questions are
devised to be either right or wrong – and decisions recorded
online. Emails are sent automatically to the learner and tutor
before the results are saved in a secure part of the database.
Any wrong answers generate text advising the learner to
revisit an activity. The process can be completed in minutes
and both internal and external verifiers can access the
records. As a more formal exercise, a final tutor-marked
assignment assesses all the skills covered in the unit.
The systems used, including the VLE – Virtual College – and
the multiple-choice question tool, were developed in house
at Coleg Sir Gâr to provide rapid feedback for learners – an
essential ingredient in maintaining confidence and morale
on a distance learning course.
30
Virtual world scenarios, simulations,
mobile and games technologies are
providing new dimensions to the
concept of e-assessment.
but, importantly, also encourages them to assess their own
preparedness. Online mock tests have proved beneficial for
this, especially if they can be accessed by learners in their
own time and in different contexts, with immediate and
appropriate feedback to both correct and incorrect answers.
This is where an item bank becomes important. Item banks
supply questions of similar difficulty by a randomised
selection process, allowing candidates to sit an assessment
at different times to others in their group without
invalidating the test. However, interoperability and IPR
issues remain considerations for the development of item
banks, if tests or individual questions are shared between
institutions or subject disciplines.
Interoperability of systems
These developments bring into sharper focus the need to join up
IT systems within and between institutions. Most UK further and
higher education institutions have large and complex systems,
which have developed in an ad-hoc way. Many of these systems
are ‘monolithic’, that is, they have been developed for particular
purposes and cannot interface with other systems.
‘One of the things we found out early
on with our lifelong learners is that
they all learn at different rates and
in different ways and get deeply
frustrated if they are forced to learn
at a rate that is not their natural rate.’
Professor Tony Toole, Director of Online Services,
Coleg Sir Gâr
The JISC e-Framework aims to establish more flexible ways
for IT systems to be used to allow each element to act as a
‘service’ rather than a discrete package so that the data held
within it can be opened up for use by other elements. Put
simply, this means that the essential technologies that
support e-assessment – for example, email, learner record
systems, VLEs and assessment tools – can interoperate,
producing the speeded-up processing and reporting of results
described in some of the case studies. Interoperability
enables e-assessment to be used to its full advantage and
facilitates the interaction between e-portfolios and the
services they draw on to evidence lifelong learning.
Key to the achievement of this vision for a joined up learning
system is the development of common standards. These
smooth the path to updating accessibility features in software
and enable, amongst other requirements, e-assessment
questions to be shared between item banks. The IMS QTI
specification already provides a basis for this to happen,
if the systems used are QTI compliant.
What are the challenges for learner-focused
practice?
g Issues of interoperability, copyright and IPR relating
to item bank development
g Adoption of common technical standards to facilitate
interoperability between systems
g Managing the availability of on-demand testing
g Harnessing the potential of new technologies in
assessment practice
31
e-Portfolios as assessment tools
An e-portfolio is defined in Harnessing Technology (DfES, 2005)
as an electronic means of recording personal learning and
achievement supported by reflective activities through which
learners can gain greater understanding of their development
and progress over time.
In this interpretation, the key purpose of an e-portfolio is
developmental – it provides opportunities to reflect on stages
in a lifelong learning journey and to plan ahead. It may well
draw on evidence generated on different programmes and in
different institutions, posing issues of interoperability and
long-term ownership and storage.
However, an e-portfolio can be used as tool for assessment
wherever a repository of evidence, combined with a reflective
analysis of the process and outcomes, is prepared for
assessment and validation against a set of external criteria.
While still considered in many ways innovative, use of an
e-portfolio for assessment has for some time been a feature
of a number of vocational programmes.
Using an e-portfolio as a tool for assessment introduces
an important shift in approach, in that the route to the
qualification no longer takes the form of universally applied
tasks. Learners may select what is to be included in the
e-portfolio and also choose the format and media to be used.
Thus e-portfolios have the potential to introduce a degree of
personalisation into assessment. They also demonstrate
additional personal skills that are valuable in the workplace –
for example, command of software, use of web technologies
and digital images – as well as recording achievement of
course objectives.
Learners’ ownership of the process is a distinguishing feature
of this mode of assessment. For the BA (Hons.) Learning,
Technology and Research degree offered by Ultraversity,
students exhibit the findings from research undertaken in
their workplace for critical feedback from their colleagues.
Their findings, together with the feedback, are then presented
for assessment in an e-portfolio, demonstrating how a
mode of assessment can capture the process as well as
the outcomes of learning. The students’ experience of
assessment is more authentic, since much is based on
their own experience of the workplace, so for many students
this approach is empowering as well as demanding.
It also develops skills valuable in a 21st century workforce:
communication, problem-solving, presentation
and collaboration.
From a marker’s perspective, however, this is a potentially
complex and time-consuming mode of assessment, as noted
in the UK Centre for Recording Achievement (CRA) study of
efficient assessment of portfolios.11 On the other hand,
evidence assembled in an e-portfolio gives employers a more
rounded picture of a learner’s achievement, and how it was
achieved – it is a particularly relevant mode of assessment
for vocational or work-based courses.
11 Strivens, J (2006) Efficient Assessment of Portfolios,
http://cetl.open.ac.uk/pbpl/pics/d58920.pdf
32
‘That’s the beauty of this degree, it’s “forcing” me to be brave and do
things that I feel I cannot do, but really I can. You’ve given me ownership
of my learning and…I’m hanging on to it.’
Kath Marshall, third-year student
The e-portfolio route to
accreditation
Ultraversity, Anglia Ruskin University
Background
The BA (Hons.) Learning, Technology and Research (BALTR)
degree course at Ultraversity – a distance learning arm of
Anglia Ruskin University – is delivered and assessed without
a single face-to-face meeting before graduation. Designed
by Ultralab, the course supports a wide variety of learners
studying from home and from the workplace. The first
cohort of 140 students graduated in summer 2006.
Technologies, systems and policies
BALTR is designed as a research-based degree. Modules
depend on action research rather than on course content,
and include development of the skills needed to undertake
research enquiries. Students, or researchers as they are
known, take charge of their own programme of learning,
negotiating with their personal facilitator a range of
activities based around their work environment.
They upload evidence of learning outcomes into their own
portfolio space on Plone™, a platform selected because it is
open source and considered simple to use and flexible.
For Ultralab project manager, Stephen Powell, the course
ethos of personalised and negotiable learning was the
deciding factor behind the choice of software. FirstClass®
– a conferencing software which facilitates group
collaboration and document sharing – is used alongside
Plone to establish an online community to provide a
network of support and to enable peer review of outcomes.
From the outset, researchers take ownership of the
technologies they use and are encouraged to set up
their own weblogs and use other emerging technologies
outside of the conferencing software to reflect on the
progress of their learning. They can choose to keep their
space on Plone private or can upload resources, such as
images or podcasts, for peer review – a comment box on
Plone offers the opportunity for this if peer-assessment
is selected as a learning pathway by the researcher.
The decision is theirs.
Rethinking assessment practice
Those opting for the BALTR degree are typically 35-40
years old and unable to commit to traditional face-to-face
courses. The design of the assessment allows them
greater flexibility in how they demonstrate their learning,
and, in doing so, provides a sense of empowerment.
‘That’s the beauty of this degree, it’s “forcing” me to be
brave and do things that I feel I cannot do, but really I can,’
was the feedback from one third-year student.
Their research may also make a lasting impact on the
way their workplace operates – at the end of the course,
researchers exhibit their findings for critical feedback from
a selected audience. The critical feedback and outcomes
from the exhibition are then assessed in a ‘patchwork’ of
evidence connected by a text or audio commentary within
the e-portfolio.
33
case study
Supporting learners through diagnostic
assessment
The Oldham College
Background
The Oldham College has improved learner achievement
through a blend of online diagnostic assessment and
personalised learning resources. Diagnostic and mock
online tests for basic and key skills were introduced in 2005
on the college VLE, Serco (Teknical) Virtual Campus™, with
top-up resources provided alongside to target specific
learning needs.
Around 30,000 individual online assessments and 3,000
nationally recognised online qualifications were completed
by learners at The Oldham College during 2005-2006. The
system has been established over two years and is credited
with having raised achievement on basic and key skills by
18% in the year 2005-2006. The cost of the scheme has been
partially offset by the higher rate of course completions.
Technologies, systems and policies
A designated e-assessment centre was set up in 2005 to
provide a venue for summative tests, including those for
City & Guilds, Edexcel and OCR qualifications taken
throughout the year, as well as providing accommodation
for mock and diagnostic tests. The e-assessment centre
is located in the library and learning resource centre,
signalling the close synergy between learning and
assessment. A testing-when-ready policy ensures that
the centre is in continuous use.
All enrolled learners undergo five diagnostic screening
tests on entry and take mock online tests before undergoing
formal assessments in basic and key skills. The results are
available within days via learners’ home pages on the VLE
and indicated by traffic light colours – green for ‘achieved’
and red for ‘not achieved’. Scores are automatically
transferred into the college management information
system (MIS) to create an individual learning record, and to
generate class lists for teaching staff. EasyOCIS – a system
built in house as a front end to the Fretwell-Downing MIS –
enrols learners on mini virtual courses on the basis of test
results. Personalised learning resources that are appropriate
for those results are then generated.
For example, a learner may achieve a pass score in a basic
skills numeracy test, but the test may reveal an underlying
uncertainty about fractions. Once the result is delivered,
appropriate learning resources and quizzes appear as links
on the home page when the learner next logs in. These will
have been uploaded as IMS-compliant objects into the VLE.
The results of the screening will also generate different
levels of support for different learners, from one-hour per
week group sessions in the learning resources centre, to
one-to-one support in the classroom if required – the college
provides about 6000 support hours per annum. Learners’
progress can also be tracked via EasyOCIS.
Rethinking assessment practice
Over 2,000 learning resources were created in support of
the scheme over an 18-month period, using re-purposed
National Learning Network (NLN) materials. The creation of
such a high number of online resources was initially timeconsuming
and required the services of a full-time
administrator. However, the benefits are now visible.
These have included a raised profile for the library and
learning resources staff, who are now more likely to be
approached for assistance in locating suitable resources.
Practitioners were able to have guidance from the
administrator in the production of online quizzes – an
unanticipated benefit from the initiative has been the
improvement in staff ICT skills. While the quizzes produced
so far have been mainly multiple choice, sound files and
34
study
digital images are more frequently used, reflecting the
increasing confidence of staff in developing their own
formative e-assessments.
For learners, fast tracking has now been made possible by
the provision of personalised routes to qualifications. Some
learners will even complete their basic and key skills tests
within two months of enrolling, if the initial screening
indicated this was appropriate. Learners can then progress
at their own rate – something that is important both in terms
of maintaining morale and in managing the high throughput.
‘We know we couldn’t do this in a traditional way,’ says
Head of Learning Resources, Roger Clegg, who has led the
initiative to produce the top-up online learning resources that
target weaknesses identified by the tests.
The scheme is now being extended to foster achievement
in other curriculum areas and to support learners in the
workplace. Resources in a wider range of media are
envisaged to promote self- and peer-assessment – for
example, the use of videos created by learners to explore
their own and others’ performance for elements of the BTEC
National Diploma in dance.
What are the challenges illustrated by
this case study?
g Exploring the role of e-assessment in supporting
personalised learning
g Funding the development of individualised learning
resources to raise learners’ achievement
g Aligning institutional strategies for e-learning and
e-assessment to ensure that diagnostic assessments
lead to supported learning experiences
‘We know we couldn’t do this in
a traditional way.’
Roger Clegg, Head of Learning Resources,
The Oldham College
35
Looking to the horizon
Much that has already happened in the field of e-assessment
would not have been predicted ten years ago, so how can we
envisage changes to come in the next decade? This section
questions some assumptions about e-assessment and
explores the most likely developments on the horizon for
practice with e-assessment.
Which aspects of e-assessment will be
established as mainstream practice by 2017?
In the assessment of the 14-19 curriculum, much will change.
Traditional paper-based summative assessments will
continue to migrate to computer delivery, led by new
qualifications, such as the cross-curricular Specialised
Diplomas, currently under development in England.
Increasingly, aspects of courses that lend themselves to
objective question types, or that use assessments based on
visualisations of concepts or procedures, will be completed
online. These strategies are likely to be combined with shortanswer
questions, marked by computer and checked by
humans, to probe the learner’s ability to form links between
areas of knowledge.
The value placed on traditional literacy is not likely to yield.
Nonetheless, assessment will become a more varied and
richer experience which will include some computer-based
or computer-assisted assessment in tandem with other
approaches. All 16 to 19-year olds will have used technology
as part of their assessed programme of study and will be
proficient in its use. Evidence of their creative achievements
or wider key or core skills is likely to be required in
multimedia formats, either in an e-portfolio record of
achievement, or as part of a formal assessment. Dependence
by UCAS on A-levels as a determinant of entry to higher
education is also likely to be reduced by 2017, in part due to
the availability of information about applicants’ interests,
achievements and performance provided in e-portfolios.
Universities, driven by increased participation in higher
education courses, will continue to undertake research and
development activities related to e-assessment, including
some groundbreaking work in the use of virtual world
scenarios for summative and formative assessments.
Written assignments will routinely be submitted and
marked electronically to combat plagiarism and to provide
retrievable evidence of previous levels of attainment.
Mobile and audiovisual technologies will play a more
prominent role in the capturing of evidence from the
workplace or from enquiry-based and work-based learning, in
many cases for submission via an e-portfolio. The assessment
of professional skills – for example, in clinical practice in
medical and pharmaceutical sciences, health and social care
– will have been advanced by the exploration of digital video,
video conferencing and mobile communication devices to
improve the validity and authenticity of practice-based
assessments.
What will be the key achievements by 2017?
The introduction of technology will have prompted a greater
understanding of how to design effective assessment
strategies. By 2017, a drive for inclusivity and personalisation
in assessment will have challenged the ‘one-size-fits-all’
assumption that dominated assessment practices in the
20th century.
36
It will be accepted as effective practice that opportunities
must be provided for all learners to demonstrate their
achievement. As a result, computer-based and computerassisted
assessments will form part of a wider portfolio of
methodologies from which assessment designers can choose.
These will include vivas, practice-based assessments,
e-portfolios, group tasks, self-directed enquiry-based
assignments, peer- and self-evaluation, as well as traditional
essay-based examinations. As a result, it will become
possible for a wider diversity of learners to achieve the
qualifications of their choice, demonstrating their attainment
in ways that are more relevant to, and more revealing of, their
level of skill and attainment.
An increase in distributed learning on a global scale, as
higher education institutions forge partnerships with
universities in the emerging world, will also encourage a
continuing re-examination of the value systems and practices
inherent in assessment.
Simulations in which both learning and assessment of
learning can take place will have demonstrated the value of
aligning learning objectives with assessment methodologies.
One such example is the second phase of development of the
Transactional Learning Environment (TLE), underway at the
University of Strathclyde.12 This simulated learning
environment allows postgraduate law students to rehearse
skills in legal transaction and negotiation in a virtual town,
Ardcalloch, supported by databases of legal documents.
Students learn from and are assessed by what they do –
in the safety of a virtual world. The TLE is transferable to any
professional discipline within higher and further education
and, by 2017, will no longer seem an innovative way of
developing and assessing professional skills.
What concerns will remain by 2017?
A drive to introduce e-assessment presupposes that
candidates are competent in, and willing to engage with ICT.
For some learners, this could create a new layer of
disadvantage, if they feel unable or unwilling to participate in
courses and assessments that take place in a computermediated
environment.
The increased diversity of approach will continue to fuel a
debate over standards. A flexible and responsive assessment
regime brings with it issues of equivalence between different
methods of assessment – for example, is it as hard to
complete a multiple-choice test as it is to write a 1,000-word
essay under examination conditions? Can there be parity
between qualifications that have been assessed in strikingly
different ways?
For a vision for e-assessment to be realised with more
certainty, it is necessary to explore more fully the
institutional and pedagogic dimensions to e-assessment,
addressing issues of sustainability and culture, as well as
technological barriers. We need to know, for example, how
technology can be utilised to enhance assessment practices,
rather than simply make them more efficient, and how to
deploy e-assessment alongside other modes, so that each
complements the other in the range and type of skill
assessed. Finally, we need to explore how best to develop the
confidence of all concerned in learning and teaching in the
efficacy and appropriateness of computers in assessment.
12 www.jisc.ac.uk/whatwedo/programmes/elearning_innovation/eli_simple.aspx
37
‘I found it very beneficial, at the time… I did not realise how much
I was learning.’
Student, basic psychology course, University of Strathclyde
Understanding the role of
assessment in learning
The REAP project, University of Strathclyde
Background
Traditional assessments result in a high workload for tutors,
and yet may do little to encourage learners or improve their
ability to learn. The Re-engineering Assessment Practices
(REAP) project, based at the Centre for Academic Practice
and Learning Enhancement at the University of Strathclyde,
is exploring enhanced assessment practice across a broad
spectrum of pilot schemes, with special emphasis on large
first-year undergraduate classes.
Technologies, systems and practice
One pilot has been undertaken with the first-year basic
psychology course at the University of Strathclyde,
which introduces students to key theories and debates
in this popular field of study. With a class size of
approximately 550, opportunities for feedback on
assessments were limited.
Course assessments were redesigned for the pilot using
online communication tools. Students were divided into
groups of six and asked to use the assignment drop box and
the discussion board in the university’s VLE. After an initial
induction task, students posted responses to tasks of
increasing difficulty in the same topic area over a four-week
period, the length of responses ranging from 50 to 300
words.
The first postings required an individual submission
followed by discussion with other group members before an
agreed group posting. Subsequent tasks involved longer
shared answers based on group discussions. In all cases,
after submitting their answer, students were able to
compare it with those produced by other students, engaging
in individual thought, peer dialogue and self-assessment
in relation to these tasks. Tutors monitored and allocated
marks for participation but did not moderate the
online discussions.
Rethinking assessment practice
The scheme was based on work by Nicol and Macfarlane-
Dick (2006)13 which identifies seven principles of effective
feedback. Formative assessments that demonstrate the
application of these seven principles can improve all
students’ ability to learn autonomously by developing
skills in evaluating their own and others’ work.
Other pedagogic features of this re-design of learning
and assessment were task scaffolding (building on
knowledge and comprehension established in earlier
tasks to support later task engagement) and social
scaffolding (individual learning supported by peer
dialogue and interaction).
The pilot demonstrated how a course structure, aligned
with the use of readily available technology and the seven
principles of effective feedback, can transform the social
dynamics of learning and improve students’ sense of
ownership of course content. The results have shown
enhanced engagement, confidence and autonomy in
learning, as well as deeper processing of content
by students.
As a result, the Department of Psychology has redesigned
the structure of first-year classes – in 2006-7, the number
of scheduled lectures has been halved and similar online
group work now spans all topic areas. Student feedback
also indicates a positive effect: ‘I found it very beneficial,
at the time… I did not realise how much I was learning…
it was learning without thinking about what I was doing.’
13 Nicol, D J and Macfarlane-Dick, D (2006) Formative assessment and self-regulated learning: a model and seven principles of good feedback
practice, Studies in Higher Education, 31(2), 199–218
38
It is estimated that around 70-80,000 e-assessments took place at
Loughborough University in 2006.
Embedding an institutional policy
for e-assessment
Loughborough University
Background
Loughborough University has been developing and refining
e-assessment practice since 1996. Large-scale deployment
of e-assessment is now a reality rather than a vision – it is
estimated that around 70-80,000 e-assessments took place
at Loughborough University in 2006.
Technologies, systems and policies
Developments have been driven, as elsewhere in the sector,
by the desire to reduce marking workload, while also
achieving appropriate and rapid feedback for students.
The focus of work has been on three fronts: increasing the
efficiency and usability of systems, responding to a need for
greater sophistication in questioning techniques, and
dealing with the logistical issues arising from increasing
demand for e-assessment.
For these reasons, Loughborough uses both computerbased
and computer-assisted assessments. Paper-based
optical data capture, in which scripts are scanned and
digitised, occurs alongside computer-based assessments
developed and delivered though Questionmark™
Perception™.
Rethinking assessment practice
Loughborough University has used Questionmark
Perception since 1999. As part of an earlier JISC-funded
project, Questionmark Perception was embedded into an
emerging managed learning environment with single login.
Each student then received an individualised menu of
assessments depending on their course registrations.
This system is used for objective diagnostic, formative and
low-stakes summative assessments. Being web-based, the
assessments can be accessed from anywhere and at any
time. Major users have been modern foreign languages,
chemistry, mathematics and engineering departments.
In a range of strategies for computer-based assessment,
objective tests have been used in conjunction with case
studies to assess responses to complex scenarios, online
surveys to establish levels of knowledge prior to the course,
and confidence-based marking to test the level of certainty
a student has in a selected answer.
However, increased numbers have meant that computerassisted
approaches using optical mark readers also play a
vital part in a holistic policy for e-assessment. A system has
been devised for optical data capture using Remark Classic
OMR® software and has been refined over time to provide
opportunities to assess higher order skills. This paperbased
solution avoids the need for large-scale, high-security
IT suites and complex contingency plans. It is also popular
with staff and students.
An e-assessment system, however, is only as good as
the content on it and the vision and skill of its users.
Appropriate compulsory training for users, combined with
clear definitions of roles and responsibilities across teams
and departments, are also considered by Loughborough
University to be key to the successful implementation
of e-assessment.
An example of a fill-in-the-blanks question from a statistics and
chemometrics test devised at Loughborough University.
39
Working towards effective practice
A tool for implementing effective practice with e-assessment
Use this checklist to explore and discuss with others the implications of implementing
e-assessment in your context.
Issues to consider Working towards effective practice
Purpose
(reason for implementing
e-assessment; alignment
with e-learning and
other strategies).
Consider the rationale for e-assessment and internal and external evidence (including
case studies) that supports it. Review how the proposal aligns with related institutional
strategies.
Learners
(their prior experience, ICT
skills, rights, responsibilities;
and access to resources and
feedback; the appropriateness
of the approach taken).
Consider the appropriateness of the pedagogic approach and likely benefits for
learners.
Review the availability for learners of technologies, digital resources, ICT support and
codes of practice.
Consider accessibility and regulatory requirements.
Context
(face to face or virtual;
tools, facilities and services;
staffing requirements; IT
infrastructure; support and
training for staff).
Identify adaptations needed to the IT and physical infrastructure.
Consider strategies for sustaining and updating the IT and physical infrastructure.
Review models for staff training and support.
Figure 4 A tool for implementing effective practice with e-assessment
40
Issues to consider Working towards effective practice
Development
(suitability and innovation in
assessment designs;
compliance with standards and
quality assurance procedures;
accredited routes for
professional development;
interoperability of systems).
Consider the design, pedagogic approach and ability to assess a diversity of skills,
including those not covered by traditional methods.
Review quality assurance procedures and how they relate to the design and
development of e-assessment, e-portfolios, item banks and the use of multimedia in
e-assessment.
Consider the professional development needs of all staff involved, including learning
technologists, IT support teams, administrators and invigilators.
Review the suitability and compatibility of technologies used with other institutional
systems. Explore adaptations to tools and systems to improve security and efficiency.
Implementation
(contingency plans; choice of
and adaptations to systems;
authentication procedures;).
Review the outcomes of pilot schemes.
Explore protocols and procedures for authenticating candidates and ensuring the
security of summative e-assessments.
Review contingency plans for summative assessments. Consider the quality and
immediacy of feedback in formative assessments.
Evaluation
(measuring achievement
of objectives).
Identify how achievement of objectives will be measured. Review methods for obtaining
evaluation data.
41

Glossary of key terms
Accessibility: The conditions needed to ensure that all
learners, including those with disabilities and learning
difficulties, can access places of learning and learning
resources, including e-learning content.
Audience response systems: A means of introducing
interactivity and assessment of understanding during
presentations and lectures which uses wireless handsets and
a receiver connected to a computer to capture audience
responses to questions. Designated software converts the
signals into displays for whole-group discussion.
Authentic assessment: An assessment that places candidates
in a real-life or simulated scenario that requires them to
apply appropriate knowledge and skills.
Awarding body: An organisation or consortium that awards
qualifications. To be eligible to award accredited qualifications
in the non-HE sectors, awarding bodies must meet the
requirements of the regulatory authorities.
Blog: A shortened form of ‘web log’ – a web page containing
periodic entries compiled by either an individual author or as
a collaborative exercise by a group within a community of
practice. For further information,
see: http://en.wikipedia.org/wiki/Weblog
Computer-assisted assessment (CAA): Used in this
publication to refer to assessment practice that relies in part
on computers, but also used in a generic sense, broadly
synonymous with e-assessment.
Computer-based assessment (CBA): Used in this publication
to refer to assessments wholly delivered and marked by
computer, but also used in a generic sense, broadly
synonymous with e-assessment.
Confidence-based marking (CBM): Measurement of the
extent to which a candidate believes his/her answer to a
question is correct.
e-Assessment: The end-to-end electronic assessment
processes where ICT is used for the presentation of
assessment activity and the recording of responses. This
includes the end-to-end assessment process from the
perspective of learners, tutors, learning establishments,
awarding bodies and regulators, and the general public.
e-Learning: The process of learning which is supported by the
use of ICT. Also used loosely to describe the actual content
delivered on screen, and the more general use of ICT to
contribute to learning processes.
e-Framework: An international effort to develop a serviceorientated
approach to the development and integration of
computer systems in the sphere of learning, research and
education administration.
e-Portfolio: An electronically-based portfolio which can act as
a repository of files including reflection and self-evaluation
assembled for the assessment of a particular course. An eportfolio
may also be used as a complete record of lifelong
learning and achievement.
Free text reading tool: Software that enables candidates’
responses to essay-style questions (typed rather than
handwritten) to be analysed and marked electronically.
High-stakes assessment: One in which the outcomes are of
high importance to both the centre and to candidates and
affect progression to subsequent roles and activities.
ILT: An abbreviation for ‘information and learning technology’,
the term commonly used in further education to refer to the
application of technology to the institution’s core functions.
IMS: The IMS Global Learning Consortium Inc., the USAbased
agency developing specifications for educational
computer systems in education.
Intellectual property rights (IPR): Rights granted to creators
and owners of works that are the results of human
intellectual creativity. For further information, see
www.jisclegal.ac.uk/ipr/IntellectualProperty.htm
Interoperability: A feature of computer systems components
that allows the components to interact according to technical
standards that define functionality useful to the user. The IMS
QTI specification is an example of an interoperability
specification within the e-assessment domain.
Intranet: A network of computers within an organisation,
which functions (from a user’s perspective) similarly to the
internet, potentially providing additional services to users,
while also preventing unauthorised external access.
43
Item bank: A storage facility for items, which allows them to
be maintained and used for automatic and manual test
generation purposes (to create tests on paper and/or on
screen).
Learning platform: A generic term covering a variety of ICT
systems that support online learning.
Local area network (LAN): A computer network in a single
physical location, which is characterised by high-speed highreliability
interconnections between client computers and
servers. A LAN with a local server is regarded as the most
reliable infrastructure for running e-assessments.
Login: The unique user name and password entered to access
a computer system.
Low-stakes assessment: One which is of low importance,
with results recorded locally.
Management information system (MIS): A computer system
used in educational institutions which stores administrative
information (such as student administrative records and
financial information) about the enterprise, its staff, learners
and programmes.
Medium-stakes assessment: One in which results are
recorded locally and nationally, but is not life-changing.
Online communication tools: A generic term to refer to
asynchronous forms of electronic communication. These
include instant messaging, chat, email, blogs, wikis and use
of discussion forums in VLEs.
Objective tests: Tests containing questions to which the
response can be marked right or wrong without the need for
expert human judgement.
Open source: Software applications and components for
which the source code of the application is made available to
customers so that they can maintain and modify the
application themselves.
Optical mark reader: A device that scans paper-based tests
and converts marks made by the student using pen or pencil
into digital data.
Personalisation: The configuring of a system by students to
suit their personal requirements, including more complex
customisations of the user experience to meet personal
learning needs.
Question and Test Interoperability (QTI): Specification for
tests and items which allows these to be authored and
delivered on multiple systems interchangeably. It is designed
to facilitate interoperability between systems.
Randomised question selection: The selection of individual
questions from a predefined set. In e-assessment,
randomisation is used to generate alternate test forms from
an item bank. It can also be used to alter the sequence in
which items are presented to different candidates, or the
order of distractors.
Regulatory authority: Government-designated statutory
organisations required to establish national standards for
qualifications and secure consistent compliance with them.
Service-oriented approach: An approach to the design of
applications in which each becomes a ‘web service’ so that
other applications or services can use it.
Simulation: An imitation of a real phenomenon with a set of
mathematical formulas and rules in such a way that a student
can interact with it – an experimental model of reality.
Technical standards: A publicly available definition of a
hardware or software component. For a range of systems to
be able to work together and exchange data, they must
conform to a set of common technical standards.
Unique learner number: An ongoing English project run by
the DfES to provide each learner with a unique identifier in
order to improve administration processes. In Scotland, the
equivalent is the SCN – Scottish Candidate Number.
Virtual learning environment (VLE): A set of learning and
teaching tools based on networked computer resources which
provide a focus for students’ learning activities and the
management and facilitation of those activities, along with the
content and resources required to help make the activities
successful.
Virtual world scenarios: Scenarios in a computer-simulated
environment for users to inhabit and interact with. For further
information, see: http://en.wikipedia.org/wiki/Virtual_World
Wiki: A type of website that allows the visitor to easily add,
remove, and edit content, making it an effective tool for
collaborative authoring. For further information,
see: http://en.wikipedia.org/wiki/Wiki
44
Further information
For further information about the organisations, publications,
software and products mentioned in this review:
JISC
For further information about JISC www.jisc.ac.uk
For further information about JISC work in e-assessment
www.jisc.ac.uk/assessment.html
JISC publications on e-assessment and
e-portfolios
e-Assessment: An overview of JISC activities (JISC, 2006)
www.jisc.ac.uk/uploaded_documents/ACFC6B.pdf
e-Assessment Glossary (JISC/QCA, 2006)
www.jisc.ac.uk/assessment.html
e-Portfolios: What institutions really need to know (JISC, 2006)
www.jisc.ac.uk/uploaded_documents/JISC-BP-ePortfoliov1-
final.pdf
Roadmap for e-Assessment Report for JISC
(Open University, 2006)
www.jisc.ac.uk/assessment.html
JISC services offering guidance on issues
related to e-assessment
Cetis – Assessment Special Interest Group
http://assessment.cetis.ac.uk
JISC infoNet www.jiscinfonet.ac.uk
JISC Legal Information Service www.jisclegal.ac.uk
Netskills www.netskills.ac.uk
JISC Plagiarism Advisory Service www.jiscpas.ac.uk
JISC Regional Support Centres (RSCs)
www.jisc.ac.uk/whatwedo/services/as_rsc/rsc_home.aspx
TechDis www.techdis.ac.uk
e-Assessment tools and resources funded
by JISC
FREMA www.frema.ecs.soton.ac.uk
IBIS www.toia.ac.uk/ibis
OpenMentor http://openmentor.comp.rgu.ac.uk
TOIA www.toia.ac.uk
Other tools and software
EasyOCIS www.oldham.ac.uk/easyocis
FirstClass® www.firstclass.com
HP iPAQ Pocket PC
www.hp.com/country/us/en/prodserv/handheld.html
Invigilator Pro
www.farnboroughsfc.ac.uk/invigilator_pro/index.html
JavaScript™ http://javascript.internet.com
Macromedia® Authorware®
www.adobe.com/products/authorware
Macromedia® Flash®
www.adobe.com/products/flash/flashpro
Plone™ http://plone.org
Questionmark™ Perception™ www.qmark.co.uk
Remark Classic OMR® www.gravic.com/remark
Respondus® www.respondus.com
Perl™ www.perl.org
TRIADS www.derby.ac.uk/assess/newdemo/mainmenu.html
45
VLEs
Blackboard® www.blackboard.com
Fretwell Downing (now part of Tribal Group plc)
www.fdgroup.co.uk
Moodle™ http://moodle.org
Serco (Teknical) Virtual Campus™ www.teknical.com
WebCT® www.webct.com
Qualifications
BA (Hons) Learning, Technology and Research
ww3.ultralab.net/projects/ultraversity
BTEC Intermediate and Advanced Award in IT Skills
for Business
www.colegsirgar.ac.uk/english/courses/search?view=647
DiDA http://dida.edexcel.org.uk
GCE Moving Image Arts www.ccea.org.uk/movingimagearts
National Unit Assessments www.sqa.org.uk
Specialised Diplomas
www.dfes.gov.uk/14-19/index.cfm?sid=3
UK awarding and regulatory bodies
Assessment and Qualifications Alliance (AQA)
www.aqa.org.uk
City & Guilds Institute www.city-and-guilds.co.uk
Council for the Curriculum Examinations and Assessment
(CCEA) – Northern Ireland www.ccea.org.uk
Department for Education and Lifelong Learning and Skills
(DELLS) – Wales
http://new.wales.gov.uk/topics/educationandskills
Edexcel www.edexcel.org.uk
OCR www.ocr.org.uk
Qualifications and Curriculum Authority (QCA) – England
www.qca.org.uk
Scottish Qualifications Authority (SQA) – Scotland
www.sqa.org.uk
Wider initiatives and projects
Accessibility in e-Assessment Guidelines (Edexcel, 2006)
www.techdis.ac.uk/index.php?p=9_1_11
Altered Learning www.alteredlearning.com
BS ISO/IEC 23988:2007
www.bsi-global.com/en/Standards-and-Publications/
Industry-Sectors/ICT
Confidence-based marking www.ucl.ac.uk/lapt
Colleges Open Learning Exchange Group (COLEG) Online
Assessments (COLA) www.coleg.org.uk/coleg
Glow www.glowscotland.org.uk
IMS Question and Test Interoperability (QTI) Standard
www.imsglobal.org/question
Innovation 4 Learning www.i4learn.co.uk
Key Stage 3 ICT www.qca.org.uk/7280.html
Lifelong Learning UK (LLUK) www.lluk.org
National Learning Network (NLN) www.nln.ac.uk
Pass-IT www.pass-it.org.uk
Project e-Scape www.qca.org.uk/14830.html
REAP project www.reap.ac.uk
SCHOLAR http://scholar.hw.ac.uk
For further research
Becta www.becta.org.uk
Centre for Recording Achievement (CRA)
www.recordingachievement.org
Harnessing Technology (DfES, 2005)
www.dfes.gov.uk/publications/e-strategy
HEFCE Strategy for e-Learning
www.hefce.ac.uk/pubs/HEFCE/2005/05_12
Higher Education Academy www.heacademy.ac.uk
46

Acknowledgements
This publication has been informed by the work, contributions
and support of others.
Acknowledgements are due to Dr Denise Whitelock and
Andrew Brasher, The Open University, for the Roadmap for
e-Assessment Report for JISC, and to the teams from The
Open University and the University of Derby that researched a
number of the case studies in this review of current practice.
Other contributors include Martin Ripley, of Martin Ripley Ltd,
and Martyn Roads, of MR Educational Consultants Ltd.
A review panel has guided the production of this publication
and their time and support have been much appreciated:
Dr Simon Ball, TechDis
Malcolm Batchelor, JISC
Amanda Black, Becta
Greg Benfield, Oxford Brookes University
Sarah Knight, JISC
Lou McGill, JISC
Janet Strivens, University of Liverpool
Professor Tony Toole, University of Glamorgan
Sincere thanks are also due to the following for their
assistance in the research for the publication:
Dr Julie Andrews, University of Manchester
Clive Church, Edexcel
Roger Clegg, The Oldham College
Patrick Craven, OCR
Mike Dawe, City & Guilds Institute
Myles Danson, JISC (formerly Loughborough University)
Dr Mike Docker, Farnborough Sixth Form College
Professor Tony Gardner-Medwin, University College London
Dr John Guy, Farnborough Sixth Form College
Richard Hanna, CCEA
Jim Harris, South East Essex College
Mary MacDonald, Colleges Open Exchange Group (COLEG)
Professor Don Mackenzie, University of Derby
Dr David Nicol, REAP Project, University of Strathclyde
Paul Oliver, DELLS
Dr Richard Parsons, University of Dundee
Bob Penrose, AQA
Stephen Powell, Ultraversity, Anglia Ruskin University
Jalshan Sabir, University of Strathclyde
Christine Wood, SQA
Images reproduced by kind permission of the City & Guilds
Institute, Farnborough Sixth Form College, The Oldham
College, and the following universities: Anglia Ruskin, Derby,
Dundee, Loughborough and Manchester.
48

Effective Practice with e-Assessment
Further information
Web: www.jisc.ac.uk
Email: [email protected]
Tel: +44 (0)117 33 10789

الوقت من ذهب

اذكر الله


المصحف الالكتروني