EDC & I CP494

E-Learning Assessment and Evaluation

Course Introduction

In the previous course you explored the technology and work processes needed to develop successful E-Learning courses or programs. In order to build or maintain support for your developed work, you'll also need to demonstrate its effectiveness. This class on assessment and evaluation, the third in the ELDD certificate program, focuses on the concepts, skills, and procedures you'll need to demonstrate success.

A basic assumption of the course is that information alone is not instruction. No matter how well done, offering information does not by itself guarantee that learners absorb and can apply the knowledge presented, nor that organizational goals for the instruction are reached. Those designing and offering instruction and training need to actively gauge the results of their efforts during and after information is offered. Assessment and evaluation of the results of E-Learning courses and programs is therefore essential to determining your success, yet is too often ignored, done poorly, or executed incompletely. Ideally, assessment and evaluation are an integral part of all stages of course and program development.

This three module course is organized around the "Four Levels," the classic model of evaluation first outlined by Donald Kirkpatrick in 1959. We begin by reading and discussing his ideas, while developing your knowledge of the vocabulary of assessment and evaluation. Then you spend some time considering important measurement concepts such as validity and reliability and how they relate to testing, performance evaluation, and the selection of assessment tools. You also visit various Websites to experiment with their assessment tools. You'll apply these concepts and experiences in a case study analysis of an actual organization's assessment and evaluation practices. Finally, you explore how to evaluate the effectiveness of the E-Learning project you are developing.

Course Objectives

When you successfully complete this course, you should be able to:

• distinguish between assessment and evaluation;
• elaborate Kirkpatrick's Four Levels of Evaluation;
• use the vocabulary of assessment;
• apply measurement concepts to the development of a test;/li>
• articulate the criteria or process for deciding what aspects of teaching and learning to assess;
• interpret teaching and learning assessment data to make meaningful decisions;
• describe the range of E-Learning assessment strategies, methods, and tools;
• choose strategies, methods, and tools appropriate for what is being assessed;
• describe specific strategies, methods, and tools used to evaluate a course or program;
• establish criteria for measuring accomplishment of objectives; and
• choose appropriate evaluation strategies, methods, and tools for your project.

Required Reading

Module One

Required

1. Foreword, Preface, Chapters 1-3 and Chapter 8 in Part One: "Concepts, Principles, Guidelines, and Techniques," in Kirkpatrick and Kirkpatrick. Evaluating Training Programs: The Four Levels. 3rd ed. San Francisco: Berrett-Koehler Publishers, Inc., 2006.

Recommended

1. Chapters 1-2, in Horton, William (2001). Evaluating E-Learning. Alexandria, VA: ASTD

Donald Kirkpatrick is widely considered to be the "father" of contemporary evaluation theory. This third edition of his classic text is authored in conjunction with his son, who is expanding on Level 3 of the original work. Part Two of this edition includes a series of case studies written exclusively for this book, which illustrate how professionals implemented Kirkpatrick's ideas. This Kirkpatrick text will become a founding volume for your professional library. You'll refer to it often.

You are familiar with William Horton from your reading in the previous course. Horton discusses assessment in several places in the text "E-Learning by Design," but elaborates in "Evaluating E-Learning," in which he provides numerous practical examples, planning documents, worksheets, and processes for realizing the theory in Kirkpatrick's Four Levels. The accompanying Web site provides electronic copies of the materials for your use.

Module Two

Required

1. Chapter 5, "Tests," and "Test learning for the topic," pp. 299-300 in Horton, William.E-Learning by Design.
2. Chapters 4-6 and Chapters 10-11 in Part One: "Concepts, Principles, Guidelines, and Techniques," in Kirkpatrick and Kirkpatrick. Evaluating Training Programs: The Four Levels.

Recommended

1. Chapters 3-5, in Horton, William (2001). Evaluating E-Learning. Alexandria, VA: ASTD

Module Three

Required

1. Chapters 7-8 in Part One: "Concepts, Principles, Guidelines, and Techniques," and a case study of your choice from Part Two, "Case Studies of Implementation," in Kirkpatrick and Kirkpatrick. Evaluating Training Programs: The Four Levels.

Recommended

1. Chapters 6-13, in Horton, William (2001). Evaluating E-Learning. Alexandria, VA: ASTD.
2. American Society of Training and Development. "E-Learning ROI & Evaluation." Learning Circuits: ASTD'S Source for E-Learning. No date.

Course Organization

There are three modules in this course. The features of each module vary, but can include a narrative, required and optional readings, quizzes, discussions, optional study questions, Web searches for a variety of resources and background information, and assignments. Following is a breakdown of what we cover in each of the modules.

Module One–Assessment of E-Learning Teaching and Learning

This two-week module introduces the Kirkpatrick model, and presents the principles, concepts, and specific challenges of assessment in E-Learning. We cover some important measurement concepts and you begin your thinking about how they might apply to the assessment plan for your project.

Module 2–Best Practices of E-Learning Assessment

This two-week module focuses on Levels 1-3 of the Kirkpatrick model. You'll explore the tools, strategies, and techniques used in E-Learning assessment at these three levels. During this time you interact with various online assessment tools and have the chance to download and investigate free assessment shareware. Through your work in this module you evaluate and choose appropriate assessment methods and tools for your project.

Module 3–Evaluation of E-Learning Programs

This two-week module focuses on Level 4 evaluation—assessing how effective the course or program was in meeting your organizational objectives. You examine various measures of program effectiveness, including enrollment, completion rate, student progress, performance on exit exams, Return on Investment (ROI), and responses to course evaluation surveys.

Technology Requirements and Skills

EDC&I CP494 uses the same electronic mail and Web-browsing software as previous courses in this certificate sequence. We'll also be using the discussion forum and the course delivery system Moodle.

Pretest and Postest

Before diving into the content of this course, please go to the pretest linked to Module One and take the short test to help your instructor understand what knowledge of the course materials you may already possess. You take this same test again at the end of the course as a way to measure your learning.

Assessment and Grading

There are several activities to complete in this course; some are graded, some are not. The Assessment page linked to this Course Introduction describes the relationship of each activity to the course.

Project

As we address various issues in the course you continually develop and revise the assessment and evaluation plans for your project. A final description of your evaluation plans submitted at the end of the course reflects what you learned. This summative assessment should mirror what you would actually submit to other professionals as a plan to demonstrate the effectiveness of your course or program.

Time Management

In online learning, it is easy to get off-track and spend copious amounts of time on assignments, particularly when they involve investigating specific Websites. We're thrilled you may want to spend extra time investigating these resources. However, you don't want to overwhelm yourself and find yourself short on time to complete required assignments. For example, many previous learners reported spending 15 to 20 hours completing the Web Exploration assignments. It is completely up to you how much time you spend on this course, but please consider these time management recommendations:

• Required Readings
  Take the time to read over the required readings carefully, but do not spend more than an hour on any given reading, such as one chapter of a text.
• Study Questions
  A good time-savings approach is to complete the study questions as you are reading. Not only will this save you time, but will also make your reading more active and purposeful. Completing the Study Questions also functions as preparation for taking the quizzes.
• Quizzes
  Spend no more than 30 seconds per quiz item. If it is taking you longer than that to answer a question, you may not have reviewed the required content sufficiently. In most cases, quizzes should take three to five minutes to complete.
• Web Explorations
  These are more time consuming assignments–ones you may wish to break into smaller tasks. The Web Questions are designed to help you focus on your intent at each Website. However, do not spend more than five hours on either of the Web Exploration activities–if you do, you are putting in much more time than expected to complete these activities.
• Discussions
  Spend no more than 30 minutes posting your assignments to a forum. Remember you also need to revisit the discussions to examine responses to your post, and provide feedback to your classmates' posts.

Most important to remember is that this is a six-week course. The weeks go by quickly, so keeping up with the work is especially important. The work is cumulative, with the most important assignments at the end of the course. If you fall behind early in the course, catching up takes significant effort.

---

©2006-2007, University of Washington. All rights reserved.
No part of this publication may be reproduced in any form or by any means without permission in writing from the publisher.