Assessment is clearly required to determine if one has a technology that works
Baker and O'Neil (1987)

Goals:

• Understand the relationship between an evaluation instrument and the types of outcomes they assess (knowledge, skills, behavior and/or attitudes)
• Select an appropriate evaluation instrument to determine if an instructional objective has been met
• Recognize the different methods of evaluation and their inherent strengths and weakness
• Understand the difference between Criterion-referenced testing, normative-referenced testing and instructor-based testing
• Understand the relationship between instructional objectives and criterion-referenced testing

In this module we discuss how to assess what students have learned as a result of working with the learning experiences we create. It may be useful as we begin to address the question of the terms evaluation and assessment. It is often the case that these terms are used interchangeably, sometimes when referring to gauging what students have learned; sometimes in measuring program effectiveness; sometimes for recommending changes to be made to an instructional sequence; and sometimes to tell if a particular course or program should continue. In the readings you will encounter both words and, additionally, the terms formative and summative evaluation. These latter two have specific meanings and techniques which we will address in Module 14. For now, to avoid confusion, we will use the term assessment when we are discussing the measurement of student learning with regard to meeting the goals and objectives established for a learning experience.

The need (or requirement) to assess student learning is always with us. Assessment may be required to determine if a student can move on to a new unit, to a new course or grade, or to obtain a certificate of proficiency (sometimes called a diploma or degree). It may be required as a part of an assessment/evaluation process by those responsible for showing that funds devoted to learning experiences have been well used; required by those wishing to show that students who are being prepared for something are well prepared; and sometimes required for other organizational, legal, or political purposes, as well.

From the position of the development of an assessment approach and items in the MRK model, and in other ISD models as well, we can see that assessment is (and, arguably should be) closely tied to the development of goals and objectives. The idea here being that we need to know how we will appropriately assess learning related to the proposed goals and objectives, and that it easier to do while we are thinking about those goals and objectives and their relationship to the problems and tasks identified earlier.

There are several types of assessment, but criterion-referenced testing, normative-referenced testing and instructor based testing are the most common examples. Choosing which type of test to administer may be determined by the kinds of information the designer wants the tests to reveal. For example, does he want to compare a student’s performance with those of others in a similar group or a similar domain? Or does he require an absolute indicator of what the student knows based on a domain of knowledge?

There are two main types of testing: criterion-referenced testing and normative-referenced testing. Normative-referenced testing (NRT) compares the test scores of individuals with those of others in a similar group. NRT tells us how well a student has performed relative to another in the same domain. In order to make the assessment more meaningful we can compare that student’s performance to some external representative group called a norm group. The data in a norm group indicates the typical performance of students in the same domain and the methods for norming are established using standardized procedures and statistical methods which we will not address in this course. An example of NRT are SAT’s, provincial departmental exams, and international comparisons of performance in various subjects.

Criterion-referenced testing (CRT) is designed to determine student competency within a specified domain which may be defined as a concept, a lesson, a unit, or a course. CRT provides for the creation of test criteria for the objectives proposed for a particular domain. The individual has demonstrated sufficient competency within the specified domain when their test score meets or exceeds the mastery level. The mastery level is the minimum score required to show competency in that domain to meet the criteria for all objectives. The course designer, or sometimes the teacher or course administrator, decides the criterion for mastery learning. Normally it is a percentage of correct responses.

Thus, CRT can provide the measurement scale used to designate the mastery level and the measuring tool used to determine whether the learner has mastered the objectives.

Teacher-made or instructor constructed testing are created by teachers for use in their classroom. The instructor chooses a representative sample of objectives to assess the knowledge and performance among students within a given class. The success of these types of tests rely on the skill and knowledge of the instructor and the extent to which they adhere to testing principles.

The argument that MRK present is that designing effective instruction is best realized using a systematic approach. Some instructional design models, such as the MRK model, offer a flexible approach to the development of instruction, by which we mean that the designer can undertake development of different sections in a differing order depending on the project and the circumstances, and that they can move back to a previous section if they need to. This is particularly useful when designing components that are closely related. Such is the case with designing instructional strategies and developing evaluation instruments. Chapters 10 and 11 in MRK explain the direct relationship between the type of evaluation and instructional objectives.

There are many types of assessment instruments: true/false, multiple choice, essay, performance, and others. Each instrument has its strengths and weaknesses that must be carefully considered when the teacher/design is selecting an evaluation instrument. The readings for this module provide useful examples and caveats. Not every learning circumstance is amenable to a mastery learning approach using criterion-referenced testing, for example. The higher up a taxonomy of learning one goes, particularly in the cognitive and affective domains, the less useful a CRT approach is. As we have discussed previously, complex analysis, synthesis tasks, complex problem solving and creative tasks do not lend themselves to CRT.

In this module you are asked to read the following two chapters in MRK and the additional chapter from Dick and Carey and participate in the threads of the online conference.

Dick, W., and Carey, L. (1990). Developing criterion-referenced test items. In Dick, W., and Carey, L, The systematic design of instruction, 3^rd Edition. Glenview, Il: Harper Collins.

Glossary of Assessment Terms

Activities:

Activity one:

Explain the value of criterion-referenced testing to instructional design.

Activity two:

Recall the two objectives that you created for Module 8. Describe what kind of assessment you would choose to determine appropriate student learning in each one.

Activity three:

You are teaching a Distance Education Art 10 class of 30 students using videoconferencing and the World Wide Web, with additional materials being sent to students for labs and practical work through Canada Post. It is report card time. After 8 weeks you have covered the following material: principles of design, color theory and soapstone carving.

Your school district has provided the following three goals:

1. To describe the personal experience of the creative process
2. To evaluate the success of a product
3. To demonstrate correct use of principles of design

Choose an approach to assessing your students' learning towards achieving these goals. Explain your choice based on this week’s readings. Include any potential disadvantages in your choice.

Post your response to the appropriate WebBoard conference.

Baker, E. L., & O'Neil, H. F. (1987). Assessing instructional outcomes. In R. M. Gagné, (Ed.), Instructional Technology Foundations, Hillsdale, NJ: Lawrence Erlbaum Associates.

Dick, W., and Carey, L. (1990). Developing criterion-referenced test items. In Dick, W., and Carey, L, The systematic design of instruction, 3^rd Edition. Glenview, Il: Harper Collins.

Hopkins, Kenneth D. (1998). Educational and Psychological Measurement and Evaluation, Eighth Edition. Toronto: Allyn & Bacon.

Kemp, J. E., Morrison, G. R. & Ross, S. M.. (1998). Designing Effective Instruction, Second Edition.Toronto: Merrill-Prentice Hall.

Sax, G. (1997). Principles of Educational and Psychological Measurement and Evaluation, Fourth Edition. Toronto: Wadsworth Publishing Company.

Hopkins, Kenneth D. (1998). Educational and Psychological Measurement and Evaluation, Eighth Edition. Toronto: Allyn & Bacon

Sax, G. (1997). Principles of Educational and Psychological Measurement and Evaluation, Fourth Edition. Toronto: Wadsworth Publishing Company.