A basic belief of instructional design is that instruction is not considered complete or released until it has been demonstrated that trainees can indeed learn from the materials.
Authors of "The Standards, 1986

Formative evaluation is the essential "lifeblood" of the instructional development process.
Thomas Reeves, Evaluating Technology-Based Learning

• To understand the purpose of evaluation
• To distinguish between formative evaluation and summative evaluation
• To identify key characteristics of formative evaluation
• To identify key characteristics of summative evaluation
• To Identify key procedures and tools used in formative evaluation
• To distinguish between pre-test, pilot tests and field tests

As the above quotations indicate, evaluation is an integral and essential part of the instructional design process. Evaluation serves two main functions: 1) provide valuable feedback to improve the design process and product, and 2) to determine if the training (product) was effective.

As you read through the literature it will be apparent that authors identify different types and techniques of evaluation. However, in this course we are primarily concerned with formative evaluation and summative evaluation.

Michael Scriven first distinguished between formative and summative evaluation in the 1960’s (Scriven, 1967). Formative evaluation addresses issues during the design process. Data gathered from formative evaluations on such components as task analysis, instructional objectives or message design provide valuable information that can be used to revise and improve the instruction/training. As you learn you adjust your process. Typical techniques in formative evaluation include: one-to-one trials, small group trials, and field tests. "Formative evaluation results are unique to the particular project and thus will have limited generalizability to projects outside the same educational context or curriculum." (Kemp, Morrison & Ross)

Summative evaluation is conducted after the implementation of the instruction. Its purpose is to evaluate the effectiveness of the instruction or training based on the original instructional objectives or for a variety of other purposes. Stakeholders such as supervisors, and management are particularly interested in the summative evaluation. Issues such as effectiveness of training, cost of program development, on-going costs in relation to efficiency and effectiveness and long term benefits of the instructional training can inform stakeholders of the results of the program. The proven success of an instructional/training program will justify the current and future allocation of resources to the program and justify performance expectations. Scriven, (1991) points out that summative evaluation is "done for or by any observers or decision makers who need evaluative conclusions for any reason other than development."  Examples of a range of techniques used for summative evaluations include: pretest-posttest with one group, one-group descriptive, experimental-control group, analysis of costs, resources and implementation.

The literature identifies different types of evaluation and the different approaches/ techniques used to evaluate. In some cases the distinction is in name only. KMR identifies four approaches to formative evaluation based on the particular project: connoisseur-based, decision-based, objectives-based and public-relations-inspired. Rothwell and Kazanas also provide formative evaluation techniques: expert review, management or executive rehearsals, individualized pretest and pilot tests, and finally, group pretests and pilot tests. These two classifications have points of contact and similarity. For example, connoisseur-based evaluation and expert review are roughly the same approach. As you read you might ask yourself how the other techniques in each category relate to those in the other.

Reeves’ facets of evaluation include effectiveness, formative and impact evaluation. Impact evaluation, while similar to summative evaluation, differs from it in important ways. Impact evaluation is most often done some considerable time after the learning experience to assess the overall impact the learning experience has had on the work environment or the organization. It uses techniques which look at the original problem and the work environment or organization. On the other hand, summative evaluation most often checks learning against the objectives established for that learning as we stated earlier. In a sense, impact evaluation provides more of a direct tie to the original determination and description of the problems which should have been identified in the needs assessment.

Generally, the underlying concepts and techniques involved in formative and summative evaluation are similar across authors. For example, pre-tests and field tests are used in both types of evaluation. As an instructional designer you may see the value in employing some or all of these evaluation approaches you will read about during the design process depending on your project and the client for whom it is being done.

The type of evaluation you choose depends on what you intend to evaluate. Instructional design is a complex process with critical interdependencies. You may need to evaluate the management of the instructional design which requires organized documentation of operations; or you might require feedback at each stage of the design process for revision purposes. You might want to evaluate the difference in knowledge, skills and attitudes before and after the training. Finally, most designers want to evaluate the degree to which the instruction has met the original instructional goals. A key distinction between formative and summative evaluation is that the former is designed to improve the current product/process while the latter is intended for future improvements.

Evaluations, formative or otherwise must not be conducted on an ad hoc basis. Careful organization and management of the evaluation process is essential to its effectiveness. A well-planned evaluation will include purpose, target audience, objectives, method for gathering data and resources, analysis and report. Each of these areas includes additional issues that must be considered when planning the evaluation.

As a novice instructional designer it is useful to use models to guide your evaluation process. A familiarity with the advantages and disadvantages of the various formative evaluation approaches will help you choose a model that addresses your needs.

Finally, remember that the formative evaluation is an iterative process that enables you to leverage the information gathered from each stage of evaluation and use it to revise and optimize your design.

Readings and Activities:

In this module you are asked to read chapter 15 in Kemp, Morrison and Ross and the following two articles and participate in the threads in the online conference.

Rothwell, William, J., & Kazanas, H. C. (1992). Evaluating instruction, Chapter 13, Mastering the instructional design process : A systematic approach. San Francisco : Jossey-Bass.(Chapter 13)

Reeves, Thomas, C. (1993). Evaluating Technology-Based Learning. In Piskurich, G. M. (Ed.), The ASTD handbook of instructional technology (pp 15.1-15.32). New York: McGraw-Hill. (Chapter 15)

1. Under what circumstances might the acceptability and feasibility of data collecting be impeded during a formative evaluation?
2. Your company’s client has recently undergone significant upper management restructuring. The new vice president of marketing believes that the planned web-based training session on customer profiling is expensive psycho-babble that will waste valuable productive time. What could you do to keep this client?
3. You have completed the screen design pilot tests of a multimedia educational product. The SME (graphic designer) has some concerns about the layout which you feel are invalid. What do you do?

References:

Kemp, J. E., Morrison, G. R. & Ross, S. M.. (1998). Designing Effective Instruction, Second Edition. Toronto: Merrill-Prentice Hall.

Rothwell, William, J., & Kazanas, H. C. (1992). Mastering the instructional design process : a systematic approach. San Francisco : Jossey-Bass.

Reeves, Thomas, C. (1993). Evaluating Technology-Based Learning. In Piskurich, G. M. (Ed.), The ASTD handbook of instructional technology (pp 15.1-15.32). New York: McGraw-Hill.

Scriven, Michael. (1967).The methodology of evaluation. In Gredler, M. E. Program Evaluation. (p. 16) New Jersey: Prentice Hall, 1996.

Scriven, Micahel. (1991). Beyond formative and summative evaluation. In Gredler, M. E. Program Evaluation. (p. 16) New Jersey: Prentice Hall, 1996.

Suggested Readings:

Flagg, B. N. Formative Evaluation for Educational Technologies. Hillsdale, NJ: Lawrence Erlbaum, 1990

Mark, M.M., and Shotland, R. L.. "Alternative Models for the Use of Multiple Methods."  In M.M. Mark and R. L. Shotland (Eds). Multiple Methods in Program Evaluation; San Francisco, CA: Jossey-Bass, 1987

Evaluation
http://www.monash.edu.au/informatics/techme/evaluati.htm

Summative Evaluation
http://uwf.edu/coehelp/studentaccounts/cfolsom/lesson/index.htm