Reprinted with permission of copyright holder.

THE ROLE OF VISUAL TESTING WHEN LEARNING FROM INSTRUCTIONAL MULTIMEDIA.

Richard Poon, Educational Technology Training Coordinator, Northern Alberta Institute of Technology, Canada, richardp@nait.ab.ca

Dr. Michael Szabo, Professor, Dept. of Ed. Psychology, Faculty of Education, University of Alberta, Canada, mike.szabo@ualberta.ca

Dr. Mohamed Ally, Organization Development Services, Northern Alberta Institute of Technology, Canada, mohameda@nait.ab.ca

Abstract | Introduction | Visual Testing Theory | A Field Experiment

Results | Discussion | Summary | Suggestion for Further Research | References

Abstract

A field experiment was conducted to test the hypothesis that after completing an Interactive Multimedia (IMM) program, students who receive visual testing will obtain higher achievement scores on identification, terminology and comprehension tests compared with verbal testing. Sixty two post secondary Emergency Medical Services students were randomly assigned to one of two groups. Both groups completed the same IMM lesson and two sets of evaluations. One evaluation was given immediately after the IMM lesson; one group was evaluated by visual testing and the other group by verbal testing only. The second evaluation was given later and the testing modes were reversed for the groups. Data were analyzed using repeated measures analysis of variance and Scheffe´ pairwise comparisons. A significant difference favoring visual identification tests was found. Several plausible hypotheses to account for the findings and recommendations for application and further research are presented.

Introduction

Research into visual testing using paper and pencil techniques has shown that students who receive instruction and evaluation complemented with visual displays achieve higher levels of performance when compared to those who don't receive evaluation complemented with visuals [e.g., Dwyer 1985; Dwyer 1986; Dwyer and De Melo 1983; Dwyer and De Melo 1984; Joseph and Dwyer 1986; and Szabo et al., 1981]. Are these research findings transferable to learning and evaluation in an IMM, i.e., computer environment?

Some research into the role of testing within microcomputer based instruction (MCBI) [Mayton, 1991; Richards, 1987; Rieber, 1990; and Taylor et al., 1989] suggests that visual testing is just as effective or in some cases more effective in evaluating learning by MCBI.

Visual Testing Theory

Batting [1979] and Nitsch [1977] reiterated the concern that any difference between the retrieval environment and the original learning environment produces marked decrements in the learner's performance. Morris, Bransford, and Franks' [1977] transfer appropriate principle implies that information processing must be appropriate to the intended use of the information. Tulving and Thompson's [1973] encoding specificity principle assumes that recognition memory is better if the cues used in the original instructional/acquisition environment are used in the testing/retrieval environment. Research by Batting [1979] and Jacoby and Craik [1979] has shown support for the encoding specificity principle.

Paivio's [1986] dual coding theory recognizes both verbal and non-verbal processing and postulates verbal and visual cognitive subsystems for encoding and retrieval of information.

A Field Experiment

A field experiment was conducted to test the hypothesis that after completing an IMM program, students who receive visual testing will obtain higher achievement scores on identification, terminology and comprehension tests compared with verbal testing.

This experimental study investigated the effects of two levels of the independent variable, (verbal or verbal plus visual testing) on the three dependent variables, (scores on identification, terminology and comprehension tests).

Sixty two post secondary students in an Emergency Medical Services certification program participated in the study on a mandatory basis. Subjects were randomly assigned to one of two research groups. Both research groups completed the same IMM lesson and were then given to two sets of evaluations.

One evaluation was given immediately after completing the IMM lesson; one group was evaluated by visual testing and the other group was evaluated by verbal testing only. The second evaluation was given within one to seven days later and the testing modes were reversed for the groups.

Table 1: Treatment Conditions

The instructional content of the IMM lesson used in the study was the anatomy and physiology of the pregnant female, the stages of labor and delivery and the procedures to prepare for an imminent delivery. The content was selected because it allowed the evaluation of learning objectives that involved identification, terminology and comprehension. The instructional content was delivered by an IMM lesson that was designed and developed by the researchers and an IMM development team. The IMM program was beta tested and reviewed by students and instructors. Formative feedback provided by the reviewers assisted the modification of the IMM lesson to its next version.

The IMM lesson was presented to the students by multimedia-capable personal computers on an individual basis operating as stand alone. The IMM lesson was delivered from the hard drives of each personal computer.

The IMM lesson content was program-directed and the students' progression through the IMM lesson was self paced. The students completed the IMM lesson in an average time of one hour. The IMM lesson content was divided into small learning sections as per the learning objectives. The content was presented to the students using both text and fifty three color visual displays. The visual displays designed and used were to present information that was identical to the text information. Color was used in the visual displays to present the actual color scheme of an anatomical structure, photograph or illustration or to highlight or differentiate various components of a graphic illustration. Audio supplemented sounds or pronunciation. Dynamic visual displays were used to show sequential processes or demonstrate psychomotor skills. At the end of the content presentations, the students completed rehearsal activities with corrective feedback.

Subjects

The study's subjects were Emergency Prehospital Medical Care students enrolled in Health Science programs at a post secondary technological institute in Western Canada and were adult learners whose age range from 18 - 40 years. The students were primarily being trained to be Emergency Medical Responders (EMR). There were four classes (n = 62) of EMR students used as subjects. Of these students, about sixty percent (n= 37) were male and forty percent (n= 25) were female.

Data Collection Instruments

One achievement test was developed from the lesson's learning objectives with three subscores for identification, terminology and comprehension using criteria defined by Dwyer and De Melo [1983]. The visuals used in the visual tests were a subset of those used in the IMM instructional lesson and there were no new visuals introduced in the visual testing.

The purpose of the identification subscore was to evaluate the students' ability to identify parts or positions of an object [Dwyer and De Melo, 1983] and required the students to identify the numbered parts on a detailed diagram of the pregnant female and fetus.

The purpose of the terminology subscore was to measure the students' knowledge of specific facts, terms, and definitions. The objectives measured by this type of test were appropriate to all content areas that required an understanding of the basic elements required for learning more complex concepts, rules, and principles inherent to the discipline.

The purpose of the comprehension subscore was measure a type of understanding that occurs when an individual can use the information to explain some other phenomenon occurring simultaneously [Dwyer and De Melo, 1983], e.g., students were given a specific point in the process of labor and delivery and then asked to identify what other activities are occurring at the same time or in the next step.

Verbal and visual versions of the data collection instrument were developed. The verbal version contained only text in the questions and answer choices. The visual version had text information complemented with visuals within the question or the answer choices. The visuals used in the tests were a subset of those used to present the IMM lesson content. For both instruments, seventy percent of the exam items were taken from the Professional Association's registration examination for EMRs. The remaining thirty percent were developed by the researchers.

The validity of these instruments was established by a review by program instructors and pilot testing with a group of students. The reliability of this instrument was determined by Cronbach's Coefficient Alpha test.

Data Analysis

A one way repeated measures analysis of variance (ANOVRM) test was performed on combined, immediate and delayed test scores converted to a standardized format. Significant findings were followed by using the Scheffe´ test of paired comparisons. Test reliability calculations were based on the Cronbach's Coefficient Alpha test.

Results

A summary of the ANOVRM test for the identification test is presented in Table 2. The ANOVRM test indicated that there was no significant difference between standardized means of the verbal and verbal plus visual test groups. There was a significant effect (F (1,60) = 32.8, p²0.05) reflecting that the visual testing mode had a higher mean score than the verbal testing mode for the identification test.

Table 2: Analysis of Variance Repeated Measures- Scores for Identification Tests

*p².0.05

A post-hoc Scheffe´ test of significance was performed to determine the source of differences between cells for the identification tests. The Scheffe´ test showed Group 1 and Group 2 visual test scores were higher than Groups 2's verbal test scores (Table 3).

Table 3: Scheffe´ Test of Significance - Identification Test Scores

(*) Indicates significant difference which are shown in the lower triangle

ANOVRM test was performed on the standard scores of the immediate and delayed terminology tests. Results indicated no significant difference between groups, testing mode or between groups and testing .

The ANOVRM) test was repeated on the standard scores of the immediate and delayed terminology tests and results indicated no significant difference between groups, testing mode or between groups and testing.

Reliability analysis was done on the all the criterion test scores and the composite test scores using the Cronbach's Coefficient Alpha test followed by an application of the Spearman-Brown prophecy formula. The Spearman-Brown prophecy formula was used because the criterion tests in this study were short in length, (nine items for each criterion and twenty seven items for the composite test). The results of the reliability tests are found in Table 4 .

Table 4: Reliability Coefficients for Criterion Measures after Application of Spearman Brown Prophecy Formula*

	Identification1	Terminology1	Comprehension1	Composite2
Verbal test	.42	.61	.72	.81
Visual test	.76	.42	.41	.76

* Cronbach's Coefficient Alpha test

1 9 items

2 27 items

The reliability coefficients for the composite verbal (0.81) and visual (0.76) tests indicate that they as a whole are reliable. The reliability of the individual criterion tests is of some concern because three of the six tests obtained reliability coefficients lower than .60. They are, however consistent with the low reliability scores, (i.e., less than .60) reported in the research by both Dwyer [1985] and McNeal [1994].

Discussion

The results of this study lend partial support for part of the study's hypothesis, i.e., students who receive visual and verbal testing obtain higher achievement scores on identification tests when compared with those who received verbal testing.

In comparing this study's findings to the previous research on visual testing, the results are identical to those found by Szabo et al [1981]. In addition, there is also some congruence between the findings of this study and the balance of the past research. See Table 5 for a summary of the current and past research findings for visual testing. A simple box score analysis indicates a visual testing effect in nine of the 30 tests. These data do not permit estimates of affect sizes. Effects for identification appear more frequently that for other outcomes of testing.

[Table 5 not included in digital conversion]

Several plausible hypotheses to account for the lack of significant difference for the terminology and comprehension scores found in this study have been suggested. They include the type of rehearsal activities used in the IMM lesson, the students' lack of familiarity with visual testing, the relationship between text and visuals, the types of visuals used for visual testing, the high competency requirements of the IMM lesson content and the novelty of learning by IMM

Rehearsal

Rehearsal activities of the instructional lesson are a possible factor that may have influenced the study's results. The instructional strategy used in the IMM program involved content being presented in small segments by the most appropriate media format followed by some rehearsal activities, e.g., complete a review quiz or practice labeling a diagram. Corrective feedback was provided to the students in all the rehearsal activities.

Rehearsal activities can be classified as either covert or overt. Covert rehearsal generally requires minimal information processing activity on the part of the learners. Examples of covert rehearsal include: reading prose passages, reading summary statements, reading questions and answering them mentally before checking with a given answer, and silently following the completed solution of a problem. Overt rehearsal provides some physical (and therefore assumed linked to the cognitive) activity where the learners are required to interact with the content material, attend to the information and spend increased time interacting with and encoding the information. Writing answers to questions, note taking, and drawing diagrams may be considered to be examples of overt rehearsal [Dwyer, 1985]. The effectiveness of covert versus overt learning strategies has produced somewhat mixed results in the literature but there are some indications that overt is more effective in increasing learning [Richards, 1987]. Dwyer [1985] found that the use of overt visual rehearsal activities with visual testing produced a significant difference in student achievement. The overt visual rehearsal activity required the students to shade and label part of a diagram of the human heart.

There were two overt rehearsal activities (dragging objects on the screen) used for the identification learning objectives in the IMM lesson and two covert verbal rehearsal activities in the terminology learning objectives. The comprehension learning objectives had one covert rehearsal activity i.e., one short multiple choice answer quiz. An hypothesis for further examination is that the difference found only for the identification exam was the result of overt rehearsal being only used for the identification learning objectives.

Familiarity with Visual Testing Formats

The students' performance on the terminology and comprehension visual tests may have been also influenced by the fact that EMR students are most familiar with verbal than with verbal plus visual tests. In EMR and prehospital emergency medical care education the typical evaluation used is a verbal multiple choice exam. McNeal's [1994] subjects were adult learners who were part time learners with other life roles and may not have had learning strategies that could benefit from visualized testing, whereas subjects in Dwyer and De Melo [1983] and Dwyer's [1986] studies were full time university students whose learning strategies might have allowed them to benefit from visual testing.

Szabo and Schlender [1996] suggested that the lack of familiarity with visual testing may have been a contributing a variable in producing no significant difference in their study. In fact, subjects performance on the test comprised of animated items was poorer than on a test comprised of similar but static visuals.

Relationship Between Text and Visuals

Another possible factor affecting this study is the way the visuals were used in the terminology and comprehension visual tests. In the majority of the visual terminology questions and some of the comprehension questions, the visuals were either complementary or be redundant to the questions stem or answer choices. Dwyer and De Melo [1984], suggested this type of redundancy may have simply provided an alternative iconic base from which students can interact and compound complex content material and noted their reasoning appear to coincide with Glanzer and Clark's [1963a, 1963b and 1964] verbal-loop hypothesis. This hypothesis states that a stimulus (illustrations or objects) viewed by a learner is translated into a series of words, is held in memory and is recalled by the learner in making a covert or overt response. Holliday et al [1977] have disputed that displays of visuals in text may result in some readers paying more attention to one cue and less to others. Might the subjects in this study have mentally translated the visuals into less important textual information or merely treated them as icons thus reducing their effectiveness for conveying information? These effects would have been also enhanced by the students' unfamiliarity with visual testing.

Nature of the Visuals

Another possible factor affecting this study is the type of visuals used in the comprehension test. In the IMM lesson both static and dynamic visuals were used to present content, and in the final evaluations only static visuals were used. It is probable that the use of static visual displays is not the most appropriate form for evaluating comprehension learning whereas dynamic visual displays may have been used instead. In the initial design of the visual comprehension test, static visuals and no textual information were to be used in the question stems and/or answers. In many cases a static visual could not encapsulate all the information or concepts presented by the question stem and/or answer choices of the verbal comprehension test. This limitation of the static visuals resulted in the inclusion of a verbal description to enable the question stem or answer to provide sufficient information to the students to comprehend and develop a response. This factor may have reduced the effectiveness of the visual comprehension test. Dwyer and De Melo [1984, p. 93] stated that they experienced the same problem in their study:

The intentional design of the visual distracters in the visual test format (so that they were congruent with the verbal distracters of the verbal items) imposed a severe limitation; the investigators, had they not been restricted by an attempt to make congruent distracters, might have been able to design visual distracters that could have assessed more validly the students' level of information retention.

Dynamic visual displays might have presented the students sufficient information to comprehend and develop a response without the inclusion of the verbal description in the question stem and/or answer choices.

Mayton [1991]; Rieber [1991]; and Szabo and Poohkay [1996] found that the use of dynamic visual displays, i.e., animation and text, in computer based instruction (CAI) increases student learning when compared to students who learned from CAI limited to static visual displays and text. The instructional material in Mayton's [1991]; Rieber's [1991]; and Szabo and Poohkay's [1996] studies were concepts and processes. In these studies the dynamic visuals were only used in the instructional lesson and not in the evaluation of learning. Chien [1991] found that students who received computer graphic instruction and were tested by video significantly outperformed students who received computer graphic instruction and were tested by computer graphics. Chien [1991] evaluated students' attitudes and abilities to discriminate performance of fast-moving objects. The findings of these studies suggest that the use of dynamic visuals may be more appropriate for assessing comprehension learning objectives. The role of dynamic visuals in the evaluation of learning is a subject that warrant further investigation. Although eleven of the 22 studies of dynamic versus static visual displays in CAI showed higher achievement [Szabo & Schlender, 1996], few if any of these studies examined different levels of achievement.

High Level of Competency

The competency requirements of the IMM lesson is another possible factor for influencing the study's findings. Emergency medical training demands a high level of competency from its practitioners because the undesirable consequences of poor performance. The pass mark for the EMR course was set at 80% by the school and the Professional Association for EMRs. Students enrolled in EMR training are fully aware of the high expectation for competency placed on them and they will devote more effort into their learning to obtain competency. The mean scores for all the criterion tests in this study were very high. The high level of mastery required and scores obtained in this study may have resulted in a restricted range of scores which could have mask the actual differences for the terminology and comprehension scores.

Novelty of Learning by IMM

The use of IMM is a fairly new instructional method for the EMR students and also for teaching the subject content. It is possible that the EMR students found learning by IMM to be a very novel experience. This novelty could have increased their attention and interest in the learning activity and subsequently could have increased their level of learning. Gagné [1985] noted that gaining attention and motivation is key to a successful learning process. The effects of novelty are considered by the researchers to be small but in combination with some of the alternative hypotheses presented, their total effect could be responsible for the no significant difference findings for the comprehension and terminology tests. Unfortunately, no data were collected to assess the participants' attitudes of learning activity. This data could have been analyzed to determine if there is any correlation between student performance and attitudes.

Summary

Students who learn by a visually rich IMM program seem to learn skills of visual identification better when evaluated by verbal plus visual testing than verbal testing. The study also found that the use of verbal plus visual testing is as adequate as verbal testing for acquiring terminology and comprehension learning skills.

Suggestion for Further Research

The implications of this study have practical application and also pose further research questions. The results of this study recommend the use of visuals in the evaluation of student learning from an IMM program, especially in situations where the learner is required to obtain a high level of competence in identifying parts or positions of an object, e.g., human anatomy in medical education. In addition, further research in to this area is recommended.

First, research of this type should be replicated with a sample and instruction content that do not require a high level of competency. The sample should be polled to determine preferences and familiarity to verbal and visual testing and this data should be analyzed to determine if there is a correlation to their final performance. The data should also be collected mid way of a course delivered totally by IMM, (this will reduce a possible effect created by the novelty of learning by IMM). The knowledge gained from this research will test some of the hypotheses postulated for creating the findings of this study

Secondly, research should be conducted to determine the role of rehearsal within an IMM lesson. Rehearsal is key to increasing the learning of any subject matter. The ever increasing capabilities and power of personal computers to convey information via IMM can also foster more elaborate and meaningful forms of rehearsal. The knowledge gained by this research will help direct instructional designers and IMM developers to increase the value of IMM as a learning tool

Finally, research should be conducted to investigate the role of dynamic visuals in evaluating learning from an IMM instructional program. The results of this study point out the limitations of the use static visuals within the evaluation phase. Research has shown that the use of dynamic visuals in the instructional phase of MCBI maybe beneficial, [Mayton, 1991; Rieber, 1991; and Szabo and Poohkay, 1996] or harmful [Szabo and Schlender, 1996]. The subsequent question raised is whether dynamic visuals are beneficial in evaluating learning? This body of knowledge will help direct instructional designers and IMM developers to determine if they are truly evaluating what is learned in an IMM lesson

References

[Batting 1979] Batting, W. F. (1979) . The flexibility of human memory. In L. S. Cermak & F. I. M. Craik (Eds.), Levels of Processing in Human Memory, Hillsdale, NJ: Erlbaum.

[Chein 1991] Chein, C. (1991) . Effects of Visual Information Presentation Mode and Testing Mode on Skill Analysis Training in an Interactive Video Setting (Video), Unpublished doctoral dissertation, The Ohio State University.

[Dwyer 1985] Dwyer, C. A. (1985) . The effect of varied rehearsal strategies in facilitating achievement of different educational objectives as measured by verbal and visual and testing modes. Journal of Experimental Education, 54 (2), 74-84.

[Dwyer 1986] Dwyer, C. A. , (1986) . The effect of rehearsal methods and rehearsal time on student information acquisition and retrieval, Journal of Instructional Psychology, 13 (1), 25-31.

[Dwyer & DeMelo 1983] Dwyer, F. M. and De Melo, H. (1983) . The effect of visual testing in assessing the instructional potential of variables associated with visualized instruction, Journal of Instructional Psychology, 10 ( 3), 126-41.

[Dwyer & DeMelo 1984] Dwyer, F. M. and De Melo, H. (1984) . Effect of mode of instruction, testing order of testing, and cued recall on student achievement, Journal of Experimental Education, 52 (2), 86-94.

[Gagné 1985] Gagné, R. (1985) . The Conditions of Learning (4th edition). New York: Holt, Rinehart and Winston.

[Glanzer & Clark 1963a] Glanzer, M., and Clark, W. H. (1963a) . Accuracy of perceptual recall: An analysis of organization, Journal of Verbal Learning and Verbal Behavior, 1, 289-299.

[Glanzer & Clark 1963b] Glanzer, M., and Clark, W. H. (1963b) . The verbal loop hypothesis: Binary numbers, Journal of Verbal Learning and Verbal Behavior, 2, 201-309.

[Glanzer & Clark 1964] Glanzer, M., and Clark, W. H. (1964) . The verbal loop hypothesis: Conventional figures, American Journal of Psychology,. 77, 621-26.

[Holliday et al 1977]Holliday, et al. (1977) . Differential cognitive and affective responses to flow diagrams in science. Journal of Research in Science Teaching, 129-43.

[Jacoby & Craik 1979] Jacoby, L. L., and Craik, F. I. M. (1979) . The effects of elaboration of processing at encoding and retrieval: Trace Distinctiveness and recovery of initial context. In L. S. Cermak & F. I. M. Craik (Eds.), Levels of Processing in Human Memory. Hillsdale, NJ: Erlbaum.

[Joseph & Dwyer 1986] Joseph, J., and Dwyer, F. M. (1986) . The effect of prior knowledge, presentation mode, and visual realism on student achievement, Journal of Experimental Education, 52 (2), 110-121

[Mayton 1991] Mayton, G. B. (1991) . Learning dynamic processes from animated visuals in microcomputer-based instruction, Proceedings of Selected Research presentations presented at the Annual Convention of the Association for Educational Communications and Technology.

[McNeal 1994] McNeal, J. M. (1994) . Effect of Rehearsal Strategies and Testing Format on Student Achievement of Different Educational Objectives. Unpublished doctoral dissertation, The Pennsylvania State University.

[Morris, Bransford & Franks 1977] Morris, C. D., Bransford, J. D., & Franks, J. J. (1977) . Levels of processing versus transfer appropriate processing. Journal of Verbal Leaning and Verbal Behavior, 16, 519-533.

[Nitsch 1977] Nitsch, K. E. (1977) . Structured Decontextualized Forms of Knowledge. Unpublished doctoral dissertation, Vanderbilt University.

[Paivio 1986] Paivio, A. (1986) . Mental Representations. New York: Oxford University Press.

[Pimolbunyong 1988] Pimolbunyong, R. (1988) . The Effects of Varied Rehearsal Strategies in Facilitating Achievement of Different Educational Objectives as Measured by Verbal and Visual Testing Modes and when Prose Instruction is in the Thai Language, Unpublished doctoral dissertation, The East Texas State University.

[Richards 1987] Richards, D. R. (1987) . An Experimental Assessment of the Relative Effectiveness of Varied Types of Computer-generated Feedback Strategies in Facilitating Achievement of Different Educational Objectives as Measured by Verbal and Visual Tests. Unpublished doctoral dissertation, The Pennsylvania State University.

[Reiber 1990] Rieber, L. P. (1990) . Effects of animated visual on incidental learning and motivation, Proceedings of Selected Research Papers presented at the Annual Meeting of the Association for Educational Communications and Technology.

[Szabo, Dwyer & DeMelo 1981] Szabo, M. , Dwyer, F. M. and De Melo, H. (1981) . Visual testing - visual literacy's second dimension, Educational Communication and Technology, 29, (4), 177-187

[Szabo & Poohkay 1996] Szabo, M., and Poohkay, B. (1996) . An experimental study of animation, mathematics achievement and attitudes toward computer assisted instruction, Journal of Research on Computing in Education, 28, 56-69.

[Szabo & Schlendeer 1996] Szabo, M., and Schlender, R. (1996) . An experimental comparison of effects of dynamic and static visual displays in computer based instruction on declarative and procedural knowledge of selected objects-oriented authoring skill. In Carlson, P., & Makedon, F. (eds.), Educational Multimedia & Hypermedia, (pp. 649-654), Charlotesville, VA: Association for the Advancement of Computing in Education.

[Taylor et al 1989] Taylor, W. et al. (1989) . The effects of MCBI program embedded cognitive strategies for performance on verbal and visual testing, Proceedings of Selected Research Papers presented at the Annual Meeting of the Association for Educational Communications and Technology. Dallas Texas

[Tulving & Thompson 1973] Tulving, E., and Thompson, D. M. (1973) . Encoding specificity and retrieval processes in episodic memory. Psychological Review, 80, 352-373.

Abstract | Introduction | Visual Testing Theory | A Field Experiment

Results | Discussion | Summary | Suggestion for Further Research | References