CMI Research at the University of Alberta in Context

M. Szabo
T. C. Montgomerie

Pages 5-6

Abstract

The University of Alberta has a long history of the use of computers as instructional devices. It began in 1968 with the introduction of an IBM 1500 computer assisted instruction system and progressed though the installation of a CDC PLATO system which was subsequently removed in 1987. Management of instruction has always been a key element of any computer related research done at the University of Alberta whether as a part of large computer assisted instruction courses, or as courses specifically designed to utilize computer managed instruction. This paper presents a context for the study of computer managed instruction and reviews a number of the key studies in computer managed instruction at the University of Alberta. With the reorientation toward microcomputer based instructional systems, it seems propitious to summarize the research in CMI to this point in time. (Keywords: computer managed instruction, attitudes, achievement, costs)

Introduction

Can researchers and teachers devise teaching-learning conditions that will enable the majority of students under group instruction to attain levels of achievement that can at present be reached only under good tutoring conditions? (Bloom, 1984, p. 5)

A CBI system that replaces part or all of a teacher's instructional role will receive little if any long-term use. (Baker, 1981, p. 23)

The University of Alberta (U of A) has a long history of involvement in computer managed instruction (CMI). In 1968, the U of A received one of the first IBM 1500 computer systems. The IBM 1500 was designed as a complete multimedia computer assisted instruction (CAI) system, with a high resolution (by 1968 standards) graphics display screen, light pen, random access film strip projector and audio record/playback units at each student station. It also utilized a specially designed CAI authoring language, Coursewriter II. While many other designers of CAI in the 1960s and 1970s concentrated on building stand-alone modules to supplement teaching, the orientation of authors at the University of Alberta was towards the development of complete courses where the computer performed all the mainline instruction, testing and record keeping. The philosophy of these courses was that the instructor should be involved only when the computer diagnosed that an individual student required remedial instruction or skill-based evaluation that the computer could not perform. This freed instructors to act as tutors, concentrating their efforts where they could do the most good.

As courses were developed during the early 1970s, authors began to realize that, while Coursewriter II provided good instructional capabilities, there was no predefined ability to manage the testing and routing of students through the course. Each author needed to build such CMI capabilities into the CAI code.

One of the first people to investigate the possibility of separating the models of instruction, including testing and prescription, from the actual content of the instruction was Romaniuk (1970). His early work on the Versatile Authoring Language for Teachers (VAULT) system was later identified by Fred Morrison as one of the inspirations of the TICCIT System.

Between 1972 and 1976, an extensive CMI model and package was developed and applied to the theory component of six university nursing courses at Penn State University (Szabo & Estes, 1978). One of several unique features of this multi-year project was that it was delivered across a large geographical area using specially designed mobile vans which housed the IBM 1500 systems and student learning stations.

Some of the earliest work in developing a CMI capability for the IBM 1500 was undertaken at the U.S. Army Signal Corps, Fort Monmouth, NJ (IBM, 1968). The source code was made available to the IBM 1500 community and was subsequently modified by staff at the Division of Educational Research Services (DERS) at the U of A to include a number of different CMI capabilities. Petruk (1975, 1978) used these modified CMI features as the basis for two sequential projects under the joint sponsorship of Alberta Advanced Education and Manpower and Canada Manpower and Immigration to use CAI to teach electrical apprentices at the Northern Alberta Institute of Technology. The instructional model used by Petruk (1978, p. 5) represented an advanced use of CMI within a CAI environment.

In CAI, this traditional model is replaced by a competency-based model, which recognizes that time, not achievement must be the variable in any training program. A basic tenet of this competency-based model is that all personnel trained must attain a specified level of competence in relation to every objective specified in the training program, and that some will achieve this level of competence in a shorter time, by a more direct route than others. (Petruk, 1978, pp. 3-4)

At the same time, the National Research Council of Canada contracted with staff at DERS to develop a set of preliminary specifications for an instructional support system for NATAL-74 (the NATional Authoring Language). A great deal of the document (particularly chapter 5) is oriented towards the provision of CMI capabilities within NATAL-74 (Hunka et al., 1978).

By the time the IBM 1500 was returned to IBM in 1980, a number of courses had been designed and implemented at the University of Alberta in which the primary instruction was by computer. Hunka (1978) describes five of these courses in which the average student time to complete the course varied from 15 to 75 terminal hours. While all of these courses were considered to be CAI, they each contained a major CMI component.

In 1980 the U of A installed a CDC PLATO system. Among the many powerful features of this system is an integration of CMI with CAI. Because of the powerful software design and mainframe capacity, a prescription to an existing CAI lesson could be carried out immediately, subject to the control of instructor and student. Courses in textiles, anatomy, physical education, nursing education, and a variety of other CMI applications were developed using this system, and many were transported to other PLATO systems.

In 1987, the PLATO system was decommissioned at the University, presumably due to budget constraints. University president Paul Davenport stated in the Edmonton Journal on 29 April 1990,

In the 1970s and the early 1980s the U of A was a leader in the use of computer technology for teaching and research. We were pioneers in computer-assisted instruction, the use of microcomputers, in computer-assisted registration and electronic mail. With the change in budgets in the later 1980s, however, the university saw that leadership, despite our best efforts, gradually slip away. (p. A7)

Learning Theory Basis for Individualized Instruction

Learning is significantly more effective when instruction can be tailored to the unique needs of each learner. For example, Bloom (1984) found that students who are individually tutored learn twice as much (score 2 standard deviations above) as students instructed in the conventional manner.

Learning is also significantly more efficient when instruction can be tailored to the unique needs of each learner. Studies of student-paced instruction (vs. instructor-paced instruction) show that students learn the same amount in 20 to 50% less time than when they are instructed in the conventional manner (e.g., Kulik, Kulik, & Cohen, 1980). Self-pacing is only one form of adaptation to the many individual differences exhibited in a typical classroom.

Research on learning severely challenges our conventional wisdom about instruction and learning, namely, that aptitude is fixed and places a limit on what our students can learn. This research has given rise to the testable hypothesis that given enough time and optimal learning conditions, most students "do have the potential to reach this high level of learning" (Bloom, 1984, p. 4). Carroll (1963) argued that aptitude could be redefined. "The amount of time that a student needs to learn a given task under optimal learning conditions, is, in the author's opinion, a reflection of some basic characteristic or characteristics of the student that may be called 'aptitude'" (p. 72). The deeper implications of the time dimension of learner aptitude has profound implications for the way we instruct.

Tennyson and Park (1984) made a significant contribution to the field of instructional theory and practice. The purpose of their research was to review four models of increasing sophistication which use CMI to link instructional procedures to learning outcomes. The underlying them of their work is that appropriate models which provide more real-time monitoring and guidance (advisement) of individual student progress (1) have shown to be effective and (2) can be administered by computer through CMI. It also represents a strong and more direct linkage between instruction and assessment.

What specific conditions of the learning process promote better learning? Bloom (1984) identified the six (after tutoring) most important alterable variables in terms of their effect on student achievement. These are reinforcement, corrective feedback, cues and explanations, classroom participation, study time on task, and improved reading/study skills. Let's now turn to the role of CMI in addressing these individual, alterable instructional variables.

The Role of CMI in Promoting Individualized Instruction

Computer managed instruction is a highly specialized application of the computer to assist in the process of managing an individualized instructional plan for students. It consists of two required functions. First, the computer tests the student to identify his/her strengths and weaknesses. The result is a diagnosis which indicates which objectives the student has and has not mastered. A unique dimension added by CMI is that the diagnosis is individually tailored for each student's pattern of performance. Based on the diagnosis, each student receives a computer-generated prescription, that is an individual course of study designed to help the student meet the specific unmastered objectives. The second function of CMI is, therefore, prescriptive. CMI is not simply computerized testing or item-banking as these do not provide individually-tailored diagnostic and prescriptive functions.

Competency-based learning/mastery imply a repeated cycle of instruction and testing until competencies have been met. The iterative nature of the computer offers tremendous potential to accommodate this cycle through its ability to administer exams with little or no instructor intervention and to clerically process the data and present the performance results in a summary form to the instructor.

Hansen (1970) listed five major functions of CMI: (a) providing diagnostic evaluations with learning prescriptions, (b) counselling students about adaptive learning strategies and career development, (c) developing an optimal scheduling system to match students and learning resources, (d) maintaining an appropriate student instructional record system, and (e) limited use of CAI for drill and practice.

While the technology used to deliver CMI has changed dramatically since the 1960s, the structure and function remain the same. Two outstanding sources for information on the structure and functions of CMI are found in Baker (1981) and Allen (1972). Figure 1 represents the core structure of CMI. Note that CMI can be and is often used in conjunction with existing learning materials. Thus, an instructor can incorporate CMI without substantially restructuring his/her ongoing instruction. In addition, CMI makes mastery learning possible in that retests to determine when mastery has been obtained are much easier to manage (administer, score, document, and report) than are conventional classroom tests.

People learn best when they can be made to actively and consciously process the information they encounter when they encounter it. Conventional instruction discourages the individual from processing information other than by trying to copy it to short-term memory, which has been shown to be most ineffective (some would say counterproductive) to learning. The computer is unique among media in that it can interact with a student in a processing way (Martin & Szabo, 1990) and stimulate deep cognitive processing (Anderson, et al, 1975).

Most of the specific conditions (alterable variables) identified by Bloom (1984) can be enhanced with the use of CMI. Individualized reinforcement can be provided for each student, item, and objective through CMI. Similarly, CMI can provide corrective feedback precisely where and when it is needed, to the individual student. Cues and explanations can be built into the CMI application and presented only to those whose performance indicates additional help is needed. Student classroom participation is interpreted to mean the active cognitive processing of information by each individual student, irrespective of level of participation. In a CMI exercise, the student must be actively processing information at a significantly more intense rate than during conventional instruction. During conventional instruction, there are many distractions which often cause the student to miss what is going on. During CMI, the student's attention is directed to the task at hand (assuming the ability level of the student is reasonably matched with the difficulty level of the CMI items). There is less 'down time.' We now turn to see whether research supports these statements.

--End of Reading for Pages 5 and 6.

Studies

This section of the paper briefly describes a number of studies of the use of CMI. Studies undertaken at the University of Alberta or which U of A equipment or personnel were involved are emphasized. A few landmark studies undertaken in other places are included to place the U of A studies in context. For each study, we have briefly described the study,

then discussed the findings and outcomes. A later section will amalgamate the findings of the studies.

Franklin and Marasco (1977) examined interactive computer based testing in post-secondary science courses. They identified (and exploded) five myths: (a) forcing students to use a computer presents an unwarranted additional obstacle their attempt to attain their educational goals, (b) computer based testing is crude and inflexible, (c) the use of computers is part of the mechanization and dehumanization of the educational process, (d) interactive computer based testing (ICBT) is too expensive, and (e) ICBT has serious problems with security and cheating. They also identified several advantages: ICBT (a) provides a broad variety of sophisticated tactics, (b) is amenable to virtually any educational strategy, (c) permits a more thorough integration of testing and teaching, (d) provides immediate and individualized feedback, (e) requires and promotes explicit pedagogical decisions, (f) allows excellent pedagogues to widen their audience and increase their impact, (g) has evolutionary potential, (h) has extensive record-keeping and clerical capabilities, and (i) is an economical approach to serious educational problems.

Several large-scale projects on individualizing instruction in the 1960s and 1970s found the computer to be valuable in managing instruction: Program for Learning in Accordance with Needs (Flanagan, 1969); Individually Prescribed Instruction (Cooley & Glaser, 1969); Instructional Management System (Silberman, 1968); and Computer Assisted Study Management System or CAISMS (Anderson et al., 1975).

Anderson and colleagues (1975) developed a computer assisted study management system (CAISMS) for use in a University of Illinois economics course. A well-controlled experimental research study was conducted among students in a large undergraduate course to compare students who used CAISMS with those who didn't. Students in the treatment group read study assignments and immediately completed on-line tests to assess their learning. Successful learning allowed a student to proceed through the syllabus, while unsuccessful learning required continued study of material already covered with retest opportunities. The treatment was developed in accordance with deep cognitive processing theory.

Results from this study indicated that students using CAISMS scored significantly higher on all course exams. One student's anecdotal comment is worth noting: "I really think CAISMS helped me a lot in this course, if not for anything else, it made me study regularly and be sure I got the right things out of what I read" (p. 8). Attitudes toward the course and method, as measured in the Anderson et al. study were significantly higher for the CAISMS students. Attitude scores for students was improved through the use of CAISMS for large but not for small classes. CAISMS had no overall effect on course completion rate.

Tennyson (1981) addressed problems of learner control associated with CMI. He was concerned with research indicating that students may be incapable of making and carrying out "decisions of content element selection and personal selection" (p. 426). Specifically, students whose instruction is self-paced may terminate their study prematurely (before they have mastered the objectives) and may not assume proper responsibility for their own learning. The Minnesota Adaptive Instructional System (MAIS) was devised to indicate to students the discrepancy between actual and mastery scores, provide study assignments based on discrepancies, and continually update the student. The MAIS system was replicated in several different content areas using 12th grade students. The criteria consisted of several examination scores covering the content of the lessons.

The results of the Tennyson series of studies are notable for their direction and their explanation. Two groups received identical CMI instruction. One group received advice from the CMI system via MAIS, while the other did not. It was found that achievement scores were significantly higher for the groups which received advice to help them to decide when to terminate instruction and which content to study next when compared with those who made the decisions in the absence of any external advice. The comparison was made between students who could terminate their study at any time and those who received performance feedback to assist them to decide whether they were ready to discontinue instruction. It was concluded that the guidance function provided by the management system enabled high school students to make better judgments about their own learning and thus improved scores. The implication drawn was "students can successfully participate in the management of their learning when provided with their own individual diagnostic and prescriptive information" (p. 430).

In the Tennyson (1981) study, students receiving advice spent 30% more time on task than students without such advice. It must be remembered here that the comparison was not with conventional instruction time but with students in self-paced instruction where they could terminate anytime.

The Nursing Studies

Szabo and Estes (1978) reported on the comparative costs of developing computer based instruction materials. Data were taken from logs of a project in which six university bachelor degree nursing courses were converted to a CMI format on the IBM 1500 System for distance delivery using a mobile van.

A comparison was made between developing computer managed instruction materials and developing computer assisted instruction materials. The CMI development was based upon the use of existing instructional resources; none were developed. On the other hand, the CAI materials were developed from scratch. It was found that an hour of CMI student contact time was developed in an average of 39.7 developmental hours. In the same project, an hour of CAI was developed in an average of 161.5 developmental hours. These data suggest a developmental ratio of 4 to 1, favoring CMI over CAI.

Kot, Skillen and Wales (1986) conducted an evaluation of a PLATO-based CMI course in nursing at the University of Alberta. Forty-six students at the post-RN baccalaureate level were randomly assigned to either a conventional lecture/discussion treatment or to a CMI treatment for the theory portion of the course. After the course had been delivered on campus for several terms, it was delivered in a distant town using microcomputers, modems, and public telephone networks.

Achievement scores, as measured by midterm, final, and practical exams, was a dependent variable in the Kot, Skillen and Wales (1986) study. Achievement scores and learning time for the two groups were equivalent, indicating equal amounts of learning. The Kot, Skillen and Wales (1986) study examined instructor time and concluded faculty "spent twice as much time per student on instructional activities for the control group" (p. iii). This time savings worked out to about 35 hours per instructor per course for this particular study. When students kept student time logs, it was concluded that they spent equivalent amounts of study time in the CMI and the conventional instruction classes.

Day and Payne (1987) compared the use of lectures with CMI to deliver basic nursing content in the first year of a baccalaureate in nursing program. This study used the same CMI system and course materials as were used in the Kot, Skillen and Wales study. The major difference was that the Day and Payne study used first year "generic B.Sc." students as subjects while the Kot, Skillen and Wales study used post-RN students as subjects.

The Day and Payne (1987) study confirmed the Kot, Skillen and Wales (1986) results in that no significant difference in achievement was observed between the two groups. The study found that students learned equal amounts based on course examinations and clinical sessions.

In the Day and Payne study, there was an indication of less positive attitude among the CMI group, although no statistical analysis was presented. If indeed lower attitude scores were present, several alternative plausible hypotheses would need to be confirmed, for instance (a) the application was well designed (some right answers were counted wrong, no feedback given for wrong answers); (b) it was not used with the appropriate target audience (the course was designed for experienced nurses returning for academic upgrading-the sample was first year baccalaureate nurses); and (c) the majority of students had recently completed a high quality computerized testing program in anatomy and may have made unobserved comparisons between it and the nursing application.

Boblin and Gibson (1986) developed a CMI component for an anatomy and physiology course for diploma nursing students at the University of Alberta Hospitals School of Nursing. This component was developed using the PLATO Learning Management (PLM) program. The CMI component consisted of a module component and an exam component. The module component consisted of 21 modules which corresponded to the the subject areas taught. These modules were optional and were "designed to assist students in acquiring the course content and preparing for the exams" (p. 127). The exam unit consisted of six compulsory unit tests. Due to cost, a "picture book" was developed for student use rather than developing on-line graphics.

Boblin and Gibson (1986) found that students in the class using CMI achieved higher marks "for all of the unit tests, for the average of the unit tests and for the final course mark" when compared to the previous three (non-CMI) classes. An interesting problem which resulted from the use of CMI was identified:

Historically, the A&P course has served as the "first screen" for students within the program. The subsequent levels within the nursing program are based upon student numbers and thus, are impacted by an alteration in the rate of attrition. The "expected" rate of attrition is approximately 7%, or 6 to 7 students. If the situation … continues (that is, reduced number of students with difficulties), then the School will need to consider whether it wishes to have the A&P course continue as the "first screen" in terms of promotional criteria, or whether it will have to alter subsequent levels of the program to reflect this increased number of students. (Boblin and Gibson, 1986, p. 132)

Boblin and Gibson reported that students were more comfortable with computers after the course; instructors were not, and instructors reported fewer testing/retesting clerical duties and consequently more time for instruction (time estimates not identified).

Boblin and Gibson (1986) state that "Records on the PLATO system show that students spent an average of 2.5 hours/week on the system" (p. 95). Further, while specific time was scheduled into student timetables for access to PLATO, only 53% of students actually used PLATO most often during scheduled time. 95% of students indicated that the "...liked and took advantage of being allowed access to the computers at times other than those scheduled" (pp. 94-95). During the time of their study, PLATO was available 22 hours a day, 7 days a week, with up time in excess of 99.7%. Boblin and Gibson (1986) made an extensive effort to keep track of time and costs. Tables 1 and 2 report the manpower requirements and the total cost of their development of 21 modules and 6 exams. Discounting costs for hardware which was used for developmental purposes only, the cost to design the program and run it on a mainframe system for one year was under $30,000.

The Alberta Vocational Centre Studies

The Alberta Vocational Centre/Edmonton (now Alberta Vocational College) has had a continuing its involvement with CBI. This originated because of the initial involvement of AVC/Edmonton in the Canadian validation of the Basic Skills curriculum delivered on the PLATO system. Fahy (1985) reports on the development of four CMI projects where PLATO PLM was piloted in existing adult basic education programs. Each of the four projects in the study was specifically designed to facilitate student self-direction and self-pacing in order to see whether current theory in adult education could be put into practice in an existing adult education program. In terms of CMI development time, other data are helpful.

Fahy (personal correspondence) kept a log of developing a CMI application using the PLATO PLM system and existing instructional materials. His data suggests that an existing course can be developed in CMI format in 5 minutes per test item. Extrapolated, a

Table 1

Manpower Requirements Per Phase

Table 2

Total Costs of the Pilot Project

course of 1200 test items requires a development time of 100 hours. The U of A PLATO Instructional Systems Group commonly used the figure of 20 minutes per test item when bidding commercial CMI development projects for PLM. While this is four times the estimate of Fahy, it must be remembered that Fahy was using existing instructional material in an internal development, while a commercial developer must be extremely conservative when estimating time and resources.

With the success of the use of CMI with the adult basic education program, the AVC/Edmonton investigated the feasibility of becoming more involved in CBI. An external contract was let to undertake a study to "investigate and report on the feasibility of implementing computer assistance in operating and managing student record keeping and testing activities as a means of attaining program objectives. The study is to consider both administrative and educational management practices and needs" (Montgomerie, 1985, pp.85-87). The resulting study made six recommendations:

AVC/Edmonton should incorporate CBI in the Nursing Assistant Program.

Implementation should be phased in (CMI followed by CAI).

A team (members with content experts, instructional design, CBI experts and management). Included was the recommendation of 1/2 time CBI coordinator and the establishment of a centralized instructional design group.

Computing facilities should be contracted from some other institution.

Plato CBI system should be used.

No integration with the student information system, rather transfer information at specific reporting points. (pp. 85-87)

Alberta Vocational Centre/Edmonton did undertake a project to incorporate CBI in the Nursing Assistant Program. Fahy (1987, p. 14) reported on this implementation:

All but one recommendation of the Montgomerie report (1985) were met fully and one recommendation met partially in this PLATO-based CML implementation project. The unimplemented recommendation, reiterated in this study, was for an institutional instructional design group (with project management responsibilities).

Fahy (1987) gave a pre- and post-questionnaire to students . He reports that

experience strengthened students' opinions that: PLATO was an efficient use of their time, was enjoyable, was interesting and was satisfying; PLATO was not slower than paper-and-pencil testing, did not make them nervous, was not too impersonal, did not make them feel isolated, and was not frustrating. (1987, p. 13)

Fahy goes on to report that, in interviews, students' major complaints about the PLATO system involved accessibility (number of terminals, hours of operation, etc.), privacy (dividers between stations, and "staff should not watch students over-the-shoulder while they are testing"), and lack of noise insulation (p. 14).

Fahy (1985, p. 16) found that "those [instructors] who actually experienced individualization in the projects found that computer management is capable of addressing many instructor reservations about, and of providing a relatively painless introduction to, individualization." Students were very positive about the use of CMI, reporting that they felt self-pacing lessened the disadvantage they faced, since they were not in competition with other, more fortunate students, and were working at a pace and on a schedule which recognized their own limitations. A second discovery was that students found learning from one another, or from the print modules, frequently more helpful and enjoyable than being tutored solely by the instructor. (p. 12)

Instructors reported fewer testing/retesting clerical duties and consequently more time for instruction (Fahy, 1987).

Fahy (1987, p. 21) compared the average cost per student using PLATO CMI with more traditional testing. The costs (in 1987 Canadian $) are summarized in the Table 3.

The Police Studies

Szabo (1987) evaluated a project in which a major city police force converted its training section to CBI. The system made heavy use of PLATO-PLM; it also used CAI modules and short videotaped demonstrations. Thirty-two modules of instruction were converted from conventional instruction to CBI delivery over a 3 year period and the CBI modules were phased in as they were completed and validated. The evaluation study involved approximately 1100 trainees using the materials over a period of one year. As an experimental study was not possible, conclusions were drawn based on comparative data gathered during the six month period prior to the introduction of specific modules.

Table 3

Costs of PLATO-Based CMI and Other Delivery Modes

The Szabo (1987) study examined indicators of both paper and pencil exam achievement as well as selected on-the-job performance measures. In those measures available, trainees scored significantly higher if they were trained using CBI as compared with conventional (previous) training. Included in the data were such measures as scores on promotional exams, use of canine unit, number of arrests made using the canine unit, and fewer costly clerical errors on critical arrest warrants. The police department study resulted in the re-assignment of six full-time instructors to their regular duties after the CBI modules were fully operational. This resulted in significant cost savings, as the instruction normally required one full year of each instructor. In the police study, the CBI instruction resulted in an average reduction in student completion time of 25%. The salary savings from this reduced time plus other considerations led to the conclusion that the CBI system resulted in a cost savings of $350,000 over a five-year period.

Moisey (1988) undertook to investigate the effects of testing frequency and feedback upon achievement and retention in a series of CAI modules in Canadian law. A sample of 171 volunteer police officers in a large urban force were divided into eight treatment groups. Four variations of testing were provided: after each module, after two modules, after four modules or after eight modules. Every subject received the same number of test questions. Two types of feedback were provided: immediately following each item, or after a 24 hour delay. All groups had significantly higher achievement and retention (10 week delay) scores than pretest scores. No significant differences were found for test frequency or feedback on the pretest, achievement test or retention test. More frequently tested groups had significantly higher module test scores than less frequently tested groups. More frequently tested groups spent significantly less time studying CAI modules than did less frequently tested groups and groups which received delayed feedback spent significantly more time on tests than groups which received immediate feedback. Moisey summarized her results by stating that

testing after each module coupled with immediate feedback clearly provides the most efficient training strategy. And, in comparison with other training strategies examined, it is also as effective. (1988, p. 133)

Computer Based Testing Studies

While we are careful to distinguish that computer based testing (CBT) is not CMI, there is a transferability of much of the work in CBT to CMI. Carbonaro (1988) undertook an extensive review of the capabilities of computerized test item banking programs and found fifty five individual features. He then surveyed 350 high school teachers in Alberta as to whether a computerized test item banking system would be useful to them and which features they would like to have. This study provides a comprehensive synthesis of computerized test item banking features.

Rumbolt (1989) undertook a study of the use of CBT in an apprenticeship training program. The study compared two systems of administration of the same test - one in which on-line testing using Quizmaster (a microcomputer CBI program) was used, and the other in which paper and pencil tests (using the same questions) were generated using the Computer Based Training System (a mainframe CMI system). A quasi-experimental design was used in which the two experimental groups "flip-flopped" between on-line and pencil and paper tests. A third (control) group was given only pencil and paper tests.

Rumbolt found no significant difference in the level of achievement between those doing on-line testing and those doing paper and pencil tests. Further, he found no significant difference in the attitude towards computers or in test anxiety between those doing on-line testing and those doing paper and pencil tests.

Rumbolt was particularly interested in actual cheating, or opportunity to cheat. No actual cheating was observed, and based upon responses to questions about cheating, Rumbolt states that "the on-line testing method was perceived as the system by which students were less likely to be able to cheat" (1989, p. 75).

Synthesis

An examination of these research studies shows that different studies concentrated on different aspects of CMI. The following areas were addressed by one or more of the studies: achievement, attitudes, cheating, instructor and student time, development time and costs, and delivery costs. Table 4 summarizes the results of the studies in each of these areas.

Conclusion

The research on pure CMI is thin. Much of the research on CBI includes components of CMI, so it is difficult to isolate the specific contribution made by CMI to the effectiveness of the instruction, attitudes towards computers, etc. Although there are single studies which do not confirm support for CMI, any scientist bases conclusions not on single studies but on the overall effect indicated by several similar studies. The very strong impression gained from a review of the studies reported here is that CMI has shown to be efficient, effective and relatively easy to develop compared with other forms of mediated instruction.

Experience indicates that CMI can be implemented by instructors, working alone or in teams. A team may or may not include computer analysts, depending upon the transparency and user interface of the particular CMI system being used. While there are advantages and disadvantages to working in teams, the former tend outweigh the latter. A comment often made within developmental teams is that one of the major benefits is the logical re-thinking of a course or curriculum required when implementing CMI.

CMI has been shown to be useful in the context of existing courses (e.g., Anderson, et al, 1975; Boblin & Gibson, 1986) as well as in brand new courses (Szabo & Estes, 1978). It has also been used as mainline instruction or as supplementary instruction, (the latter required or optional). CMI has been successfully applied over a wide range of content areas, ranging from nursing to special education to pilot training.

Although not systematically reported here, there is a rapid proliferation of microcomputer based CMI applications and programs as people begin to apply the microcomputer to the management of student learning. The only bit of advice we would offer here is for aspiring developers to carefully examine the features of the most outstanding mainframe-based CMI systems and avoid reinvention of the CMI wheel.

One might criticize CMI for being costly to develop and deliver. However, such conclusions need to be evaluated in the light of several factors.

1. In spite of years of formal instruction using conventional methods, there is not a clear picture of what it costs to create that conventional instruction.

2. In developing CMI, the heavy effort is at the beginning - in the design and development phases (which can be condensed if a good pool of items/objectives/ learning resources is available). Once the system is in place, it can be delivered repeatedly with minimal instructor involvement in clerical matters of test administration, etc.

3. The cost of computer hardware and software to handle CMI has steadily declined and by all indications will continue to decline.

4. There are costs which are incalculable but are much more important than hardware and software costs. One of these is the tremendous price a country pays in lack of basic literacy, a problem which can be partially addressed through individualized instruction. In addition, what costs are paid by educational institutions which are unable to lead in instructional technology and, in many cases, don't even keep up to provide a good 'mirror' of society and its march to technology?

Regardless of what we as instructors would like to believe about our work, many of our students will do anything to avoid our lectures; most of which do not contribute as much to learning as we instructors would like to believe. The data suggests that CMI can play a useful and productive role while satisfying student urges.

Contributors

Michael Szabo is a professor in the Department of Adult, Career and Technology Education at the University of Alberta. He has been involved in Computer Based Instruction as a researcher, instructor, course author and as the manager of CBI installations. His current research interests include instructional technology, individualized learning and visual learning.

Craig Montgomerie is an associate professor in the Department of Educational Administrationat the University of Alberta. He has been involved in Computer Based Instruction since 1971, as a programmer, researcher, course author and project manager. His current research interests include information systems and the use of computer technology to support distance education.

References

Allen, M. (1972). Computer managed instruction: A definitive design. In D. O. Lecarme & R. Lewis (Eds.), Computers in education (pp.115-122). Amsterdam: North Holland.

Anderson, T., Anderson, R., Dalgaard, B., Paden, D., Biddle, B., Durber, J., & Alessi, S. M. (1975). An experimental evaluation of a computer based study management system. Educational Psychologist, 11, 184-190.

Baker, F. (1981). Computer managed instruction: A context for computer based instruction. In H. F. O'Neil, (Ed.). Computer based instruction: A state-of-the-art assessment (pp. 23 - 64). Toronto: Academic Press.

Bloom, G.S. (1984). The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13 (4), 4-16.

Boblin, S.L. & Gibson, B. (1986). Computer managed instruction at the University of Alberta Hospitals School of Nursing. Unpublished master's project report, University of Alberta. Edmonton.

Carbonaro, M.D.. (1988). Computerized Test Item Banking:Features . Unpublished master's thesis. Edmonton: University of Alberta.

Carol, J. A. (1963). A model of school learning. Teachers College Record, 64, 723-733.

Cooley, W.W., & Glaser, R. (1969). An information management system for individually prescribed instruction. Pittsburgh: University of Pittsburgh Learning Research Development Center, (ERIC Document Reproduction Services No. ED 026 862).

Davenport, P. (1990, April 29). Letters to Editor, The Edmonton Journal.

Day, R., & Payne, L. (1987). Computer managed instruction: An alternative teaching strategy. Journal of Nursing Education, 26 (January), 30-36.

Fahy, P. (1985). Introducing individualization with computer managed learning: An example from adult basic education. Edmonton, AB: Alberta Vocational Centre, Edmonton. (ERIC Document Reproduction Service No. ED 260 258)

Fahy, P. (1987). PLATO computer managed learning report. Summative evaluation of PLATO computer managed learning in the nursing assistant program. Edmonton, AB: Alberta Vocational Centre, Edmonton. (ERIC Document Reproduction Service No. ED 283 043)

Flanagan, J.C. (1969). Program for learning in accordance with needs. Psychology in the Schools, 6, 133-136.

Franklin, S. & Marasco, J. (1977). Interactive computer based testing. Journal of College Science Teaching (September), 15-20.

Hansen, D.N. (1970). The role of computers in education during the 70's. Technical Memo #15). Florida: Computer Assisted Instruction Center, Florida State University.

Hunka, S.M. (1978). CAI: A primary source of instruction in Canada. Technological Horizons in Education, 5 , 32-37.

Hunka, S.M., Hunka, J., Kearsley, G., Maguire, T., Margolus, N., McGinnis, N., Montgomerie, T.C., & Romaniuk, G. (1978). Preliminary specifications for an instructional support system for NATAL-74. Contract OSU77-00260 with the National Research Council of Canada. Edmonton: Division of Educational Research Services, University of Alberta.

International Business Machines, Inc. (1968). A feasibility study of computer assisted instruction in United States Army basic electronic training: Final report. (Contract #DAAB 07-67-C-0578). Gaithersburg, MD: Author.

Kearsley, G.P. (1976). Some "facts" about CAI: A quantitative analysis of the 1976 Index to Computer Based Instruction. Journal of Computer Based Instruction, 3 (2), 34-41.

Kot, P., Skillen, L.D., & Wales, M. (1986). Learning effectiveness and efficiency using computer managed learning and lecture for health assessment theory (Final Report). Edmonton: University of Alberta.

Kulik, J. A., Kulik, C. C. & Cohen, P. A. (1980). Effectiveness of computer-based college teaching: A meta-analysis of findings. Review of Educational Research, 50, pp. 525-544.

Martin, J. M., & Szabo, M. (1990). The dialogue and computer-assisted instruction in chemistry. Journal of Computer-Based Instruction, 17, 41-45.

Moisey, S. D. (1988). The effect of test frequency and feedback on achievement and retention in a computer-based training program. Unpublished doctoral thesis. Edmonton: University of Alberta.

Montgomerie, T. C.. (1985). Alberta Vocational Centre/Edmonton nursing assistant program: Computer based learning investigation final report. Edmonton, AB: Alberta Vocational Centre/Edmonton.

Petruk, M. W. (1975). Effectiveness of a CAI-taught electrical theory course for training electrical apprentices. Edmonton: Alberta Advanced Education and Manpower.

Petruk, M. W. (1978). Evaluation of electrical apprentice training by CAI. Edmonton: Alberta Advanced Education and Manpower.

Romaniuk, E. W. (1970). A versatile authoring language for teachers. Unpublished doctoral thesis. Edmonton: University of Alberta.

Rumbolt, R. (1989). Apprenticeship and computer managed learning - a trial run by microcomputer. Unpublished master's thesis. Edmonton: University of Alberta.

Silberman, H. G. (1968). Design objectives of the instructional management system SP 3038/001/00, Santa Monica, CA: Systems Development Corporation.

Szabo, M. (1987). A cost-effective implementation of computer based training within a large government training organization. Interactive Learning International, 4, 81-84.

Szabo, M. & Estes, C. A. (April, 1978). A strategy for developing cost effective military instruction using available instructional materials (AIM). A paper presented at the meeting of the American Educational Research Association, Toronto

Tennyson, R. D. (1981). Use of adaptive information for advisement in learning concepts and rules using computer assisted instruction. American Educational Research Journal, 18, 425-438.

Tennyson, R.D., & Park, O. (1984). Computer-based adaptive instructional systems: A review of empirically based models. Machine-Mediated Learning, 1, 129-153.

Wang, A. (1976) Index to Computer Based Learning. Milwaukee: Instructional Media Laboratory, University of Wisconsin-Milwaukee.