University of Alberta Edmonton, Canada
AN EVALUATION REPORT:
THE UNIVERSITY STUDIES
PROJECT
A collaborative distance
education venture between the Special Sessions
Office, University of Alberta,
Fairview College, and Kayas Cultural College
Special Sessions
David A. Keast, Ph.D.
September 3, 1996
© Copyright 1996 University of
Alberta
All Rights Reserved
Contemporary Debates
in Distance Education Research
The implementation and evaluation of this project takes place at a time and within a milieu of heated debate on contemporary issues in distance education. One such issue is the effect of technological media on the structuring of knowledge, and indeed, on how different technological media represent reality. One point of view suggests that devices (the technological medium), through influencing and even creating new methods and techniques of delivery, influence educational culture and the way we view the learning process and may even affect students' creation and understanding of content (Bates, 1995). On the other hand, extensive empirical research has been conducted on technological media, levels of interaction, satisfaction levels, and student achievement (see numerous studies including Fulford & Zhang, 1993; Kendall & Oaks, 1992; Ritchie & Newby, 1989; Russell 1992, 1993, 1994; Stubbs & Burnham, 1990; and Tobin, 1995). Although different instructional delivery environments lend themselves to differing levels of interaction, and in turn, the extent of interaction (Kendall & Oaks, 1992) and perceptions of interaction (Fulford & Zhang, 1993) impact on student satisfaction, no clear and positive link has been established with student performance.
This "no significant difference" phenomenon (see Russell 1992, 1993, 1994) also raises an additional question on the influence of satisfaction. Although type of media, levels of interaction, and other variables such as student support (Tallman, 1994) cannot be shown to directly affect achievement, they do appear to influence student satisfaction levels (Fulford & Zhang, 1993; Kendall & Oaks, 1992; Ritchie & Newby, 1989; Tallman, 1994). The question then arises: How important is satisfaction? Although the link between satisfaction and achievement is dubious at best, student satisfaction will most surely influence motivational levels, retention and completion in courses and programs, and students' tendencies to return to a medium again for additional learning (Biner, Dean, & Mellinger, 1994). Thus, aside from the educationally desirable condition of providing learning to satisfied clients, much needed longitudinal research may shed valuable light on the relationship between satisfaction and the future of distance education programs.
A significant percentage (slightly over 70%) of students in the University Studies Project are of Aboriginal descent. Thus, a further unavoidable question in evaluation relates to how the needs of this particular group were met. Research on minority populations and distance education is sparse, and studies pertaining to Aboriginal learners specifically appears scarce indeed. Tobin (1995) suggests that "review of the literature did not find that research has paid attention to a wide spectrum of the disparate learner groups" (p.204). Given cultural traits endemic to Aboriginal heritage such as the centrality of oral traditions, an orientation to the present, the importance of face-to-face communication, and the importance of community and group solidarity, the use of synchronous, highly interactive technology along with structured schedules of learning and a clearly defined support system does not seem an unreasonable approach. Delivery systems demanding a high degree of learner autonomy may not be appropriate, and learner independence most likely should not be an immediate program goal or a measure of program effectiveness. Indeed, Spronk (1995) is quite clear on what not to do: "What clearly does not work with Aboriginal learners is the home study model" (p.91).
Finally, the research findings discussed above
point to a need for additional study focused on the
administration of distance education. This need was clearly
articulated by Bates as early as 1981. He called for
"preparation programs or at least courses for planners and
managers of distance education and particularly audio-visual
media; ..." (p. 89). Paul (1995) indicates a lack of
emphasis in distance education research on emerging technologies
and innovations, and instead, a preoccupation with
well-established and traditionally researched technological and
educational problems. Tobin (1995) suggests a number of areas
within the administration of distance education in need of
further study including organizational change which has occurred
or is required in relation to distance delivery, satisfaction
with and effectiveness of organizational structures established
for distance education, the nature and extent of
inter-organizational cooperation, and professional development
for the changing roles of faculty within a milieu of rapid
technological and educational advancement. Moore (1994)
reinforces the need for examination of administrative issues. He
suggests that "the barriers impeding the development of
distance education are not technological, nor even
pedagogical.... The major problems are associated with
organizational change, change in faculty roles, and change in
administrative structures" (p.4). It is within this context
of the need for administrative understanding that the present
program evaluation is discussed.
Project History and Description
As early as February 1995, informal discussions took place between administrators at Fairview College and the Director of Special Sessions concerning the possibility of this program. Initial discussion focused on the provision of Faculty of Arts, university credit courses as the academic structure desirable for the project. In late May and early June 1995 the Special Sessions office received two formal program proposals (one from the President of Fairview College, and the other from the Vice-President (Academic)), which outlined the parameters of the program and the respective institutional responsibilities. Implementation of the program began in late June 1995, and was essentially complete by the start of the 1995-1996 academic year, with most major operational elements in place.
Given the brief time frame for implementation, an interim arrangement for admission status of students was negotiated with the Office of the Registrar at the University of Alberta. For the first year of the program, students were registered in the "Unclassified Student" status, with the option to carry more than one course per term so as to facilitate the completion of one year of university studies. The program reflects a potentially streamlined administrative approach for initiation into university studies for new groups of students which at the same time avoids the bureaucratic complexities of university transfer. All courses offered (except Learning Strategies 130) were University of Alberta first-year Faculty of Arts courses, taught and evaluated by University of Alberta faculty appointed by their respective academic departments. Courses were administered through the Special Sessions office. In September 1995, the student cohortfor this program consisted of 39 students at three major sites in the Fairview College region including Fairview, Peace River, and High Level, and the Kayas Cultural College, which, in turn, subdivides into three sites including John D'or Prairie, Fox Lake, and Garden River.
The mode of delivery for all courses in the program was synchronized, multi-point videoconferencing. Multi-point videoconferencing allows for full audio and visual interaction between instructors and students, and among student groups at each site. In addition, the technology is supplemented with a document camera, video recording equipment, and at the University of Alberta site, computer interface with the - videoconferencing medium. The equipment is controlled at each site, and by the instructor on the University of Alberta site, through a switcher keypad or console at one end of the room opposite T.V. monitors. The fact that all sites are equipped for this mode of electronic communication, combined with the installation of a new central bridging unit at Fairview College, made this form of delivery an attractive option. However, important questions needed to be answered on the impact of this medium on teaching and learning. This was suggested as one component of the evaluation.
The basic responsibilities of each institution were defined in the initial Fairview College proposals. Roughly, the University of Alberta provided the requested courses, appointed instructors, distributed textbooks, administered the admission and registration of students, and provided some library support. The responsibilities of Fairview College and Kayas Cultural College included making accessible the facilities and equipment including the central bridging site at Fairview College, recruiting and screening students, promotion of the project within their own regions, coordination of registration within their own regions, and establishing a student support system consisting of program advisors or counsellors, technical support, and tutors.
As an added component of this project, Special
Sessions and Fairview College established Internet capability as
a form of instructional support. E-mail communication between
University of Alberta instructors and counsellors at Fairview
College sites was possible, along with access to the Special
Sessions WWW site, which contained materials for each of the
courses being delivered. It was hoped that this medium would
enhance communication and turn-around time for assessment, and
transfer of documents and feedback.
Purpose
A detailed formative evaluation of the
University Studies Project was begun toward the latter part of
the first academic term in late November to early December of
1995. Plans to include the evaluation as a crucial part of
implementation were set a number of months earlier. The main
purpose of the evaluation was to provide a comprehensive
assessment of the operation and effectiveness of the University
Studies Project, and to provide the necessary information base
for decisions on the future development of other similar
initiatives.
Evaluation Questions
This evaluation examined six general aspects or dimensions of the University Studies Project which, in the context of this report, are best expressed as research questions or question areas as follows:
What is the demographic profile of students
participating in this project?
What planning and administrative issues emerge
as obstacles to the effectiveness of the project, and what areas
are identified by respondents for improvement?
What curricular and instructional methods and
adaptations were employed by teaching faculty in order to
accommodate this type of delivery medium?
What is the nature and extent of support
services needed for student success?
What is the perceived educational effectiveness
of the delivery medium itself?
Overall, how satisfied were students and other
respondents with various aspects of the program, and how well
were students needs served?
Examination of these aspects of the project was
approved by major stakeholders at all institutions and
subsequently used in designing data collection instruments and as
a conceptual framework for the analysis and interpretation of
data.
6. SUMMARY AND RECOMMENDATIONS
Within the context of the evaluation questions
presented in Section 1, this final section provides a summary of
findings drawn from more detailed analyses of data in previous
sections, and offers a number of specific recommendations based
on these findings.
Summary of Findings
Student Profile
The profile of students participating in this project is quite distinct. On average, slightly more than 90% of participants were 25 years of age or older and female. A significant percentage (around 67%) of respondents had dependent children. Many students also indicated some college level training, a college or technical school diploma, or completion of a general education diploma or high school diploma. However, review of interviews with respondents suggests that many of these students were not ideally prepared for university studies.
Implementation time did not allow for a planned
promotional campaign. This is reflected in findings since over 50%
of respondents first became aware of the project through word
of mouth. Approximately 70% of students indicated that they were
employed either full-time or part-time prior to admission, and a
significant percentage (76%) stated that their main reason for
course enrollment was that they planned to seek university
transfer and on-campus degree completion.
Planning and Administration
Based on students surveys, and through analysis of other data sets, planning and administrative issues stood out as one major problem area. Concern was raised, particularly by students and support staff, over the status of students in the program. Despite efforts by the Registrar's office and Special Sessions at the University of Alberta to clarify student status and provide the necessary information on admission and course registration, some question seemed to persist over the "credibility" and transferability of courses being offered. Students expressed a desire for more course and program information during the admissions and registration process, more effective program advising and counselling support, and improvement in turnaround time for screening and acceptance into courses. In addition, support staff members expressed concern over lack of knowledge of the program and lack of understanding of policy and procedures on admission and registration.
When students were questioned on course
scheduling, the majority of students preferred the more
conventional course schedules. Courses offered for one hour three
times a week or one-and-a-half hours twice a week were preferred
choices. 8:00 am to 4:00 p.m. weekdays was the preferred time
block.
Technology
Overall, students seemed satisfied with various
aspects of the videoconferencing medium such as concentration,
interaction, attendance, and disruptions. However, review of
written comments by students showed concern over the extent of
student-instructor communication, and also over student access to
instructors through additional teleconferencing "office
hours." Students complained of the need for more convenient
scheduling of office hours, and more immediate access to
instructors. Despite the highly interactive real-time nature of
the videoconferencing medium, many students indicated that they
would have preferred more face-to-face contact with instructors
through on-site visits. Findings also indicate general
satisfaction with environmental conditions and videoconferencing
quality. However, concern was expressed by students over the need
for adequate study space outside of class times.
Instructional Methods and Adaptations
The instruction provided and course content delivered was clearly indicated as a strength of this program. Student responses indicated "high praise" for the quality of instruction by the University of Alberta faculty, and general satisfaction with the content of courses. Results of course and instructional assessments conducted by university departments through the Special Sessions office corroborate these findings.
Additional comments by instructors revealed some differences in perception on the effectiveness of the medium and other instructional factors. The problems of maintaining student attention, engaging students in dialogue, and reading students' nonverbal cues were magnified somewhat within this distance education context. In general, however, all instructors viewed the technology as flexible and adaptable. Instructors also commented on the problem of student readiness for participation in university studies, and their general lack of contact with and involvement in on-site support. Instructors were generally not aware of the extent of academic support students were receiving. Opinions varied slightly on the amount of preparation time required compared to conventional instruction, however, there was general agreement on the teaching and learning potential available through the use of this medium, given adequate resources and time for course development.
One additional point emerges in relation to
instructional adaptation. Defining the nature of cultural
differences and how such differences impact on the teaching and
learning process was not a focus of this investigation. However,
more sensitivity to cultural differences could enhance the
quality of programs such as this. Concern was expressed by one
researcher over the manner in which instructors and Native
students related to one another and communicated during on-line
broadcasts. Differences in cultural orientation and learning
styles, and differences in first language use may have had an
impact on the depth of communication with Aboriginal students,
which, in this study, remains undetermined. Overall, however,
curricular and instructional adaptations did not appear to be an
overriding concern for respondents.
Student Support Systems
Findings on the extent and value of contact with counsellors, tutors, and instructors outside of class time are somewhat surprising and perhaps mildly alarming. It was found that significant percentages of students had much less contact with support staff and instructors than was originally assumed (see Figure 3.3). Since the question of whether students actually needed consultation and attempted to seek it out was not explored, the significance of these findings and the impact on student achievement is unclear. As well, due to missing responses on some question items, exact numbers of students who had contact and those that did not is unclear. However, a picture emerges of significant numbers of students who did not consult with support staff and instructors to the extent originally hoped for. This may be somewhat reflected in students' overall rating of program counselling services. The mean for this question was a rather modest 2.86 (SD = 1.36) on a 5-point scale.
Overall, the most frequently mentioned weakness of the project was poor turnaround times for transfer of documents, assignments, and instructor feedback and evaluations. This was somewhat predictable given the geographic obstacles present. Students, support staff, and instructors consistently reported inefficient turnaround times. This finding points clearly to the need for well-defined, enhanced support in this area. Technological support in the form of Internet capability was not developed or used to the extent originally anticipated.
Counsellors, instructors, and particularly
students also reported a need for more university library
support. Initially, attempts had been made to define and
establish electronic access to library resources, however, given
the constraints involved, it was not possible to provide library
support to the extent originally hoped for. Students called for
access to more books and other materials, and the possibility of
small reference collections made available at various sites.
General Aspects
General questions were also asked on the extent of student learning, whether respondents would recommend the program to others, and on program strengths and weaknesses. Over half of the students indicated ratings of 4 or 5 on a 5-point scale for the extent of learning which they felt had occurred. A high percentage of students (96.2% of those responding) indicated that they would recommend such a program to others. Program strengths ranged from access to university studies within the home community, to preparation for the university experience. Weaknesses ranged from perceived interaction and communication problems and lack of various resources, to the need for more program information and support, and the problem of turnaround times. Weaknesses and changes suggested by students were congruent with findings from other data sources. The reader is encouraged to examine previous sections of this report for a more detailed discussion of findings.
Despite these inadequacies, this program can be
considered a moderate success, given the 51% completion rate and
an overall mean GPA for those completing of5.2 (on a 9-point
scale). In one course (Eng 101), those students who completed
clearly fell within the expected GPA range for first-year,
on-campus, university students.
Recommendations
The following recommendations are offered as
general guidelines for action in the design, development, and
implementation of other similar initiatives. It is recommended
that:
Specific
Recommendation 1: Adequate planning time
be established beforehand for the implementation of these types
of programs.
Recommendation 2: Strategies be
developed for the assessment of students' entry level knowledge
and skills, and other academic needs for university studies.
Recommendation 3: Distribution of
textbooks and other materials be centralized or otherwise
organized to facilitate prompt delivery and timely student access
to necessary course materials.
Recommendation 4: An orientation be
provided for all students on the technological systems used to
deliver instruction.
Recommendation 5: Instructors receive
resources and support for course development and use of the
technology.
Recommendation 6: The roles of all
support personnel be clearly defined with the possibility of
designing a pre- service and even in-service support staff
training and development program. Such a program would have at
least the following characteristics:
6.1 A support staff training and development
program must be collaboratively defined, involving major
stakeholders at all participating institutions;
6.2 Support staff require adequate orientation
in the University of Alberta policiesand procedures which form
the necessary administrative structure for course offerings;
6.3 Adequate training is required on whatever
technology is being used both for course delivery and
instructional support;
6.4 Orientation is required on the academic
component (subject-matter and instructional) of the program which
will allow support staff to provide an adequate level of
high-quality, on-site, academic assistance;
6.5 The roles and responsibilities of support staff need to be formalized, with adequate funding and resources set aside to support this formalization.
Recommendation 7: Three-way lines of
communication be established among instructors, students, and
support staff sufficient for addressing students' academic needs
and satisfying educational achievement goals. This will include:
7.1 Mechanisms to improve communication between
instructors and students; and,
7.2 Mechanisms to enhance communication between
instructors and on-site support staff.
Recommendation 8: Mechanisms be
established to improve turnaround times for transfer of
documents, assignments, instructor feedback, and evaluations.
These mechanisms could be technological or conventional, but must
be oriented toward time efficiency. Recommendation 9:
Students be given access to University of Alberta services
commensurate with fees paid for those services. Adequate library
access was identified as a key component, and is necessary for
student success.
Recommendation 10: Time and resources be
directed toward the planning and implementation of program
promotion and advertising.
Recommendation 11: Continued dialogue be
maintained among inter-institutionalstakeholders in order to
monitor the impact of an ongoing project on the student learning
experience.
Recommendation 12: The institutions
involved recognize other forms of community support which may be
necessary and available locally. These might include:
12.1 adequate study space;
21.2 transportation to classes for sites where
roads and weather conditions present major obstacles;
12.3 community daycare for students with
dependent children;
12.4 supplementary funding for additional
materials, resources, or personnel.
General
Recommendation 13: Policies and
procedures be formulated at institutional levels in order to:
13.1 accommodate the needs of distance
education students; and,
13.2 enhance inter-institutional cooperation
where such cooperation plays an important role in program
delivery at a distance.
Recommendation 14: For programs of this nature, significant emphasis be placed on the design of various forms of student support as a key factor related to program success.