Reprinted with permission of copyright holder.
Effects of Instruction on Search Success, and Satisfaction on the World Wide Web.
Johnson, B. L. and Szabo, M. (1997).
University of Alberta.
The purpose of this study was to determine if one hour of instruction in keyword selection plus Boolean logic would increase high school students levels of keyword selection and Boolean knowledge, proficiency in constructing complex search statements, success in finding topic related documents and satisfaction while searching a world wide web database. 125 students from a Western Canadian High School participated. Instruction in keyword selection plus Boolean logic significantly increased both student knowledge levels in using advanced search strategies (49.6% higher than control group), and correct student use of keywords and Boolean logic in constructing advanced search statements (39.5% higher). There was, however, no significant difference between the instructional treatment groups in the level of success or satisfaction experienced. Females experienced significantly higher levels of success than males, while both genders experienced similar levels of satisfaction.
In North American classrooms an increasing number students are turning to the World Wide Web and electronic databases as an information resource. Oley (1989) suggests that, "the increase in electronic storage of information indicates that information retrieval skills ... will become increasingly more relevant for students at all levels" (p. 591). However, there is often little guidance or structure given to the student, requiring them to utilize resources that have not been tailored to meet their specific information needs (Small & Ferreira, 1994). The recent appearance of the World Wide Web (WWW) databases offers a potentially valuable new source of information.
The increase in amounts, availability and usage of electronically stored information on the WWW databases suggests that effective search strategies are increasingly more relevant to students at all levels. According to Harris (1996):
the variety of search engines (such as WebCrawler) on the World Wide Web, are certainly powerful, useful tools that can help our students locate large numbers of diverse and timely documents. (p.36)
With the entrance of the WWW into most classrooms in North America, students of increasingly younger age are attempting to search the WWW databases. Studies of grade school students using bibliographic databases have shown that most students do not construct effective search statements or utilize successful search strategies as they lack the knowledge or skill, particularly at the elementary and junior high levels.. Oley (1989), Neuman (1995), and Solomon (1993) found that students had problems conducting effective searches due to problems in; generating search terms, using a limited number of synonyms, and combining two or more terms into a single search. Even at the high school level Chen (1993) noted that high school students are having some difficulty formulating proper search terms. Ala and Cerabona (1992) recommend that teaching Boolean search strategies should begin at the high school level. Clearly, there is a need for research into effective instruction on and application of search strategies by grade school students, particularly at the senior high levels. The aim of this research is to increase end-user success and satisfaction for those students who are starting to utilize the WWW databases as an information resource.
Until the recent appearance of the WWW databases, database-type studies predominantly examined end-user success and satisfaction with bibliographic databases. Furthermore, most of this research was done at the college or university level, as this is usually when students first encounter large electronic databases, such as ERIC, PsycLIT and MEDLINE (Moore & St. George, 1991; Solomon, 1993). In general these studies indicate that many end-users are not successfully utilizing Boolean logic in their search strategies, even though research has shown Boolean logic to be a dominant and potentially effective search strategy
Bibliographic Databases: End-user Search Strategy Errors
In considering the types of search strategies used, research has focused on problems encountered and mistakes made by searchers. Numerous studies point to the lack of general search strategies being used by end-users. Often end-users are not appropriately selecting search terms and are not effectively applying Boolean logic in their search strategies. Findings of the studies vary in regards to search strategies and errors made, particularly when the ages of the subjects are considered.
Elementary School Studies
Studies done at the elementary level suggest that most students simply lack the cognitive skills required to develop complex search strategies. Moore and St. George (1991), studied the cognitive demands of library systems on sixth graders and found that "children have great difficulty generating alternative terms and for nine- and ten-year-olds, 70% of the words selected would access no relevant information" (p. 164). In a study of elementary children's information retrieval behavior, Solomon (1993), found that there were often mismatches between the children's natural selection of search terms, and the actual terms used in the online catalogues database, syntax errors, the ability to use nouns and plural forms of words, and an inability to combine 2 or more terms into a single search.
High School Studies
Studies at the high school level also found students lacking in many basic search skills and concepts. In a survey of 25 library media specialists from 22 secondary schools, Neuman (1995) found that the major problems the students encountered in using online and CD-ROM databases included: overcoming mismatches between personal ideas of how information is organized versus actual ideas of how information is organized in databases, generating search terms, narrowing searches, designing effective search strategies, and designing searchable topics and questions. Chen (1993) also studied high school students use of an online catalogue. Chen noted that the students tended to use strings of words or major phrases taken directly from the search problem as their search term. These phrases frequently fell outside of the system's controlled vocabulary for the subject. As well, the students had difficulty extracting main concepts and expressing them using appropriate key words.
WWW Databases Versus Bibliographic Databases: Search Features and Strategies
Bibliographic databases are organized hierarchically, whereas the WWW databases are organized, and searched by document title, URL, content, or any combination of the three (Protherore & Wilson, 1994). Furthermore, WWW databases can differ remarkably in their approaches to indexing, searching and displaying results (Courtois, Baer, & Stark, 1995). Consequently different search strategies are often needed, with materials being searched in very different ways allowing for considerable potential to affect the end-users success and satisfaction. No studies were found in my search of the literature regarding end-user success and satisfaction in utilizing search strategies for WWW databases. Instead, research into end-user searching for the recently created WWW databases has focused mainly on comparing the different WWW databases in regards to their varying search features, and comparing these to bibliographic databases. As well, researchers have attempted to determine the most effective search strategies to effectively use with the different search engine features (Courtois, 1996; Courtois, Baer, & Stark, 1995; Zorn, Emanoil, Marshall and Panek,1996)
Tenopir and Cahn (1994) compared Boolean logic and relevance ranking using two new bibliographic database search engines, both offering natural language search techniques. Natural language input combined with relevancy ranking had an average precision of 56% compared to 61% for Boolean, on the DIALOG database while NEXIS FREESTYLE's database's average precision was 53% for natural language input combined with relevancy ranking compared to 64% for NEXIS Boolean. They do however point out that the better overall precision with Boolean should be contrasted with the greater number of total documents retrieved through relevance searching. Tenopir and Cahn recommend that relevance searching should be used when: doing a subject search, searching full-text databases or databases with lengthy abstracts, when a Boolean search is too broad and retrieves too many items or is too precisely specified and retrieves too few items. Boolean searching should be used when: you are looking for a known item or known citation, you want everything by a particular author, non-subject fields are an important part of your search, or your search has concepts that are of equal weight and you want everything on the topic. Tenopir and Cahn conclude that both methods are powerful search techniques, with neither one offering strong advantages over the other. Furthermore, although Boolean has the advantage of over twenty years of testing in real world searching, relevance searching as the newcomer has many enhancements yet to come and may soon be the most effective search feature.
Boolean Logic on the WWW
Bibliographic databases allow searchers to use a "building block" strategy. The searcher breaks the query down into its distinct conceptual elements, or building blocks. Each building block is entered in a separate statement, then all the blocks are combined in a single, final statement using combinations of Boolean operators (Siegfried,Bates, & Wilde, 1993). This allows the end-user to keep the distinct concepts clearly in mind and reduces confusion with using Boolean logic. Zorn, Emanoil, Marshall & Panek (1996) suggest that even experienced researchers prefer to split concepts and operators into multiple search statements. Unfortunately, WWW databases do not provide for the reuse of previously numbered search statements or sets. The entire search strategy must be entered in one statement, unlike OPAC's and online periodical indexing tools (Courtois, Baer & Stark, 1995; Zorn, Emanoil, Marshall & Panek, 1996; Webster and Paul, 1996). This has the potential to reduce the benefits of using Boolean logic as a search strategy on the WWW.
The Need for Teaching Electronic Search Strategies
It is becoming increasingly apparent that among the life skills students are going to need as they head towards the twenty-first century is the ability to manage the ever-expanding amount of information they are encountering. High school is the right time for these students to learn how to do this (Ala & Cerabona, 1992). To successfully do this our students need to effectively utilize the new information sources, including the WWW databases. The ability to analyze problem statements, and adopt problem-solving strategies in constructing search statements is essential. Teachers need to give more time and attention to teaching students advanced searching techniques including the use of synonyms, Boolean operators,and truncation of search terms (Bellardo, 1985).
There is no limit to the number of databases and amount of information that students have access to. It is important that students realize they can find specific information without being buried in a data avalanche. To do this they must learn how to conduct an effective search using Boolean logic as a key tool (Ala & Cerabona, 1992). Chen (1993) agrees, stating: " For productive searches, students must be able to read search problems or statements, extract key concepts, use appropriate terms to express the concepts" (p. 38).
Although many authors argue for the introduction of instruction in Boolean logic to allow students to effectively search electronic databases (Ala & Cerabona, 1992; Bellardo, 1985; Chen,1993) others (Tenopir and Cahn,1994) argue that Boolean logic is not an effective technique for full-text WWW databases. This study will attempt to resolve this by considering whether or not Boolean logic is an effective search strategy for searching a WWW database.
This study examined the effect of differing instructional treatments on the success and satisfaction of 125 Western Canadian senior high school students searching the WWW's Excite database. An experimental approach was used to determine the effect of two independant variables; differing search strategy instruction, and gender upon four dependant variables; search strategy knowledge levels, search statement construction proficiency, searcher success, and searcher satisfaction when using a WWW database that uses Boolean logic, and relevancy ranking together.
The instructional content involved keyword selection and Boolean logic search strategies. More specifically, the students received instruction on either keyword selection or keyword selection plus Boolean logic as compared to a control group that received no instruction on search strategies. Search strategy instruction required a one hour period using student work booklets with three pages of instruction, a two page worksheet, a self correction period, and a twenty item quiz.
Random assignment was done for all three treatment groups with each completing four posttests measuring keyword selection and Boolean knowledge levels, success in constructing search statements, success in finding topic related documents, and search satisfaction. Search strategy knowledge levels were determined by measuring both encoding and decoding skills used in keyword selection, and Boolean logic application. Statement construction was measured by analyzing how effectively the subject creates their Excite database search statements. Searcher success was determined by totaling the number of topic related documents retrieved for each of the three separate searches on the Excite database. Searcher satisfaction was determined using a twenty item likert-type scale.
The methodology is partly based on research done by Nahl and Harada (1996), Poohkay (1994), and Reese (1988).
Analyses of variance were conducted for each of the four posttest scores using treatment and gender as factors to examine if significant differences between the mean posttest scores of each treatment group existed. Where significant differences were found, ScheffÈ's F-tests were conducted to determine the statistical significance of the differences between the three treatment group means.
Keyword/Boolean Knowledge Posttest Scores
The analysis of variance yielded an F ratio of 171.86 which is statistically significant. An examination of sub-population posttest means was then performed. The findings are displayed in Table 1.
Table 1
(Table lost in conversion of text to digital form)
The keyword plus Boolean logic treatment group scored significantly higher on the keyword/Boolean knowledge posttest than both the keyword treatment group and the control group.
Statement Construction Posttest Scores
The analysis of variance yielded an F ratio of 25.1 which is also statistically significant. An examination of sub-population posttest means was then performed. The findings are displayed in Table 2.
Table 2
ScheffÈ Test of Significance for Statement Construction
(Table lost in conversion of text to digital form)
The keyword plus Boolean logic treatment group scored significantly higher on the keyword/Boolean knowledge posttest than the keyword treatment group, and the control group.
Searcher Success and Satisfaction Posttest Scores
The analysis of variance for searcher success yielded an F ratio of 2.00, while the analysis of variance for satisfaction yielded an F ratio of 0.44, both are not statistically significant.
Further supporting this finding was the regression analysis done between the keyword/Boolean knowledge and searcher success posttest scores. An r value of 0.02 was determined, the analysis of variance yielded an F value of 0.05 which was not statistically significant. In other words, subjects with higher levels of keyword/Boolean knowledge did not tend to experience any higher levels of success in finding topic related documents than subjects with lower levels of keyword/Boolean knowledge.
The Effect of Gender upon Search Success and Satisfaction
Females experiences significantly higher levels of success than males. The two by three factor analysis of variance yielded an F ratio of 4.64 for gender which is statistically significant. An examination of sub-population posttest means was then performed. The findings are displayed in Table 3.
Table 3
Summary of Searcher Success Posttest Means By Gender and Treatment
(Table lost in conversion of text to digital form)
In all three treatment groups females had higher mean scores than males for searcher success. Total searcher success mean scores are significantly greater for the females.
The two by three factor analysis of variance for gender and satisfaction yielded an F ratio of 0.25 which was not statistically significant.
Based on these results, the instructional methodology and practices used in teaching keyword selection plus Boolean logic are effective in improving subjects keyword selection and Boolean knowledge levels compared to other instructional treatments used in this study. This result supports Nahl and Harada (1996) who found that students receiving instruction in keyword selection and Boolean logic performed better on a posttest measuring keyword selection and Boolean knowledge than those not receiving this instruction. Support is also given to Leverence (1994) who suggests that a formal search strategy session followed by the opportunity to create search statements in writing before going on-line is the best way to teach twenty-five to thirty people the necessary tools they need to become successful searchers, and ensure logical, well planned search strategies.
The instruction in keyword selection plus Boolean logic was also more effective in improving a subjects correct use of keywords and Boolean logic in constructing search statements as compared to the other two instructional treatments. This finding lends support to the findings of Nahl and Harada (1996) who found that students receiving instruction in keyword selection and Boolean logic application produced search statements containing a higher percentage of correctly used Boolean operators.
The instruction in keyword selection plus Boolean logic was not more effective in improving a subjects success in finding topic related documents as compared to the other two instructional treatments. These findings contradict the recommendations of Ala and Cerabona (1992), and Bellardo (1985). Ala and Cerabona suggest that high school students need to learn how to conduct effective searches using Boolean logic. Bellardo suggests that teachers need to give more time and attention to teaching students advanced searching techniques including the use of synonyms (keyword selection), and Boolean operators. Support is however given to Tenopir and Cahn (1994) who suggest that Boolean searching should not be done when searching full-text databases (such as Excite), and instead searchers should rely on the relevancy ranking features of the search engine, which is what many of those in the control group lacking keyword/Boolean knowledge did.
Instruction in keyword selection plus Boolean logic was no more effective in improving a subjects satisfaction levels than then other instructional treatments. This contradicts Jackson-Brown and Pershing (1993) who found that when using ERIC and PsycLIT databases, trained searchers expressed greater levels of satisfaction with search results than searchers with no training. Perhaps the differences in findings may relate to the length of training, amount of practice, or maturity of the subjects. These findings do lend support to Lepoer and Mularski (1989), and Ankeny (1991). Lepoer and Mularski's research revealed that most searchers in their study seemed to be satisfied with their searches regardless of their previous levels of training or experience.
Females were significantly more successful than the males in finding topic related documents, regardless of the instructional treatment used. This contradicts the notion, or perhaps adds clarification to the conventional wisdom that boys do better at computer related activities than girls. Neither gender's satisfaction was significantly more affected by the treatments, regardless of the type of instructional treatment they received. Once again, information about the male advantage in computer related activities may be questioned.
The insignificant correlation between keyword/Boolean knowledge levels and searcher success suggests that subjects with higher levels of keyword/Boolean knowledge are no more likely to experience higher levels of success than subjects with low levels of keyword/Boolean knowledge. This finding further substantiates the recommendations of Tenopir and Cahn (1994), and the findings of Zorn, Emanoil, Marshall & Panek, (1996). Tenopir and Cahn suggest that Boolean searching should not be done when searching full-text databases such as those found on the WWW. Zorn et al. argue that the inability of WWW search interfaces to allow searchers to split concepts and operators into multiple search statements, then combine them into a more complex search statement leads to confusion when using Boolean logic. Consequently, Boolean logic may actually inhibit success when searching on a WWW database.
Based on the results of this study, it is recommended that teachers of high school students can increase student levels of keyword/Boolean logic knowledge by using work booklet based instruction on these search strategies. Furthermore, the students receiving this instruction will have a greater likelihood of being able to construct search statements that correctly use keywords combined with Boolean operators and parentheses.
The findings of this study do suggest that teachers of high school students planning to utilize full text WWW search engines as a resource tool should not spend instructional time teaching Boolean logic as a search strategy in the hope of increasing student success in finding topic related documents or increasing student satisfaction with that search engine.
Ala, J. and Cerabona, K. (1992). "Boolean searches: A life skill. School Library Journal, 38 (November), 42.
Ankeny, M. L. (1991). Evaluating end-user services: Success or satisfaction? Journal of Academic Librarianship, 16, 352-356.
Bellardo, T. (1985). An investigation of online searcher traits and their relationship to search outcome. Journal of the American Society for Information Science, 36, 241-50.
Chen, S. (1993). A study of high school students' online catalog searching behavior. School Library Media Quarterly, 22,(Fall) 33-39.
Courtois, M. P., Baer, W. M., & Stark, M. (1995). Cool tools for searching the web: A performance evaluation Online, 6, 14-32.
Courtois, M. P. (1996). New trends in WWW services. Online, May/June, 30-36.
Harris, J (1996). Mining the internet. Learning and Leading with Technology, December/January, 36-39
Jackson-Brown, G. & Pershing, G. (1993). Comparisons of Graduate and Undergraduate End-users of ERIC and PsycLIT on CD-ROM. (ERIC Document Reproduction Service No. ED 372 738)
LePoer, P. M. & Mularski, C. A. (1989). CD-ROM's impact on libraries and users. Laserdisk Professional, 2 (4), 39-45.
Leverence, M. E. (1994). Time-tested ERIC Bibliographic Instruction (BI): Give Them a Pole and Teach Them to Fish (Fiche). (ERIC Document Reproduction Service No. ED 381 123)
Moore, P. A. , & St. George, A. (1991). Children as information seekers: The cognitive demands of books and library systems. School Library Media Quarterly, 19, 161-168.
Nahl, D. & Harada, V. H. (1996). Composing Boolean search statements: Self-confidence, concept analysis, search logic, and errors. School Library Media Quarterly, 24, 199-205.
Neuman, D. (1995). High school students' use of databases: Results of a national delphi study. Journal of the American Society for Information Science, 46, 284-98.
Oley, E (1989). Information retrieval in the classroom. Journal of Reading, 590-597.
Poohkay, B. (1994). Effects of Computer-displayed Animation on Achievement and Attitude in Mathematics Computer Based Instruction. Unpublished master's thesis, University of Alberta, Edmonton, Alberta, Canada.
Protherore, N. & Wilson, E. (1994). The Internet Handbook for School Users. (ERIC Document Reproduction Service No. ED 375 821)
Reese, C. (1988). Manual indexes versus computer aided indexes: Comparing the Readers' Guide to Periodical Literature to InfoTracII. RQ, 27, 384-389.
Siegfried, S., Bates, M. J., & Wilde, D. N. (1993). A profile of end-user searching behavior by humanities scholars: The Getty online searching project report no. 2. Journal of the American Society for Information Science, 44, 273-291.
Small R. V. & Ferreira, S. M. (1994). Information location and use, motivation, and learning patterns when using print or multimedia information resources. Journal of Educational Multimedia and Hypermedia, 3 (3/4), 251-273.
Solomon, P. (1993). Children's information retrieval behavior: A case analysis of an OPAC. Journal of the American Society for Information Science, 44, 245-64.
Tenopir, C. and Cahn, P. (1994). TARGET & FREESTYLE: DIALOG and Mead join the relevance ranks. Online, 18 (3), 31-47.
Webster, K. & Paul, K. (1996) Beyond Surfing: Tools and Techniques for Searching the Web [On-Line]. Available: http://magi.com/~mmelick/it96jan.htm#search engines
Zorn, P., Emanoil, M., Marshall, L., & Panek, M. (1996). Web searching. Tricks of the trade. Online May/June, 15-28.