The purpose of this study was to determine if one hour of instruction in keyword selection plus Boolean logic would increase high school students levels of keyword selection and Boolean knowledge, proficiency in constructing complex search statements, success in finding topic related documents and satisfaction while searching a world wide web database. 125 subjects from a Western Canadian High School participated. Instruction in keyword selection plus Boolean logic significantly increased both student knowledge levels in using advanced search strategies (49.6% higher than control group), and correct student use of keywords and Boolean logic in constructing advanced search statements (39.5% higher). There was, however, no significant difference between the instructional treatment groups in the level of success or satisfaction experienced. Females experienced significantly higher levels of success than males, while both genders experienced similar levels of satisfaction.

1. Introduction

An increasing number students around the world are turning to the World Wide Web and electronic databases as an information resource. The recent appearance of the World Wide Web (WWW) databases offers a potentially valuable new source of information. [Oley 1989] suggests that, "the increase in electronic storage of information indicates that information retrieval skills ... will become increasingly more relevant for students at all levels" (p. 591). However, there is often little guidance or structure given to the student, requiring them to utilize resources that have not been tailored to meet their specific information needs [Small and Ferreira 1994].

Students at increasingly younger ages are attempting to search the WWW databases. Studies of grade school students using bibliographic databases have shown that most students do not construct effective search statements or utilize successful search strategies as they lack the knowledge or skill, particularly at the elementary and junior high levels.. [Oley 1989; Neuman 1995; Solomon 1993] found that students had problems conducting effective searches due to problems in; generating search terms, using a limited number of synonyms, and combining two or more terms into a single search. At the high school level [Chen 1993] noted that high school students are having some difficulty formulating proper search terms. Clearly, there is a need for research into effective instruction on and application of search strategies by grade school students, particularly at the senior high levels. The aim of this research is to increase end-user success and satisfaction for those students who are starting to utilize the WWW databases as an information resource.

2. Literature Review

Recently database-type studies predominantly examined end-user success and satisfaction with bibliographic databases, mostly at the college or university level, when students first encounter large electronic databases, such as ERIC, PsycLIT and MEDLINE [Moore and St. George 1991; Solomon 1993]. In general these studies indicate that many end-users are not successfully utilizing Boolean logic in their search strategies, even though research has shown Boolean logic to be a dominant and potentially effective search strategy.

2.1 Bibliographic Databases: End-user Search Strategy Errors

Research has focused on problems encountered and mistakes made by searchers and points to the lack of general search strategies being used by end-users. Findings of the studies vary in regards to search strategies and errors made, particularly when the ages of the subjects are considered.

Studies done at the elementary level suggest that most students simply lack the cognitive skills required to develop complex search strategies. [Solomon 1993] for example, found that there were often mismatches between the children's natural selection of search terms, and the actual terms used in the online catalogues database, syntax errors, the ability to use nouns and plural forms of words, and an inability to combine 2 or more terms into a single search.

Studies at the high school level also found students lacking in many basic search skills and concepts. [Neuman 1995] found that the major problems the students encountered in using online and CD-ROM databases included: overcoming mismatches between perceived ideas and the reality of how of how information is organized in databases, generating search terms, narrowing searches, designing effective search strategies, and designing searchable topics and questions. [Chen 1993] noted that the students tended to use strings of words or major phrases taken directly from the search problem as their search term; phrases which frequently fell outside of the system's controlled vocabulary for the subject.

2.2 WWW Databases Versus Bibliographic Databases: Search Features and Strategies

Bibliographic databases are organized hierarchically, whereas the WWW databases are organized, and searched by document title, URL, content, or any combination of the three [Protherore and Wilson 1994]. Furthermore, WWW databases can differ remarkably in their approaches to indexing, searching and displaying results [Courtois 1996; Courtois and Baer and Stark 1995]. Consequently different search strategies are often needed, with materials being searched in very different ways allowing for considerable potential to affect the end-users success and satisfaction. Research into end-user searching for the recently created WWW databases has focused mainly on 1) comparing the features of different WWW databases and 2) determining the most effective search strategies to effectively use with the different search engine features [Courtois 1996; Courtois and Baer and Stark 1995; Zorn and Marshall and Panek 1996].

Tenopir and Cahn [1994] compared Boolean logic and relevance ranking using two new bibliographic database search engines, both offering natural language search techniques. Natural language input combined with relevancy ranking had an average precision of 56% compared to 61% for Boolean, on the DIALOG database while NEXIS FREESTYLE's database's average precision was 53% for natural language input combined with relevancy ranking compared to 64% for NEXIS Boolean. They contrasted better overall precision with Boolean vs. the greater number of total documents retrieved through relevance searching and made recommendations on when relevance or Boolean searching should be used. Tenopir and Cahn [1994] conclude that both methods are powerful search techniques, with neither one offering strong advantages over the other. Furthermore, although Boolean has the advantage of over twenty years of testing in real world searching, relevance searching as the newcomer has many enhancements yet to come and may soon be the most effective search feature.

2.3 Boolean Logic on the WWW

Bibliographic databases allow searchers to use a "building block" strategy. The searcher breaks the query down into its distinct conceptual elements, or building blocks. Each building block is entered in a separate statement, then combined in a single, final statement using combinations of Boolean operators [Siegfried and Bates and Wilde 1993]. This allows the end-user to keep the distinct concepts clearly in mind and reduces confusion with using Boolean logic. Zorn and Marshall and Panek [1996] suggest that even experienced researchers prefer to split concepts and operators into multiple search statements. Unfortunately, WWW databases do not provide for the reuse of previously numbered search statements or sets. The entire search strategy must be entered in one statement, unlike OPAC's and online periodical indexing tools [Courtois and Baer and Stark 1995; Zorn and Marshall and Panek 1996; Webster and Paul 1996] This has the potential to reduce the benefits of using Boolean logic as a search strategy on the WWW.

2.4 The Need for Teaching Electronic Search Strategies

It is becoming increasingly apparent that among the life skills students are going to need as they head towards the twenty-first century is the ability to manage the ever-expanding amount of information they are encountering [Ala and Cerabona 1992]. The ability to analyze problem statements, and adopt problem-solving strategies in constructing search statements is essential. High school teachers need to give more time and attention to teaching students advanced searching techniques including the use of synonyms, Boolean operators, and truncation of search terms [Bellardo 1985].

Student access to databases and amount of information that students can access is enormous and growing. Students can find specific information without being buried in a data avalanche but to do this they must learn how to conduct an effective search using Boolean logic as a key tool [Ala and Cerabona 1992]. Chen [1993] agrees, stating: " For productive searches, students must be able to read search problems or statements, extract key concepts, use appropriate terms to express the concepts" (p. 38).

Although many authors argue for the introduction of instruction in Boolean logic to allow students to effectively search electronic databases, Ala and Cerabona [1992] Bellardo [1985] Chen [1993] and Tenopir and Cahn [1994] argue that Boolean logic is not an effective technique for full-text WWW databases. This study investigated this by considering whether or not Boolean logic is an effective search strategy for searching a WWW database.

3. Method

This study examined the effect of differing instructional treatments on the success and satisfaction of 125 Western Canadian senior high school subjects searching the WWW's Excite database. An experimental approach was used to determine the effect of two independent variables; differing search strategy instruction, and gender upon four dependent variables; search strategy knowledge levels, search statement construction proficiency, searcher success, and searcher satisfaction when using a WWW database that uses Boolean logic, and relevancy ranking together.

Subjects received instruction on either keyword selection or keyword selection plus Boolean logic as compared to a control group that received no instruction on search strategies. Search strategy instruction required a one hour period using student work booklets with three pages of instruction, a two page worksheet, a self correction period, and a twenty item quiz. They were randomly assigned to one of three treatment groups with each completing four posttests measuring keyword selection and Boolean knowledge levels, success in constructing search statements, success in finding topic related documents, and search satisfaction. Search strategy knowledge levels were determined by measuring both encoding and decoding skills used in keyword selection, and Boolean logic application. Statement construction was measured by analyzing how effectively the subject creates their Excite database search statements. Searcher success was determined by totaling the number of topic related documents retrieved for each of the three separate searches on the Excite database. Searcher satisfaction was determined using a twenty item Likert-type scale. The methodology is partly based on research done by Nahl and Harada [1996], Szabo and Poohkay [1996] and Reese [1988].

4. Data Analysis and Findings

Two factor analyses of variance were conducted for each of the four posttest scores using treatment and gender as factors with Scheffé's F-tests to examine any differences among the three treatment group means.

The analysis of variance for knowledge posttest scores yielded a significant ratio. An examination of sub-population means was then performed. The keyword plus Boolean logic treatment group scored significantly higher on the keyword/Boolean knowledge posttest than both the keyword treatment group and the control group. The findings are displayed in Table 1 [Tab. 1].

Significant differences in statement construction scores were found, revealing that keyword plus Boolean logic treatment group scored significantly higher on the keyword/Boolean knowledge posttest than the keyword treatment group, and the control group. Performance of the control and keyword only groups was equivalent Table 2 [Tab. 2].

Comparison	Mean Difference	Scheffé F-Test
Control vs. Keyword	-2.67	10.14*
Control vs. Keyword+Boolean	-9.95	151.45*
Keyword vs. Keyword+Boolean	-7.28	75.96*

* p < .05.

[Tab. 1] Table 1. Scheffé Test of Significance for Keyword/Boolean Knowledge Posttest Scores

Comparison	Mean Difference	Scheffé F-Test
Control vs. Keyword	-0.60	1.45
Control vs. Keyword+Boolean	-2.37	24.27*
Keyword vs. Keyword+Boolean	-1.77	12.66*

* p < .05.

[Tab. 2] Table 2. Scheffé Test of Significance for Statement Construction Posttest Scores

Results for searcher success yielded no significant differences among the groups, while the analysis of variance for satisfaction yielded an F ratio of 0.44, both within reasonable probability of error. In other words, subjects with higher levels of keyword/Boolean knowledge did not tend to experience any higher levels of success in finding topic related documents than subjects with lower levels of keyword/Boolean knowledge.

Females experiences significantly higher levels of search success than males. In all three treatment groups females had higher mean scores than males for searcher success. Total searcher success mean scores were significantly higher for the females than the males. Analysis of the search satisfaction dependent variable showed equal levels of search satisfaction for males and females Table 3 [Tab. 3].

Gender
	Male		Female
Treatment	N	Mean	N	Mean
Control	24	74.71	19	72.05
Keyword	21	74.95	17	75.35
Keyword + Boolean	23	70.65	21	75.67
Totals	68	73.41	57	74.37

* p < .05.

[Tab. 3] Table 3. Summary of Searcher Success Posttest Means By Gender and Treatment

5. Discussion

Based on these results, the instructional methodology and practices used in teaching keyword selection plus Boolean logic are effective in improving subjects keyword selection and Boolean knowledge levels compared to other instructional treatments used in this study. This result is consistent with research findings of Nahl and Harada [1996] and Leverence [1994] who support keyword selection and Boolean and the development of a formal search strategy before going on line to search.

The instruction in keyword selection plus Boolean logic was also more effective in improving a subject's correct use of keywords and Boolean logic in constructing search statements as compared to the other two instructional treatments. This finding lends support to the findings of Nahl and Harada [1996] who found that students receiving instruction in keyword selection and Boolean logic application produced search statements containing a higher percentage of correctly used Boolean operators.

The instruction in keyword selection plus Boolean logic was not more effective in improving a subjects success in finding topic related documents as compared to the other two instructional treatments in contrast to Ala and Cerabona [1992] and Bellardo [1985] who concluded that high school students need to learn how to conduct effective searches using Boolean logic as taught by teachers. The findings are also consistent with Tenopir and Cahn [1994] who suggest that Boolean searching should not be done when searching full-text databases (such as Excite); searchers should rely on the relevancy ranking features of the search engine, which is what many of those in the control group lacking keyword/Boolean knowledge did.

Instruction in keyword selection plus Boolean logic was no more effective in improving a subjects satisfaction levels than then other instructional treatments. This contradicts Jackson-Brown and Pershing [1993] who found that when using ERIC and PsycLIT databases, trained searchers expressed greater levels of satisfaction with search results than searchers with no training. Perhaps the differences in findings may relate to the length of training, amount of practice, or maturity of the subjects. These findings lend support to high levels of search satisfaction reported by Ankeny [1991].

Females were more successful than the males in finding topic related documents, regardless of the instructional treatment used. This adds clarification to the conventional wisdom that boys do better at computer related activities than girls. Neither gender's satisfaction was significantly more affected by the treatments, regardless of the type of instructional treatment they received. Once again, information about the male advantage in computer related activities may be questioned.

The lack of significant correlation between keyword/Boolean knowledge levels and searcher success suggests that subjects with higher levels of keyword/Boolean knowledge are no more likely to experience higher levels of success than subjects with low levels of keyword/Boolean knowledge and is consistent with Tenopir and Cahn [1994], and the findings of Zorn and Marshall and Panek [1996] . The former advise against using Boolean methodologies when searching full-text WWW databases. Zorn and Marshall and Panek [1996] have argued that the inability of WWW search interfaces to allow searchers to split concepts and operators into multiple search statements, then combine them into a more complex search may actually inhibit success when searching on a WWW database.

Based on the results of this study, it is recommended that teachers of high school students can increase student levels of keyword/Boolean logic knowledge by using work booklet based instruction on these search strategies. Furthermore, the students receiving this instruction will have a greater likelihood of being able to construct search statements that correctly use keywords combined with Boolean operators and parentheses.

The findings of this study suggest that teachers of high school students planning to utilize full text WWW search engines as a resource tool should not spend instructional time teaching Boolean logic as a search strategy in the hope of increasing student success in finding topic related documents or increasing student satisfaction with that search engine.

6. References

[Ala Cerabona 1992] Ala, J. and Cerabona, K. (1992). "Boolean searches: A life skill. School Library Journal, 38 (November), 42.

[Ankeny 1991] Ankeny, M. L. (1991). Evaluating end-user services: Success or satisfaction? Journal of Academic Librarianship, 16, 352-356.

[Bellardo 1985] Bellardo, T. (1985). An investigation of online searcher traits and their relationship to search outcome. Journal of the American Society for Information Science, 36, 241-50.

[Chen 1993] Chen, S. (1993). A study of high school students' online catalog searching behavior. School Library Media Quarterly, 22, (Fall) 33-39.

[Courtois Baer Stark 1995] Courtois, M. P., Baer, W. M., & Stark, M. (1995). Cool tools for searching the web: A performance evaluation Online, 6, 14-32.

[Courtois 1996] Courtois, M. P. (1996). New trends in WWW services. Online, May/June, 30-36.

[Jackson-Brown and Pershing 1993] Jackson-Brown, G. & Pershing, G. (1993). Comparisons of Graduate and Undergraduate End-users of ERIC and PsycLIT on CD-ROM. (ERIC Document Reproduction Service No. ED 372 738)

[Leverence 1994] Leverence, M. E. (1994). Time-tested ERIC Bibliographic Instruction (BI): Give Them a Pole and Teach Them to Fish (Fiche). (ERIC Document Reproduction Service No. ED 381 123)

[Moore and St. George 1991] Moore, P. A. , & St. George, A. (1991). Children as information seekers: The cognitive demands of books and library systems. School Library Media Quarterly, 19, 161-168.

[Nahl and Harada 1996] Nahl, D. & Harada, V. H. (1996). Composing Boolean search statements: Self-confidence, concept analysis, search logic, and errors. School Library Media Quarterly, 24, 199-205.

[Neuman 1995] Neuman, D. (1995). High school students' use of databases: Results of a national delphi study. Journal of the American Society for Information Science, 46, 284-98.

[Oley 1989] Oley, E (1989). Information retrieval in the classroom. Journal of Reading, 590-597.

[Protherore and Wilson 1994] Protherore, N. & Wilson, E. (1994). The Internet Handbook for School Users. (ERIC Document Reproduction Service No. ED 375 821)

[Reese 1988] Reese, C. (1988). Manual indexes versus computer aided indexes: Comparing the Readers' Guide to Periodical Literature to Info TracII. RQ, 27, 384-389.

[Siegfried and Bates and Wilde 1993] Siegfried, S., Bates, M. J., & Wilde, D. N. (1993). A profile of end-user searching behavior by humanities scholars: The Getty online searching project report no. 2. Journal of the American Society for Information Science, 44, 273-291.

[Small and Ferreira 1994] Small R. V. & Ferreira, S. M. (1994). Information location and use, motivation, and learning patterns when using print or multimedia information resources. Journal of Educational Multimedia and Hypermedia, 3 (3/4), 251-273.

[Solomon 1993] Solomon, P. (1993). Children's information retrieval behavior: A case analysis of an OPAC. Journal of the American Society for Information Science, 44, 245-64.

[Szabo and Poohkay 1996] Szabo, M. & Poohkay, B. (1996). An experimental study of animation, mathematics achievement and attitude toward computer assisted instruction. Journal of Research on Computing in Education, 28, 56-69.

[Tenopir and Cahn 1994] Tenopir, C. and Cahn, P. (1994). TARGET & FREESTYLE: DIALOG and Mead join the relevance ranks. Online, 18 (3), 31-47.

[Webster and Paul 1996] Webster, K. & Paul, K. (1996) Beyond Surfing: Tools and Techniques for Searching the Web [On-Line]. Available: http://magi.com/~mmelick/it96jan.htm#search engines

[Zorn and Marshall and Panek 1996] Zorn, P., Emanoil, M., Marshall, L., & Panek, M. (1996). Web searching. Tricks of the trade. Online, May/June, 15-28.