JAMES H. MCMILLAN
Virginia Commonwealth University

HarperCollins College Publishers

Executive Editor: Christopher Jennison
Project Coordination and Text Design: Ruttle, Shaw & Wetherill, Inc.
Cover Designer: Kay Petronio
Electronic Production Manager: Christine Pearson
Electronic Page Makeup: Ruttle, Shaw & Wetherill, Inc.
Manufacturing Manager: Helene G. Landers
Printer and Binder: R. R. Donnelley & Sons Company
Cover Printer: The Lehigh Press, Inc.

For permission to use copyrighted material, grateful acknowledgment is made
to the copyright holders on pp. 358-359, which are hereby made part of this
copyright page.

Educational Research: Fundamentals for the Consumer, Second Edition

Copyright C 1996 by James H. McMillan

HarperCollins ® and ® are registered trademarks of HarperCollins Publishers Inc.

All rights reserved. Printed in the United States of America. No part of
this book may be used or reproduced in any manner whatsoever without
written permission, except in the case of brief quotations embodied in
critical articles and reviews. For information address HarperCollins College
Publishers, 10 East 53rd Street, New York, NY 10022. For information about
any HarperCollins title, product, or resource, please visit our World Wide Web site at
http://www.harpercollins.com/college.

Library of Congress Cataloging-in-Publication Data

McMillan, James H.
Educational research: fundamentals for the consumer /James H.
McMillan. - 2nd ed. Ed.
p.cm.
Includes bibliographical references and index.
ISBN 0-673-99864-9
1. Education-Research,1. Title.
LB1028.M2815 1996
370'.78-dc20 95-16506 CIP 95 96 97 98 9 8 7 6 5 4 3 2 1

Contents

To the Instructor	xv
To the Student	xix

1.	Introduction to Research in Education1
	SOURCES OF KNOWLEDGE2 Personal Experience2Tradition3 Authority3The Scientific Approach4
	THE NATURE OF SCIENTIFIC INQUIRY4 The Purpose of Scientific Inquiry4 Characteristics of Scientific Inquiry5 The Purpose of Theories6
	APPLYING SCIENTIFIC INQUIRY TO EDUCATION7
	TYPES OF EDUCATIONAL RESEARCH9 Two Traditions of Research: Quantitative and Qualitative9Basic Research10 Applied Research10Action Research12 Evaluation Research12Nonexperimental Research12Experimental Research13
	FORMAT TO REPORT EDUCATIONAL RESEARCH14 Title and Author(s)16Abstract16 Introduction16Review of Literature16 Specific Research Question or Hypothesis17 Method and Design17Results17 Discussion17Conclusions18 References18

v

viCONTENTS

	ANATOMY OF A RESEARCH ARTICLE18 OUTLINE SUMMARY18 STUDY QUESTIONS27 SAMPLE TEST QUESTIONS28
2.	Variables, Research Problems, and Hypotheses31
	VARIABLES IN EDUCATIONAL RESEARCH32 Constitutive and Operational Definitions32 Types of Variables33
	RESEARCH PROBLEMS36 Sources for Research Problems39
	CONSUMER TIPS: CRITERIA FOR EVALUATING RESEARCH PROBLEMS42
	HYPOTHESES46 Why Researchers Use Hypotheses46 Types of Hypotheses47
	CONSUMER TIPS: CRITERIA FOR EVALUATING RESEARCH HYPOTHESES49
	OUTLINE SUMMARY51
	STUDY QUESTIONS52
	SAMPLE TEST QUESTIONS53
3.	Locating and Reviewing Related Literature55
	THE PURPOSE OF REVIEWING RELATED LITERATURE56 Refining the Research Problem56 Developing Significance for the Research56 Identifying Methodological Techniques57 Identifying Contradictory Findings57 Developing Research Hypotheses57 Learning About New Information57

CONTENTSvii

	STEPS TO REVIEW RELATED LITERATURE58 Step One: Locate Existing Reviews and Other Information in Secondary Sources58 Step Two: Identify Key Terms65 Step Three: Identify the Appropriate Journal Indexes and Abstracts67 Step Four: Search Indexes for Primary Sources68 Step Five: Summarize and Analyze Primary Source Information74 Step Six: Organize the Review76
	CONSUMER TIPS: CRITERIA FOR EVALUATING THE REVIEW OF LITERATURE77
	OUTLINE SUMMARY80
	STUDY QUESTIONS81
	SAMPLE TEST QUESTIONS82
4.	Subjects and Sampling84
	INTRODUCTION TO SAMPLING85 What Is a Subject?85What Is a Population?85What Is a Sample?86
	TYPES OF SAMPLING PROCEDURES86 Probability Sampling86Nonprobability Sampling91
	HOW SUBJECTS AND SAMPLING AFFECT RESEARCH94 Knowledge of Sampling Procedures94 Volunteer Samples94Sample Size96 Subject Motivation97Sample Bias98
	CONSUMER TIPS: CRITERIA FOR EVALUATING SUBJECTS SECTIONS OF REPORTS AND SAMPLING PROCEDURES98
	OUTLINE SUMMARY100
	STUDY QUESTIONS101
	SAMPLE TEST QUESTIONS101

viiiCONTENTS

5.	Foundations of Educational Measurement104
	INTRODUCTION TO MEASUREMENT105 Definition of Measurement105 The Purpose of Measurement for Research106 Scales of Measurement106
	FUNDAMENTAL PRINCIPLES OF DESCRIPTIVE STATISTICS FOR UNDERSTANDING MEASUREMENT108 Frequency Distributions109Measures of Central Tendency112Measures of Variability113Correlation115
	VALIDITY OF EDUCATIONAL MEASURES118 Definition of Validity118Types of Evidence for Judging Validity119Effect of Validity on Research122
	RELIABILITY OF EDUCATIONAL MEASURES123 Types of Reliability124Effect of Reliability on Research127
	OUTLINE SUMMARY129
	STUDY QUESTIONS130
	SAMPLE TEST QUESTIONS131
6.	Types of Educational Measures134
	CLASSIFYING EDUCATIONAL MEASURES135
	TESTS136 Norm- and Criterion-Referenced Tests136 Standardized Tests137Interpreting Test Scores141
	PERSONALITY ASSESSMENT143

CONTENTSix

	ATTITUDE, VALUE, AND INTEREST INVENTORIES144 Types of Inventories145Problems in Measuring Noncognitive Traits148
	OBSERVATIONS150 Inference150 Laboratory Observation151 Structured Field Observations152 Observer Effects153
	INTERVIEWS154 Types of Interview Questions155 Interviewer Effects155
	LOCATING AND EVALUATING EDUCATIONAL MEASURES157
	CONSUMER TIPS: CRITERIA FOR EVALUATING INSTRUMENTATION158
	OUTLINE SUMMARY162
	STUDY QUESTIONS164
	SAMPLE TEST QUESTIONS165
7.	Descriptive, Correlational, and Causal-Comparative Research 167
	THE PURPOSE OF NON-EXPERIMENTAL RESEARCH168
	DESCRIPTIVE STUDIES168 Characteristics of Descriptive Studies168
	CONSUMER TIPS: CRITERIA FOR EVALUATING DESCRIPTIVE STUDIES170
	RELATIONSHIP STUDIES171 Relationship Determined by Differences171 Simple Correlational Studies172 Prediction Studies176
	CONSUMER TIPS: CRITERIA FOR EVALUATING CORRELATIONAL STUDIES178

xCONTENTS

	USING SURVEYS IN DESCRIPTIVE AND RELATIONSHIP STUDIES182 Cross-Sectional Surveys182 Longitudinal Surveys183
	CAUSAL-COMPARATIVE STUDIES184 Ex Post Facto Research184Correlational Causal-Comparative Research186
	CONSUMER TIPS: CRITERIA FOR EVALUATING CAUSAL-COMPARATIVE RESEARCH186
	OUTLINE SUMMARY187
	STUDY QUESTIONS188
	SAMPLE TEST QUESTIONS189
8.	Experimental and Single-Subject Research192
	CHARACTERISTICS OF EXPERIMENTAL RESEARCH193
	EXPERIMENTAL VALIDITY194 History195Selection196 Maturation196Pretesting197 Instrumentation197Treatment Replications198Subject Attrition198 Statistical Regression198 Diffusion of Treatment199Experimenter Effects199Subject Effects200
	TYPES OF EXPERIMENTAL DESIGNS201 Single-Group Posttest-Only Design201 Single-Group Pretest-Posttest Design202 Nonequivalent-Groups Posttest Only Design203 Nonequivalent-Groups Pretest-Posttest Design204 Random ized-Grou ps Posttest-Only Design206 Randomized-Groups Pretest-Posttest Design207 Factorial Experimental Designs209
	CONSUMER TIPS: CRITERIA FOR EVALUATING EXPERIMENTAL RESEARCH210

CONTENTSxi

	SINGLE-SUBJECT RESEARCH212 Characteristics of Single-Subject Research212 Types of Single-Subject Designs213
	CONSUMER TIPS: CRITERIA FOR EVALUATING SINGLE,SUBJECT RESEARCH215
	OUTLINE SUMMARY217
	STUDY QUESTIONS218
	SAMPLE TEST QUESTIONS219
9.	Analyzing Statistical Inferences221
	THE PURPOSE AND NATURE OF INFERENTIAL STATISTICS221 Degree of Certainty222Estimating Errors in Sampling and Measurement222The Null Hypothesis223
	INTERPRETING RESULTS OF INFERENTIAL TESTS225 The t-Test226Simple Analysis of Variance227Factorial Analysis of Variance228Analysis of Covariance229 Multivariate Statistics230Chi-Square230
	CONSUMER TIPS: CRITERIA FOR EVALUATING INFERENTIAL STATISTICS232
	OUTLINE SUMMARY234
	STUDY QUESTIONS234
	SAMPLE TEST QUESTIONS235
10.	Qualitative and Historical Research238
	QUALITATIVE RESEARCH239 Characteristics of Qualitative Research239 Qualitative Research Problems241

xiiCONTENTS

	Entering the Research Site243Selecting Participants243Obtaining Qualitative Information244Analyzing Qualitative Data248Credibility of Qualitative Research250
	CONSUMER TIPS: CRITERIA FOR EVALUATING QUALITATIVE RESEARCH253
	HISTORICAL RESEARCH254 The Historical Method255
	CONSUMER TIPS: CRITERIA FOR EVALUATING HISTORICAL RESEARCH258
	OUTLINE SUMMARY259
	STUDY QUESTIONS261
	SAMPLE TEST QUESTIONS262
11.	Analyzing Discussion and Conclusions265
	PURPOSE AND NATURE OF THE DISCUSSION265
	INTERPRETATION OF THE RESULTS266 Interpretation Related to the Problem and/or Hypothesis266Interpretation Related to Methodology266 Interpretation Based on Statistical Procedures269Interpretation Related to Previous Research270
	CONCLUSIONS272 Limitations273Recommendations and Implications276
	CONSUMER TIPS: CRITERIA FOR EVALUATING DISCUSSION SECTIONS277
	OUTLINE SUMMARY279
	STUDY QUESTIONS279
	SAMPLE TEST QUESTIONS280

CONTENTSxiii

12.	The Intelligent Consumer: Putting It All Together282
	QUESTIONS FOR QUANTITATIVE STUDIES283
	QUESTIONS FOR QUALITATIVE STUDIES286
	QUESTIONS FOR HISTORICAL STUDIES287
	EXAMPLES OF RESEARCH ARTICLES288 Article 1: A Study of Academic Time-on-Task in the Elementary School288Evaluation of Article 1298Article 2: Reducing Teacher Stress304Evaluation of Article 2316 Article 3: Kindergarten Readiness and Retention: A Qualitative Study of Teachers' Beliefs and Practices322 Evaluation of Article 3349

Appendix: Answers to Sample Test Questions	352
References	353
Acknowledgments	358
Index	360

It was not too long ago that I sat, somewhat nervously, in a university auditorium waiting for my first class in educational research. Perhaps you have had, or will have, a similar experience. I distinctly remember thinking, given what I had heard about "research," that I needed to learn only enough to pass the course and would not have to worry about it again! It was another hurdle that I was forced to jump to graduate. I was not bad in mathematics but my interest was in working with people, not numbers. It was incomprehensible that I would someday teach and write about educational research. But something happened to me as I grudgingly struggled through the course. What I discovered was that research is a way of thinking, a tool that I could use to improve the work I do with other people. My hope is that this book can instill a similar disposition in you, providing knowledge, skills, and attitudes to improve your life and the welfare of others. Although learning the content and skills needed to become an intelligent consumer of research is not easy, my experience in teaching hundreds of students is that you will improve yourself, professionally and otherwise, through your efforts. In the beginning, especially as you read research articles, not everything will make sense. But as your experience in being an informed consumer increases, so will your understanding. Good luck and best wishes, and please write to me if you have suggestions for improving the book.

James H. McMillan

xix

		CHAPTER
Subjects and Sampling		4

The third major part of research reports is the methodology or methods section. As noted in Chapter 1, the first subsection of the methodology section usually describes the subjects from whom data are collected. The manner in which subjects are selected has important implications for identifying factors that affect subject performance and for
84 SUBJECTS AND SAMPLING85
generalizing the results. Hence it is necessary to understand who the subjects are and how they were selected. INTRODUCTION TO SAMPLING What Is a Subject?
A subject is an individual who participates in a research study or is someone from whom data are collected. In experiments, for example, each person who is given a treatment and whose behavior is measured is considered to be a subject. The term subject may also identify individuals whose behavior, past or present, is used as data, without their involvement in some type of treatment or intervention. For instance, a researcher might use last year's fourth-grade test scores as data, and each fourth-grader included is considered to be a subject, In qualitative research individuals are identified as participants rather than subjects.		Subject: Person from whom data are collected.
What Is a Population? A population is a group of elements or cases, whether individuals, objects, or events, that conform to specific criteria and to which we intend to generalize the results of the research. This group is also referred to as the target Population or universe. The specification of the population begins with the research problem and review of literature, through which a population is described conceptually or in broad terms, for example, seventh-grade students, beginning teachers, principals, special education teachers, and so forth. A more specific definition is then needed, based on demographic characteristics. These characteristics are sometimes referred to as delimiting variables. For example, in a study of first grade minority students, there are three delimiting characteristics: students, first grade (age), and minority. Further delimiting variables should be added to provide as precise a definition as possible. What about geographic region, socioeconomic status, gender, type of community, and types of schools? Are both public and private students included? How is "minority" defined? It is also important to distinguish the target population from a list of elements from which a group of subjects is selected, which is termed the survey population or sampling frame. In a study of beginning teachers, the target population may be beginning teachers across the United States, in all types of schools. The survey population may be a list of beginning teachers that was obtained from four states. Although the intent may be all beginning teachers, the results are limited, or delimited, to beginning teachers in the four states. Thus, generalization from subjects to populations should be based on the survey population.		Population: Persons to whom results can be generalized.
86CHAPTER 4
What Is a Sample? The sample is the group of elements, or a single element, from which data are obtained. Although the phrase "the sample included . . ." is used to indicate the characteristics of the people or events in the sample, the nature of the sampling procedure is usually described by one or more adjectives, such as random sampling or stratified random sampling. These types of sampling procedures are defined, with illustrations from actual studies, in the following section. It is important for the researcher to define as specifically as possible both the sampling procedure and the characteristics of the sample used in the study. Here is an example of a good description of the sample.		Sample: Group of subjects from whom data are collected.
Example: Description of a Sample The sample for this study consisted of nine seventh-grade mathematics teachers and their students. All teachers had volunteered for the study and each teacher received a $100 stipend. The teachers taught in four public schools in a medium-sized Western city, a low-middle to middle class community with a small proportion of minorities. According to a school district brochure, the district had approximately 32, 000 students in 42 elementary schools, 9 middle schools, and 9 high schools during the school year the study was conducted. The class sizes ranged from 16 to 34. There were 5 female'and 4 male teachers. The teachers had an average of approximately 11 years, of teaching experience (range = 2-26). Of these 11 years, 8 were as math teachers (range 2 = 22) and 6 were as middle-school math teachers (range = 2-10). All had secondary certification, and 4 had Master's degrees in Administration. One of the teachers had a math major in college, 6 a math minor, and 2 had no special training in math." (Burns and Lash, 1986, p 395)
TYPES OF SAMPLING PROCEDURES The purpose of sampling is to obtain a group of subjects who will be representative of the larger population or will provide specific information needed. The degree of representativeness is based on the sampling technique employed. I will first describe different sampling procedures and then consider the strengths and weaknesses of each in obtaining a representative sample. Probability Sampling In social science and educational research it is usually impractical and unnecessary to measure all the elements in the population of interest. Typically, a relatively small number of subjects or cases is selected from
SUBJECTS AND SAMPLING87
the larger population. The goal is to select a sample that will adequately represent the population, so that what is described in the sample will also be true of the population. The best procedure for selecting such a sample is to use probability sampling, a method of sampling in which the subjects are selected randomly in such a way that the researcher knows the probability of selecting each member of the population. Random selection implies that each member of the population as a whole or of subgroups of the population has an equal chance of being selected. As long as the number of cases selected is large enough, it is likely that a very small percentage of the population, represented by the sample, will provide an accurate description of the entire population. It should be noted, however, that there is always some degree of error in sampling, and that error must be considered in interpreting the results of the sample. In probability sampling this calculation can be made very precisely with some statistical procedures. Consider a population of 1,000 third-graders, from which you will select randomly 5 percent, or 50, to estimate the attitudes of all the third-graders toward school. If the attitude score was 75 for the sample of 50 subjects, 75 can be used to estimate the value for the entire population of third-graders. However, if another sample of 50 students is selected, their score might be a little different, say 73. Which one is more correct? Since all 1,000 students have not been tested to obtain the result we do not know for sure, but the results can be used to estimate the error in sampling. This is basically the technique that political polls follow when it is reported that the vote is 45 percent ñ 3. The plus or minus 3 is the estimate of error in sampling. There are many types of probability sampling procedures. You will probably encounter four types in educational research: simple random, systematic, stratified, and cluster.		Probability sampling: Known probability of selection from the population.
Simple Random Sampling In simple random sampling every member of the population has an equal and independent chance of being selected for the sample. This method is often used with a small number in the population, for example, putting the names or numbers of all population members in a hat and drawing some out as the sample. If every member of the population can be assigned a different number, a table of random numbers can identify the population members that will make up the sample. This approach is not convenient if the population is large and not numbered. The most common way of selecting a random sample from a large population is by computer. There are computer programs that will assign numbers to each element in the population, generate the sample numbers randomly, and then print out the names of the people corresponding to the numbers.		Simple random sampling: Each member of the population has the same probability of being selected.
88CHAPTER 4
Simple random sampling is illustrated in the following study of mothers' strategies for influencing their children's schooling.
Example: Simple Random Sampling "We interviewed a sample of 41 mothers of eighth graders from one middle school. These mothers were randomly selected from a list of 129 mothers provided by the principal of the school."(Baker and, Stevenson, 1986, p. 157)
Systematic Sampling In systematic sampling every nth element is selected from a list of all elements in the population, beginning with a randomly selected element. Thus, if there is a need to select 100 subjects from a population of 50,000, every nth element would correspond to every 500th subject. The first element is selected randomly. In this example that would be some number between I and 500. Suppose 240 were randomly selected as a starting point. The first subject chosen for the sample would be the 240th name on a list, the next subject would be the 740th, then the 1,240th, and so on until 100 subjects were selected. Systematic sampling is virtually the same as simple random sampling. It is certainly much more convenient. There is a possible weakness in systematic sampling if the list of cases in the population is arranged in a systematic pattern. For instance, if a list of fourth-graders in a school division is arranged by classroom and students in the classrooms are listed from high to low ability, there is a cyclical pattern in the list (referred to as periodicity). If every nth subject that is selected corresponds to the pattern, the sample would represent only a certain level of ability and would not be representative of the population. Alphabetical lists do not usually create periodicity and are suitable for choosing subjects systematically.		Systematic sampling:Every nth member of the population is selected.
Stratified Sampling A modification of either simple random or systematic sampling is first to divide the population into homogeneous subgroups and then select subjects from each subgroup, using simple random or systematic procedures, rather than the population as a whole. This is termed stratified sampling. The strata are the subgroups. Stratified sampling is used primarily for two reasons. First, as long as the subgroups are identified by a variable related to the dependent variable in the research (e.g., socioeconomic status in a study of achievement) and results in more homogeneous groups, the sample will be more representative of the population than if taken from the population as a whole. This result reduces error and means that a smaller sample can be chosen.		Stratified sampling: Subjects are selected from strata or groups of the population.
SUBJECTS AND SAMPLING89
Second, stratified sampling is used to ensure that an adequate number of subjects is selected from different subgroups. For example, if a researcher is studying beginning elementary school teachers and believes that there may be important differences between male and female teachers, using simple random or systematic sampling would probably not result in a sufficient number of male teachers to study the differences. It would be necessary in this situation first to stratify the population of teachers into male and female teachers and then to select subjects from each subgroup. The samples can be selected in one of two ways. A proportional stratified sample, or proportional allocation, is used when the number of subjects selected from each stratum is based on the percentage of subjects in the population that have the characteristic used to form the stratum. Thus, in the previous example, if 5 percent of the population of elementary teachers is male, 5 percent of the sample would also be male teachers.		Proportional stratified sampling: Reflects proportion of stratum in population.
A second approach is to take the same number of subjects from each stratum, regardless of the percentage of subjects from each stratum in the population. This method is used often because it ensures that a sufficient number of subjects will be selected from each stratum. For instance, if only 10 percent of a population of 200 elementary teachers are male, a proportional sample of 40 would include only 4 male teachers. To study male teachers it would be better to include all 20 male teachers in the population for the sample and randomly select 20 female teachers. This sampling procedure is referred to as disproportional because the number of subjects in the sample from each subgroup is not proportional to the percentage of the subgroups in the population. Disproportional stratified sampling is not limited to taking the same number of subjects from each subgroup. When disproportional sampling is used the results of each stratum need to be weighted to estimate values for the population as a whole. In the following example disproportional stratified sampling ensures that the same number of first and third graders are selected randomly.		Disproportional stratified sampling: Number of subjects in each strata does not reflect proportion in population.
Example: Disproportional Stratified Sampling "From a pool of all children who returned a parental permission form (more than 80% return rate) 24 first graders (10 girls, 14 boys; mean 6 years 6 months), and 24 third graders (13 girls, 11 boys; mean age, 8 years, 8 months) were randomly selected." (Clements and Nastasi, 1988, p. 93)
Stratified random sampling is illustrated in Figure 4.1. In this example the population is divided first into three different age groups,
90CHAPTER 4
Figure 4.1 Example of stratified sampling with two strata.
then by gender. Once the groups are stratified by gender, random samples are selected from each of the six subgroups. Cluster Sampling When it is impossible or impractical to sample individual elements from the population as a whole, usually when there is no exhaustive list of all the elements, cluster sampling is used. Cluster sampling involves the random selection of naturally occurring groups or areas and then the selection of individual elements from the chosen groups or areas. Examples of naturally occurring groups would be universities, schools, school divisions, classrooms, city blocks, and households. For example, if there is a need to survey a state for the television viewing habits of middle school students, it would be cumbersome and difficult to select children at random from the state population of all middle-schoolers. A clustering procedure could be employed by first listing all the school divisions in the state and then randomly selecting 30 school divisions from the list. One middle school could then be selected from each division, and students selected randomly from each school. This is a multistage clustering procedure. Although cluster sampling saves time and money, the results are less accurate than other random sampling techniques.		Cluster sampling: Naturally occurring groups are selected.
SUBJECTS AND SAMPLING91
Nonprobability Sampling In many research designs it is either unfeasible or unnecessary to obtain a probability sample. In these situations a nonprobability sample is used. A nonprobability sample is one in which the probability of including population elements is unknown. Usually, not every element in the population has a chance of being selected. It is also quite common for the population to be the same as the sample, in which case there is no immediate need to generalize to a larger population. In fact you will find that much of the educational research reported in journals, especially experimental studies, uses a group of subjects that has not been selected from a larger population.		Nonprobability sample: Probability of selection not known.
Convenience Sampling A convenience sample is a group of subjects selected because of availability, for example, a university class of a professor conducting some research on college students, classrooms of teachers enrolled in a graduate class, schools of principals in a workshop, people who decide to go to the mall on Saturday, or people who respond to an advertisement for subjects. There is no precise way of generalizing from a convenience sample to a population. Also, the nature of the convenience sample may bias the results. For example, if the available sample for studying the impact of college is the group of alumni who return on alumni day, their responses would probably be quite different from those of all alumni. Similarly, research on effective teaching that depends on teachers in a particular geographic area, because they are available, may result in different findings than research done in other geographic areas. Although we need to be very wary of convenience samples, often this is the only type of sampling possible, and the primary purpose of the research may not be to generalize but to better understand relationships that may exist. Suppose a researcher is investigating the relationship between creativity and intelligence, and the only available sample is a single elementary school. The study is completed, and the results indicate a moderate relationship: Children who have higher intelligence tend to be more creative than children with lower intelligence. Because there was no probability sampling, should we ignore the findings or suggest that the results are not valid or credible? That decision seems overly harsh. It is more reasonable to interpret the results as valid for children similar to those studied. For example, if the school serves a low socio-economic area, the results will not be as useful as those from a school that serves all socioeconomic levels. The decision is not to dismiss the findings but to limit them to the type of subjects in the sample. As more and more research accumulates with different convenience samples, the overall credibility of the results is enhanced.		Convenience sample: Nonprobability available sample.
92CHAPTER 4
Although it is not common for a researcher to state explicitly that a convenience sample was used, it will be obvious from the subjects subsection of the article. If some type of probability sampling procedure was used it will be described. Thus, in the absence of such particulars you can assume that the sample was an available one. The following examples are typical.
Examples: Convenience Samples "Participants in the study were sixth grade students enrolled in four classes at a public school in a suburb north of Minneapolis, Minnesota. Of the total number, 65 students were boys and 56 were girls. From the pool of 121 subjects, 7 were not included in the final analysis for various reasons, leaving 114 subjects." (Carrier and Williams, 1988, pp. 291-292) "Twelve volunteer third-grade teachers and their students participated in the study. The teachers were employed in 10 public schools located in three school districts in suburban areas of northern Califomia." (Mitman, 1985, p. 151) "The initial group of subjects in this study was composed of 42 undergraduate secondary education students majoring in a variety of disciplines. They were about to be placed in classrooms to student teach for their first semester. Thirty-five of these students also participated in the concluding part of the experiment at the end of the semester, following a 10-week student teaching experience." (Tiene and Buck, 1987, p. 262) "The study was conducted in a school system of approximately 2,800 elementary school students attending 6 schools. All kindergarten and first grade teachers using intraclassroom ability grouping were asked toparticipate. Of 22 teachers invited to participate, 20 agreed and were subsequently observed." (Haskins, Walden, and Ramey, 1983, pp. 867-868)
Purposive Sampling In purposive sampling (sometimes referred to as purposeful, judgment or judgmental sampling) the researcher selects particular elements from the population that will be representative or informative about the topic. Based on the researcher's knowledge of the population, ajudgment is made about which cases should be selected to provide the best information to address the purpose of the research. For example, in research on effective teaching it may be most informative to observe "expert" or "master" teachers rather than all teachers. To study effective schools it may be most informative to interview key personnel, such as the principal and teachers who have been in the school a number of years. The use of "selected precincts" for political polls is a type of purposive sampling.		Purposive sampling: Selection of particularly informative or useful subjects.
SUBJECTS AND SAMPLING93
Purposive sampling is not widely used in quantitative studies. In qualitative research, on the other hand, some type of purposive sampling is almost always used. Purposive sampling is illustrated by the following excerpts. Further discussion of sampling for qualitative studies is included in Chapter 10.
Examples: Purposive Sampling "Introductory psychology students (N 210) volunteered to take the Dogmatism Scale (Form E) for experimental credit. From the upper and lower quartiles on the Dogmatism Scale, 44 high and 44 low dogmatic subjects were selected for the experiment." (Rickards and Slife, 1987, pp. 636-637) Notice also that this is a convenience sample. "Four second-grade and two first-grade teachers from public schools in the San Francisco Bay Area participated in the study. All were women with at least 10 years of teaching experience at the elementary level. Teachers were recruited to include as wide a range of backgrounds and approaches in the teaching of mathematics as possible. Some were recommended by their principals as being strong mathematics teachers who had been involved in various inservice and curriculum development activities. Others agreed to participate in the study because they were interested but did not consider themselves to be particularly outstanding mathematics teachers." (Putnam, 1987, pp. 17-18) "Six schools were selected from the 26 in the district. Selection was governed by the need to capture the variability of retention practices within the district. For example, two schools with high-retaining and three with low-retaining kindergartens were selected, along with one school that had a developmental kindergarten and a transition (between kindergarten and first grade) class." (Smith and Shepard, 1988, p. 311)
Quota Sampling Quota sampling is used when the researcher is unable to take a probability sample but still wants a sample that is representative of the entire population. Different composite profiles of major groups in the population are identified, and then subjects are selected, nonrandomly, to represent each group. A type of quota sampling that is common in educational research is conducted to represent geographic areas or types of communities, such as urban, rural, and suburban. Typically, a state is divided into distinct geographic areas, and cases are selected to represent each area. As in availability and purposive sampling, there is a heavy reliance on the decisions of the researcher in selecting the sample, and appropriate caution should be used in interpreting the results.		Quota sampling: Nonrandom sampling representative of a target population.
94CHAPTER 4
HOW SUBJECTS AND SAMPLING AFFECT RESEARCH In reading and interpreting research you will need to be conscious of how the sampling procedures might have affected the results and how the characteristics of the subjects affect the usefulness and the generalizability of the results. Knowledge of Sampling Procedures To understand how sampling may affect research it is essential to know the characteristics of different sampling procedures. This knowledge will help you interpret the sample that is used. You should first be able to identify the sampling procedure and then evaluate its adequacy in addressing the research problem and in supporting the conclusions. It will be helpful to know the strengths and weaknesses of each sampling procedure, as summarized in Table 4. 1. Volunteer Samples A continuing problem in educational research, as well as in most social science research, is the use of volunteers as subjects. It is well documented that volunteers differ from nonvolunteers in important ways. Volunteers tend to be better educated, higher socioeconomically, more intelligent, more in need of social approval, more sociable, more unconventional, less authoritarian, and less conforming than nonvolunteers. Obviously, volunteer samples may respond differently than nonvolunteers because of these characteristics. One way volunteers are used is in survey research. The researcher typically sends questionnaires to a sample of individuals and tabulates the responses of those who return them. Often the percentage of the sample returning the questionnaire will be 50 to 60 percent or even lower. In this circumstance the sample is said to be biased in that the results may not be representative of the population. Thus, the nature of the results depends on the types of persons who respond, and generalizability to the target population is compromised. The specific effect that a biased sample has on the results depends on the nature of the study. For example, a study of the relationship between educational level and occupational success would be likely to show only a small relationship if only those who are most successful respond. Without some subjects who are not successful in the sample, success cannot be accurately related to the level of education. If a survey of teachers is conducted to ascertain their general knowledge and reading and writing skills, the results would probably be higher than the true case because of the tendency of volunteers to be better educated.
SUBJECTS AND SAMPLING95
Table 4.1 STRENGTHS AND WEAKNESSES OF SAMPLING METHODS Method of sampling Strengths Weaknesses Probability Simple random Usually representative of the population Easy to analyze and interpret results Easy to understand Requires numbering each element in the population Larger sampling error than in stratified sampling Systematic 1, 2, and 3 above Simplicity of drawing sample Periodicity in list of population elements Proportional stratified 1, 2, and 3 of simple random Allows subgroup comparisons Usually more representative than simple random or systematic Fewer subjects needed Results represent population without weighting Requires subgroup identification of each population element Requires knowledge of the proportion of each subgroup in the population May be costly and difficult to prepare lists of population elements in each subgroup. Disproportional stratified 1, 2, 3, and 4 of proportional stratified Assures adequate numbers of elements in each subgroup 1, 2, and 3 of proportional stratified Requires proper weighting of subgroup to represent population Less efficient for estimating population characteristics Cluster Low cost Requires lists of elements Efficient with large populations Less accurate than simple random, systematic, or stratified May be difficult to collect data from all elements in each cluster Requires that each population element be assigned to only one cluster (continued)
96CHAPTER 4
Table 4.1 (continued)
Method of sampling Strengths Weaknesses Nonprobability Convenience Less costly Less time-consuming Ease of administration Usually assures high participation rate Generalization possible to similar subjects Difficult to generalize to other subjects Less representative of an identified population Results dependent on unique characteristics of the sample Purposive 1, 2, 3, 4, and 5 of convenience Adds credibility to qualitative research Assures receipt of needed information 1, 2, and 3 of convenience Quota 1, 2, 3, 4, and 5 of convenience More representative of population than convenience or purposive 1, 2, and 3 of convenience Usually more time consuming than convenience or purposive
Volunteers are commonly used in research because the availability of subjects is often limited by time and resources. There have been thousands of studies with teachers who volunteer their classes for research. Much research on school-age children requires written permission from parents, and this necessity can result in a biased sample. Suppose a researcher needed parents' permission to study their involvement in the education of their children. Chances are good that parents who are relatively involved would be most likely to agree to be in the study, affecting a description of the nature of parental involvement for "all" students. Sample Size An important consideration in judging the credibility of research is the size of the sample. In most studies there are restrictions that limit the number of subjects, although it is difficult to know when the sample is
SUBJECTS AND SAMPLING97
too small. Most researchers use general rules of thumb in their studies, such as having at least 30 subjects for correlational research, and at least 15 subjects in each group in an experiment. In surveys that sample a population, often a very small percentage of the population must be sampled, for example, less than 5 or even I percent. Of course if the survey sample is too small, it is likely that the results obtained cannot characterize the population. Formal statistical techniques can be applied to determine the number of subjects needed, but in most educational studies these techniques are not used. In educational research a major consideration with sample size is concluding that a study with a relatively small sample that found no difference or no relationship is true. For example, suppose that you are studying the relationship between creativity and intelligence and, with a sample of 20 students, found that there was no relationship. Is it reasonable to conclude that in reality there is no relationship? Probably not, since a probable reason for not finding a relationship is because such a small sample was used. In addition to the small number of subjects, it is likely that there may not be many differences in either creativity or intelligence, and without such differences it is impossible to find that the two variables are related. That is, with a larger sample that has different creativity and intelligence scores, a relationship may exist. This problem, interpreting results that show no difference or relationship with small samples, is subtle but very important in educational research since so many studies have small samples. As we will see in Chapter 9, it is also possible to misinterpret what is reported as a "significant" difference or relationship with a very large sample. Also, a sample that is not properly drawn from the population is misleading, no matter what the size. Subject Motivation Sometimes subjects will be motivated to respond in certain ways. Clues for this phenomenon will be found in the description of how the subjects were selected. For example, if a researcher was interested in studying the effectiveness of computer simulations in teaching science, one approach to the problem would be to interview teachers who used computer simulations. The researcher might even want to select only those science teachers who had used the simulations more than two years. It is not hard to understand that the selected teachers, because they had been using the simulations, would be motivated to respond favorably toward them. The response would be consistent with the teachers' decision to use simulations. Psychology students may be motivated to give inaccurate responses in studies conducted by their psychology professor if
98CHAPTER 4
they do not like the professor, or they may respond more favorably if they want to help a professor they like. Sampling Bias
In selecting a sample from a population there is always some degree of sampling error. This error is the discrepancy between the true value of a variable for the population and the value that is calculated from the sample, and it is expected and precisely estimated as part of sampling. A different type of error is due to sampling bias, a type of sampling error that is controlled or influenced by the researcher to result in misleading findings. Occasionally researchers will deliberately skew the sampling. The most obvious deliberate bias is selecting only those subjects that will respond in a particular way to support a point or result. For instance, if a researcher is measuring the values of college students and wants to show that the students are concerned about helping others and being involved in community service, bias would result if the researcher deliberately selected students in education or social work and ignored majors that might not be so altruistically oriented. Selecting friends or colleagues may also result in a biased sample. An even more flagrant type of bias occurs when a researcher discards some subjects because they have not responded as planned or keeps adding subjects until the desired result is obtained. Sampling bias also occurs nondeliberately, often because of inadequate knowledge of what is required to obtain an unbiased sample and the motivation to "prove" a desired result or point of view. In qualitative studies the researcher needs to be particularly careful about possible unintended bias if sampling changes during the study. Bias can also result from selecting subjects from different populations and assigning them to different groups for an experiment or comparison. Suppose a researcher used graduate sociology students to receive a treatment in an experiment and graduate psychology students as a control group. Even if the samples were selected randomly from each population, differences in the populations, and consequently samples, in attitudes, values, knowledge, and other variables could explain why certain results were obtained. CONSUMER TIPS: CRITERIA FOR EVALUATING SUBJECTS SECTIONS OF REPORTS AND SAMPLING PROCEDURES 1. The subjects in the study should be clearly described, and the description should be specific and detailed. Demographic characteris-		Sampling bias: Sampling error caused by the researcher.
SUBJECTS AND SAMPLING99
tics, such as age, gender, socioeconomic status, ability, and grade level, should be indicated, as well as any unique characteristics, for example, gifted students, students enrolled in a psychology class, or volunteers. 2. The population should be clearly defined. It is especially important to provide a specific definition of the population in studies using probability sampling. Vague descriptions, such as "retired workers" or "high-ability students," should be avoided. The characteristics of each stratum in a stratified sampling procedure should also be included. 3. The method of sampling should be clearly described. The specific type of sampling procedure, such as simple random, stratified, cluster, or convenience, should be explicitly indicated in sufficient detail to enable other researchers to replicate the study. 4. The return rate should be indicated and analyzed. In studies that survey a population, the return rate of questionnaires should be indicated. If the return rate is less than 60 percent, the researcher should analyze the implications of excluding a significant portion of the population. This step is accomplished by comparing the nonrespondents to those who returned the questionnaires to determine if there are significant differences between the groups. 5. The selection of subjects should be free of bias. The procedures and criteria for selecting subjects should not result in systematic error. Bias is more likely when a researcher is "proving" something to be true, with convenience samples, and when volunteers are used as subjects. 6. Selection procedures should be appropriate for the problem being investigated. If the problem is to investigate science attitudes of middle school students, it would be inappropriate to use high school students as subjects. If the problem is to study the characteristics of effective teaching, the work of student teachers would probably not be very representative of effective teaching behaviors. 7. There should be an adequate number of subjects. If the sample is selected from a population, the sample size must be large enough to represent the population accurately. There must also be a sufficient number of subjects in each subgroup that is analyzed. Studies with small samples that report no differences or no relationships should be viewed with caution since a higher number or a better selection of subjects may result in meaningful differences or relationships. Studies that have a very large number of subjects may report "significant" differences or relationships that are of little practical utility. 8. Qualitative studies should have informative and knowledgeable subjects. Since the purpose of qualitative research is to understand a phenomenon in depth, it is important to select subjects that will provide the richest information. The researcher should indicate the criteria used to select subjects, the reasons why these particular individuals were selected, and the strategies used for selecting subjects during the study.
100CHAPTER 4
OUTLINE SUMMARY Subject selection. Participants from whom data are gathered. Population. Group to whom results are generalized. Described by delimiting variables. Sample. Procedures for selecting subjects. Probability sampling. Subjects selected from a larger population. Always some error in sampling. Simple random sampling. Every member of the population has the same chance of being selected. Every member of the population must be numbered. Systematic random sampling. Subjects are selected without numbering each member of the population. Periodicity may cause bias in the result. Stratified random sampling. Divides population into groups before sample selection. Often provides a more accurate sample. Desirable for comparing subgroups. Proportional or disproportional selection. Cluster sampling. Naturally occurring groups of subjects are selected at random. Usually less accurate. Nonprobability sampling. Very common and over time results in generalizable conclusions. Convenience samples. Purposive samples. Quota sampling. Subjects and sampling procedures affect research in several ways. Volunteer subjects. Sample size. Subject motivation. Sampling bias. Criteria for evaluating subjects sections of reports and sampling. Clearly defined subjects, population, and sampling design. Adequate and/or analyzed return rate. Selection should be free of bias.
SUBJECTS AND SAMPLING101
Selection should be appropriate to the problem. Sample size should be adequate. Qualitative research should use the most knowledgeable and informative subjects. STUDY QUESTIONS What is a sample and a population? Why is it important to define the population as specifically as possible? What is the difference between probability and nonprobability sampling? When should a researcher use stratified random sampling? How is cluster sampling different from stratified sampling? Why should readers of research be cautious of studies that use a convenience sample? What are some strengths and weaknesses of various types of sampling? How can volunteer subjects cause bias in a study? Why is sample size an important consideration in research that fails to find a "significant" difference or relationship? In what ways can sampling be biased? Give an example of a study that used both stratified and systematic sampling. What is the difference between a convenience and a purposive sample? What criteria should be used in judging the adequacy of a subjects section in a report or sampling procedure? SAMPLE TEST QUESTIONS Answers are provided in Appendix A. The sampling frame is most similar to the population. sample. participants. elements. We use the results obtained from a sample to
102CHAPTER 4
generalize to the population. stratify the sample. select convenience samples. identify the subjects used in the study. It is important to have a complete description of the sample to be able to stratify the sample. describe the population. generalize the results. select informative subjects. Probability sampling is to systematic sampling as nonprobability sampling is to stratified sampling. proportional sampling. disproportional sampling. purposive sampling. Systematic sampling is preferred when stratified sampling is not possible. certain subjects need to be selected because of their position or special knowledge. it is not possible to number all members of the population. there is periodicity in a list of the population. A researcher decides to select a sample by taking simple random samples from three subgroups that have been identified from the population. What type of sampling was used? Proportional. Cluster. Convenience. Stratified. In qualitative research the sampling procedure is most likely to be purposive. cluster. quota. systematic. If your subjects have volunteered to participate in your study, what will you need to be careful about so that the research is credible? Sample size that is inadequate.
SUBJECTS AND SAMPLING103
Sampling bias. Whether the sampling was proportional or disproportional. Whether the sampling was systematic. When a study has a small number of subjects and finds no relationship among the variables studied, it can typically be concluded that there is no relationship among the variables. there is a relationship among the variables. it is not possible to conclude that there is no relationship among the variables. the study is not very credible. Each of the following about the subjects should be indicated in a research report EXCEPT return rate of surveys. method of sampling. a clear description of the subjects. names of the subjects.