Educational
Research

Fundamentals for
the Consumer

SECOND EDITION

 

JAMES H. MCMILLAN
Virginia Commonwealth University

HarperCollins College Publishers


Executive Editor: Christopher Jennison
Project Coordination and Text Design: Ruttle, Shaw & Wetherill, Inc.
Cover Designer: Kay Petronio
Electronic Production Manager: Christine Pearson
Electronic Page Makeup: Ruttle, Shaw & Wetherill, Inc.
Manufacturing Manager: Helene G. Landers
Printer and Binder: R. R. Donnelley & Sons Company
Cover Printer: The Lehigh Press, Inc.

For permission to use copyrighted material, grateful acknowledgment is made
to the copyright holders on pp. 358-359, which are hereby made part of this
copyright page.

Educational Research: Fundamentals for the Consumer, Second Edition

Copyright C 1996 by James H. McMillan

HarperCollins ® and ® are registered trademarks of HarperCollins Publishers Inc.

All rights reserved. Printed in the United States of America. No part of
this book may be used or reproduced in any manner whatsoever without
written permission, except in the case of brief quotations embodied in
critical articles and reviews. For information address HarperCollins College
Publishers, 10 East 53rd Street, New York, NY 10022. For information about
any HarperCollins title, product, or resource, please visit our World Wide Web site at
http://www.harpercollins.com/college.

Library of Congress Cataloging-in-Publication Data

McMillan, James H.
Educational research: fundamentals for the consumer /James H.
McMillan. - 2nd ed. Ed.
p.cm.
Includes bibliographical references and index.
ISBN 0-673-99864-9
1. Education-Research,1. Title.
LB1028.M2815 1996
370'.78-dc20 95-16506 CIP 95 96 97 98 9 8 7 6 5 4 3 2 1



Contents

To the Instructor xv
To the Student xix

1. Introduction to Research in
Education1

 
 
  SOURCES OF KNOWLEDGE2
Personal Experience2Tradition3
Authority3The Scientific Approach4
 
  THE NATURE OF SCIENTIFIC INQUIRY4
The Purpose of Scientific Inquiry4
Characteristics of Scientific Inquiry5
The Purpose of Theories6
 
  APPLYING SCIENTIFIC INQUIRY TO
EDUCATION7

 
  TYPES OF EDUCATIONAL RESEARCH9
Two Traditions of Research: Quantitative and
Qualitative9Basic Research10
Applied Research10Action Research12
Evaluation Research12Nonexperimental
Research12Experimental Research13
 
  FORMAT TO REPORT EDUCATIONAL
RESEARCH14

Title and Author(s)16Abstract16
Introduction16Review of Literature16
Specific Research Question or Hypothesis17
Method and Design17Results17
Discussion17Conclusions18
References18
 

v


viCONTENTS


  ANATOMY OF A RESEARCH ARTICLE18

OUTLINE SUMMARY18

STUDY QUESTIONS27

SAMPLE TEST QUESTIONS28
 

 
2. Variables, Research Problems,
and Hypotheses31

 
 
  VARIABLES IN EDUCATIONAL RESEARCH32
Constitutive and Operational Definitions32
Types of Variables33
 
  RESEARCH PROBLEMS36
Sources for Research Problems39
 
  CONSUMER TIPS: CRITERIA FOR EVALUATING
RESEARCH PROBLEMS42

 
  HYPOTHESES46
Why Researchers Use Hypotheses46
Types of Hypotheses47
 
  CONSUMER TIPS: CRITERIA FOR EVALUATING
RESEARCH HYPOTHESES49

 
  OUTLINE SUMMARY51
 
  STUDY QUESTIONS52
 
  SAMPLE TEST QUESTIONS53
 
3. Locating and Reviewing
Related Literature55

 
 
  THE PURPOSE OF REVIEWING RELATED
LITERATURE56

Refining the Research Problem56
Developing Significance for the Research56
Identifying Methodological Techniques57
Identifying Contradictory Findings57
Developing Research Hypotheses57
Learning About New Information57
 


CONTENTSvii


  STEPS TO REVIEW RELATED LITERATURE58
Step One: Locate Existing Reviews and Other
Information in Secondary Sources58
Step Two: Identify Key Terms65
Step Three: Identify the Appropriate Journal
Indexes and Abstracts67
Step Four: Search Indexes for Primary
Sources68
Step Five: Summarize and Analyze Primary
Source Information74
Step Six: Organize the Review76
 
  CONSUMER TIPS: CRITERIA FOR EVALUATING
THE REVIEW OF LITERATURE77

 
  OUTLINE SUMMARY80
 
  STUDY QUESTIONS81
 
  SAMPLE TEST QUESTIONS82
 
4. Subjects and Sampling84
 
 
  INTRODUCTION TO SAMPLING85
What Is a Subject?85What Is a
Population?85
What Is a Sample?86
 
  TYPES OF SAMPLING PROCEDURES86
Probability Sampling86Nonprobability
Sampling91

 
  HOW SUBJECTS AND SAMPLING AFFECT
RESEARCH94

Knowledge of Sampling Procedures94
Volunteer Samples94Sample Size96
Subject Motivation97Sample Bias98
 
  CONSUMER TIPS: CRITERIA FOR EVALUATING
SUBJECTS SECTIONS OF REPORTS AND
SAMPLING PROCEDURES98

 
  OUTLINE SUMMARY100
 
  STUDY QUESTIONS101
 
  SAMPLE TEST QUESTIONS101
 


viiiCONTENTS


5. Foundations of Educational
Measurement104

 
 
  INTRODUCTION TO MEASUREMENT105
Definition of Measurement105
The Purpose of Measurement for Research106
Scales of Measurement106
 
  FUNDAMENTAL PRINCIPLES OF DESCRIPTIVE
STATISTICS FOR UNDERSTANDING
MEASUREMENT108

Frequency Distributions109Measures of
Central Tendency112Measures of
Variability113Correlation115
 
  VALIDITY OF EDUCATIONAL MEASURES118
Definition of Validity118Types of
Evidence for Judging Validity119Effect of
Validity on Research122
 
  RELIABILITY OF EDUCATIONAL
MEASURES123

Types of Reliability124Effect of
Reliability on Research127
 
  OUTLINE SUMMARY129
 
  STUDY QUESTIONS130
 
  SAMPLE TEST QUESTIONS131
 
6. Types of Educational
Measures134

 
 
  CLASSIFYING EDUCATIONAL MEASURES135
 
  TESTS136
Norm- and Criterion-Referenced Tests136
Standardized Tests137Interpreting Test
Scores141
 
  PERSONALITY ASSESSMENT143
 


CONTENTSix


  ATTITUDE, VALUE, AND INTEREST
INVENTORIES144

Types of Inventories145Problems in
Measuring Noncognitive Traits148
 
  OBSERVATIONS150
Inference150
Laboratory Observation151
Structured Field Observations152
Observer Effects153
 
  INTERVIEWS154
Types of Interview Questions155
Interviewer Effects155
 
  LOCATING AND EVALUATING EDUCATIONAL
MEASURES157

 
  CONSUMER TIPS: CRITERIA FOR EVALUATING
INSTRUMENTATION158

 
  OUTLINE SUMMARY162
 
  STUDY QUESTIONS164
 
  SAMPLE TEST QUESTIONS165
 
7. Descriptive, Correlational, and
Causal-Comparative Research
167

 
 
  THE PURPOSE OF NON-EXPERIMENTAL
RESEARCH168

 
  DESCRIPTIVE STUDIES168
Characteristics of Descriptive Studies168
 
  CONSUMER TIPS: CRITERIA FOR EVALUATING
DESCRIPTIVE STUDIES170

 
  RELATIONSHIP STUDIES171
Relationship Determined by Differences171
Simple Correlational Studies172
Prediction Studies176
 
  CONSUMER TIPS: CRITERIA FOR EVALUATING
CORRELATIONAL STUDIES178

 


xCONTENTS


  USING SURVEYS IN DESCRIPTIVE AND
RELATIONSHIP STUDIES182

Cross-Sectional Surveys182
Longitudinal Surveys183
 
  CAUSAL-COMPARATIVE STUDIES184
Ex Post Facto Research184Correlational
Causal-Comparative Research186
 
  CONSUMER TIPS: CRITERIA FOR EVALUATING
CAUSAL-COMPARATIVE RESEARCH186

 
  OUTLINE SUMMARY187
 
  STUDY QUESTIONS188
 
  SAMPLE TEST QUESTIONS189
 
8. Experimental and
Single-Subject Research192

 
 
  CHARACTERISTICS OF EXPERIMENTAL
RESEARCH193

 
  EXPERIMENTAL VALIDITY194
History195Selection196
Maturation196Pretesting197
Instrumentation197Treatment
Replications198Subject Attrition198
Statistical Regression198
Diffusion of Treatment199Experimenter
Effects199Subject Effects200
 
  TYPES OF EXPERIMENTAL DESIGNS201
Single-Group Posttest-Only Design201
Single-Group Pretest-Posttest Design202
Nonequivalent-Groups Posttest Only Design203
Nonequivalent-Groups Pretest-Posttest
Design204
Random ized-Grou ps Posttest-Only Design206
Randomized-Groups Pretest-Posttest
Design207
Factorial Experimental Designs209
 
  CONSUMER TIPS: CRITERIA FOR EVALUATING
EXPERIMENTAL RESEARCH210

 


CONTENTSxi


  SINGLE-SUBJECT RESEARCH212
Characteristics of Single-Subject Research212
Types of Single-Subject Designs213
 
  CONSUMER TIPS: CRITERIA FOR EVALUATING
SINGLE,SUBJECT RESEARCH215

 
  OUTLINE SUMMARY217
 
  STUDY QUESTIONS218
 
  SAMPLE TEST QUESTIONS219
 
9. Analyzing Statistical
Inferences221

 
 
  THE PURPOSE AND NATURE OF INFERENTIAL
STATISTICS221

Degree of Certainty222Estimating Errors
in Sampling and Measurement222The
Null Hypothesis223
 
  INTERPRETING RESULTS OF INFERENTIAL
TESTS225

The t-Test226Simple Analysis of
Variance227Factorial Analysis of
Variance228Analysis of Covariance229
Multivariate Statistics230Chi-Square230
 
  CONSUMER TIPS: CRITERIA FOR EVALUATING
INFERENTIAL STATISTICS232

 
  OUTLINE SUMMARY234
 
  STUDY QUESTIONS234
 
  SAMPLE TEST QUESTIONS235
 
10. Qualitative and Historical
Research238

 
 
  QUALITATIVE RESEARCH239
Characteristics of Qualitative Research239
Qualitative Research Problems241
 


xiiCONTENTS


  Entering the Research Site243Selecting
Participants243Obtaining Qualitative
Information244Analyzing Qualitative
Data248Credibility of Qualitative
Research250
 
  CONSUMER TIPS: CRITERIA FOR EVALUATING
QUALITATIVE RESEARCH253

 
  HISTORICAL RESEARCH254
The Historical Method255
 
  CONSUMER TIPS: CRITERIA FOR EVALUATING
HISTORICAL RESEARCH258

 
  OUTLINE SUMMARY259
 
  STUDY QUESTIONS261
 
  SAMPLE TEST QUESTIONS262
 
11. Analyzing Discussion and
Conclusions265

 
 
  PURPOSE AND NATURE OF THE
DISCUSSION265

 
  INTERPRETATION OF THE RESULTS266
Interpretation Related to the Problem
and/or Hypothesis266Interpretation
Related to Methodology266
Interpretation Based on Statistical
Procedures269Interpretation Related to
Previous Research270
 
  CONCLUSIONS272
Limitations273Recommendations and
Implications276
 
  CONSUMER TIPS: CRITERIA FOR EVALUATING
DISCUSSION SECTIONS277

 
  OUTLINE SUMMARY279
 
  STUDY QUESTIONS279
 
  SAMPLE TEST QUESTIONS280
 


CONTENTSxiii


12. The Intelligent Consumer:
Putting It All Together282

 
 
  QUESTIONS FOR QUANTITATIVE STUDIES283
 
  QUESTIONS FOR QUALITATIVE STUDIES286
 
  QUESTIONS FOR HISTORICAL STUDIES287
 
  EXAMPLES OF RESEARCH ARTICLES288
Article 1: A Study of Academic Time-on-Task in
the Elementary School288Evaluation of
Article 1298Article 2: Reducing Teacher
Stress304Evaluation of Article 2316
Article 3: Kindergarten Readiness and
Retention: A Qualitative Study of Teachers'
Beliefs and Practices322
Evaluation of Article 3349
 
Appendix: Answers to Sample
Test Questions
352
References353
Acknowledgments358
Index360



To the Student

It was not too long ago that I sat, somewhat nervously, in a university auditorium waiting for my first class in educational research. Perhaps you have had, or will have, a similar experience. I distinctly remember thinking, given what I had heard about "research," that I needed to learn only enough to pass the course and would not have to worry about it again! It was another hurdle that I was forced to jump to graduate. I was not bad in mathematics but my interest was in working with people, not numbers. It was incomprehensible that I would someday teach and write about educational research. But something happened to me as I grudgingly struggled through the course. What I discovered was that research is a way of thinking, a tool that I could use to improve the work I do with other people. My hope is that this book can instill a similar disposition in you, providing knowledge, skills, and attitudes to improve your life and the welfare of others. Although learning the content and skills needed to become an intelligent consumer of research is not easy, my experience in teaching hundreds of students is that you will improve yourself, professionally and otherwise, through your efforts. In the beginning, especially as you read research articles, not everything will make sense. But as your experience in being an informed consumer increases, so will your understanding. Good luck and best wishes, and please write to me if you have suggestions for improving the book.

James H. McMillan

xix


    CHAPTER

Subjects
and
Sampling
  4

The third major part of research reports is the methodology or methods section. As noted in Chapter 1, the first subsection of the methodology section usually describes the subjects from whom data are collected. The manner in which subjects are selected has important implications for identifying factors that affect subject performance and for
 
   

84



SUBJECTS AND SAMPLING85


 
generalizing the results. Hence it is necessary to understand who the subjects are and how they were selected.

INTRODUCTION TO SAMPLING
What Is a Subject?

 
A subject is an individual who participates in a research study or is someone from whom data are collected. In experiments, for example, each person who is given a treatment and whose behavior is measured is considered to be a subject. The term subject may also identify individuals whose behavior, past or present, is used as data, without their involvement in some type of treatment or intervention. For instance, a researcher might use last year's fourth-grade test scores as data, and each fourth-grader included is considered to be a subject, In qualitative research individuals are identified as participants rather than subjects.
 
Subject: Person from whom data
are collected.
What Is a Population?
A population is a group of elements or cases, whether individuals, objects, or events, that conform to specific criteria and to which we intend to generalize the results of the research. This group is also referred to as the target Population or universe. The specification of the population begins with the research problem and review of literature, through which a population is described conceptually or in broad terms, for example, seventh-grade students, beginning teachers, principals, special education teachers, and so forth. A more specific definition is then needed, based on demographic characteristics. These characteristics are sometimes referred to as delimiting variables. For example, in a study of first grade minority students, there are three delimiting characteristics: students, first grade (age), and minority. Further delimiting variables should be added to provide as precise a definition as possible. What about geographic region, socioeconomic status, gender, type of community, and types of schools? Are both public and private students included? How is "minority" defined? It is also important to distinguish the target population from a list of elements from which a group of subjects is selected, which is termed the survey population or sampling frame. In a study of beginning teachers, the target population may be beginning teachers across the United States, in all types of schools. The survey population may be a list of beginning teachers that was obtained from four states. Although the intent may be all beginning teachers, the results are limited, or delimited, to beginning teachers in the four states. Thus, generalization from subjects to populations should be based on the survey population.
 
 
Population: Persons to whom
results can be generalized.

86CHAPTER 4



What Is a Sample?
The sample is the group of elements, or a single element, from which data are obtained. Although the phrase "the sample included . . ." is used to indicate the characteristics of the people or events in the sample, the nature of the sampling procedure is usually described by one or more adjectives, such as random sampling or stratified random sampling. These types of sampling procedures are defined, with illustrations from actual studies, in the following section. It is important for the researcher to define as specifically as possible both the sampling procedure and the characteristics of the sample used in the study. Here is an example of a good description of the sample.
 
 
Sample: Group of subjects from
whom data are collected.
Example: Description of a Sample

The sample for this study consisted of nine seventh-grade mathematics teachers and their students. All teachers had volunteered for the study and each teacher received a $100 stipend. The teachers taught in four public schools in a medium-sized Western city, a low-middle to middle class community with a small proportion of minorities. According to a school district brochure, the district had approximately 32, 000 students in 42 elementary schools, 9 middle schools, and 9 high schools during the school year the study was conducted.
The class sizes ranged from 16 to 34. There were 5 female'and 4 male teachers. The teachers had an average of approximately 11 years, of teaching experience (range = 2-26). Of these 11 years, 8 were as math teachers (range 2 = 22) and 6 were as middle-school math teachers (range = 2-10). All had secondary certification, and 4 had Master's degrees in Administration. One of the teachers had a math major in college, 6 a math minor, and 2 had no special training in math." (Burns and Lash, 1986, p 395)

 
TYPES OF SAMPLING PROCEDURES
The purpose of sampling is to obtain a group of subjects who will be representative of the larger population or will provide specific information needed. The degree of representativeness is based on the sampling technique employed. I will first describe different sampling procedures and then consider the strengths and weaknesses of each in obtaining a representative sample.

Probability Sampling
In social science and educational research it is usually impractical and unnecessary to measure all the elements in the population of interest. Typically, a relatively small number of subjects or cases is selected from
 



SUBJECTS AND SAMPLING87


 
the larger population. The goal is to select a sample that will adequately represent the population, so that what is described in the sample will also be true of the population. The best procedure for selecting such a sample is to use probability sampling, a method of sampling in which the subjects are selected randomly in such a way that the researcher knows the probability of selecting each member of the population. Random selection implies that each member of the population as a whole or of subgroups of the population has an equal chance of being selected. As long as the number of cases selected is large enough, it is likely that a very small percentage of the population, represented by the sample, will provide an accurate description of the entire population.
It should be noted, however, that there is always some degree of error in sampling, and that error must be considered in interpreting the results of the sample. In probability sampling this calculation can be made very precisely with some statistical procedures. Consider a population of 1,000 third-graders, from which you will select randomly 5 percent, or 50, to estimate the attitudes of all the third-graders toward school. If the attitude score was 75 for the sample of 50 subjects, 75 can be used to estimate the value for the entire population of third-graders. However, if another sample of 50 students is selected, their score might be a little different, say 73. Which one is more correct? Since all 1,000 students have not been tested to obtain the result we do not know for sure, but the results can be used to estimate the error in sampling. This is basically the technique that political polls follow when it is reported that the vote is 45 percent 3. The plus or minus 3 is the estimate of error in sampling.
There are many types of probability sampling procedures. You will probably encounter four types in educational research: simple random, systematic, stratified, and cluster.
 
 
Probability sampling: Known
probability of selection from the
population.
Simple Random Sampling In simple random sampling every member of the population has an equal and independent chance of being selected for the sample. This method is often used with a small number in the population, for example, putting the names or numbers of all population members in a hat and drawing some out as the sample. If every member of the population can be assigned a different number, a table of random numbers can identify the population members that will make up the sample. This approach is not convenient if the population is large and not numbered. The most common way of selecting a random sample from a large population is by computer. There are computer programs that will assign numbers to each element in the population, generate the sample numbers randomly, and then print out the names of the people corresponding to the numbers. Simple random sampling:
Each member of the population
has the same probability of being
selected.

88CHAPTER 4



Simple random sampling is illustrated in the following study of mothers' strategies for influencing their children's schooling.
 
Example: Simple Random Sampling
"We interviewed a sample of 41 mothers of eighth graders from one middle school. These mothers were randomly selected from a list of 129 mothers provided by the principal of the school."(Baker and, Stevenson, 1986, p. 157)
 
Systematic Sampling In systematic sampling every nth element is selected from a list of all elements in the population, beginning with a randomly selected element. Thus, if there is a need to select 100 subjects from a population of 50,000, every nth element would correspond to every 500th subject. The first element is selected randomly. In this example that would be some number between I and 500. Suppose 240 were randomly selected as a starting point. The first subject chosen for the sample would be the 240th name on a list, the next subject would be the 740th, then the 1,240th, and so on until 100 subjects were selected. Systematic sampling is virtually the same as simple random sampling. It is certainly much more convenient.
There is a possible weakness in systematic sampling if the list of cases in the population is arranged in a systematic pattern. For instance, if a list of fourth-graders in a school division is arranged by classroom and students in the classrooms are listed from high to low ability, there is a cyclical pattern in the list (referred to as periodicity). If every nth subject that is selected corresponds to the pattern, the sample would represent only a certain level of ability and would not be representative of the population. Alphabetical lists do not usually create periodicity and are suitable for choosing subjects systematically.
 
 
Systematic sampling:Every
nth member of the population is
selected.
Stratified Sampling A modification of either simple random or systematic sampling is first to divide the population into homogeneous subgroups and then select subjects from each subgroup, using simple random or systematic procedures, rather than the population as a whole. This is termed stratified sampling. The strata are the subgroups. Stratified sampling is used primarily for two reasons. First, as long as the subgroups are identified by a variable related to the dependent variable in the research (e.g., socioeconomic status in a study of achievement) and results in more homogeneous groups, the sample will be more representative of the population than if taken from the population as a whole. This result reduces error and means that a smaller sample can be chosen.  

Stratified sampling: Subjects are
selected from strata or groups of
the population.



SUBJECTS AND SAMPLING89


 
Second, stratified sampling is used to ensure that an adequate number of subjects is selected from different subgroups. For example, if a researcher is studying beginning elementary school teachers and believes that there may be important differences between male and female teachers, using simple random or systematic sampling would probably not result in a sufficient number of male teachers to study the differences. It would be necessary in this situation first to stratify the population of teachers into male and female teachers and then to select subjects from each subgroup. The samples can be selected in one of two ways. A proportional stratified sample, or proportional allocation, is used when the number of subjects selected from each stratum is based on the percentage of subjects in the population that have the characteristic used to form the stratum. Thus, in the previous example, if 5 percent of the population of elementary teachers is male, 5 percent of the sample would also be male teachers. Proportional stratified sampling:
Reflects proportion of stratum in
population.
A second approach is to take the same number of subjects from each stratum, regardless of the percentage of subjects from each stratum in the population. This method is used often because it ensures that a sufficient number of subjects will be selected from each stratum. For instance, if only 10 percent of a population of 200 elementary teachers are male, a proportional sample of 40 would include only 4 male teachers. To study male teachers it would be better to include all 20 male teachers in the population for the sample and randomly select 20 female teachers. This sampling procedure is referred to as disproportional because the number of subjects in the sample from each subgroup is not proportional to the percentage of the subgroups in the population. Disproportional stratified sampling is not limited to taking the same number of subjects from each subgroup. When disproportional sampling is used the results of each stratum need to be weighted to estimate values for the population as a whole.
In the following example disproportional stratified sampling ensures that the same number of first and third graders are selected randomly.
 
Disproportional stratified
sampling:
Number of subjects in
each strata does not reflect
proportion in population.
Example: Disproportional Stratified Sampling
"From a pool of all children who returned a parental permission form (more than 80% return rate) 24 first graders (10 girls, 14 boys; mean 6 years 6 months), and 24 third graders (13 girls, 11 boys; mean age, 8 years, 8 months) were randomly selected." (Clements and Nastasi, 1988, p. 93)
 
Stratified random sampling is illustrated in Figure 4.1. In this example the population is divided first into three different age groups,

90CHAPTER 4




Figure 4.1 Example of stratified sampling with two strata.
 
then by gender. Once the groups are stratified by gender, random samples are selected from each of the six subgroups.

Cluster Sampling When it is impossible or impractical to sample individual elements from the population as a whole, usually when there is no exhaustive list of all the elements, cluster sampling is used. Cluster sampling involves the random selection of naturally occurring groups or areas and then the selection of individual elements from the chosen groups or areas. Examples of naturally occurring groups would be universities, schools, school divisions, classrooms, city blocks, and households. For example, if there is a need to survey a state for the television viewing habits of middle school students, it would be cumbersome and difficult to select children at random from the state population of all middle-schoolers. A clustering procedure could be employed by first listing all the school divisions in the state and then randomly selecting 30 school divisions from the list. One middle school could then be selected from each division, and students selected randomly from each school. This is a multistage clustering procedure. Although cluster sampling saves time and money, the results are less accurate than other random sampling techniques.

Cluster sampling: Naturally
occurring groups are selected.


SUBJECTS AND SAMPLING91


 
Nonprobability Sampling
In many research designs it is either unfeasible or unnecessary to obtain a probability sample. In these situations a nonprobability sample is used. A nonprobability sample is one in which the probability of including population elements is unknown. Usually, not every element in the population has a chance of being selected. It is also quite common for the population to be the same as the sample, in which case there is no immediate need to generalize to a larger population. In fact you will find that much of the educational research reported in journals, especially experimental studies, uses a group of subjects that has not been selected from a larger population.
 
 

Nonprobability sample: Probability
of selection not known.

Convenience Sampling A convenience sample is a group of subjects selected because of availability, for example, a university class of a professor conducting some research on college students, classrooms of teachers enrolled in a graduate class, schools of principals in a workshop, people who decide to go to the mall on Saturday, or people who respond to an advertisement for subjects. There is no precise way of generalizing from a convenience sample to a population. Also, the nature of the convenience sample may bias the results. For example, if the available sample for studying the impact of college is the group of alumni who return on alumni day, their responses would probably be quite different from those of all alumni. Similarly, research on effective teaching that depends on teachers in a particular geographic area, because they are available, may result in different findings than research done in other geographic areas.
Although we need to be very wary of convenience samples, often this is the only type of sampling possible, and the primary purpose of the research may not be to generalize but to better understand relationships that may exist. Suppose a researcher is investigating the relationship between creativity and intelligence, and the only available sample is a single elementary school. The study is completed, and the results indicate a moderate relationship: Children who have higher intelligence tend to be more creative than children with lower intelligence. Because there was no probability sampling, should we ignore the findings or suggest that the results are not valid or credible? That decision seems overly harsh. It is more reasonable to interpret the results as valid for children similar to those studied. For example, if the school serves a low socio-economic area, the results will not be as useful as those from a school that serves all socioeconomic levels. The decision is not to dismiss the findings but to limit them to the type of subjects in the sample. As more and more research accumulates with different convenience samples, the overall credibility of the results is enhanced.
Convenience sample: Nonprobability
available sample.

92CHAPTER 4



Although it is not common for a researcher to state explicitly that a convenience sample was used, it will be obvious from the subjects subsection of the article. If some type of probability sampling procedure was used it will be described. Thus, in the absence of such particulars you can assume that the sample was an available one. The following examples are typical.
 
Examples: Convenience Samples
"Participants in the study were sixth grade students enrolled in four classes at a public school in a suburb north of Minneapolis, Minnesota. Of the total number, 65 students were boys and 56 were girls. From the pool of 121 subjects, 7 were not included in the final analysis for various reasons, leaving 114 subjects." (Carrier and Williams, 1988, pp. 291-292)
"Twelve volunteer third-grade teachers and their students participated in the study. The teachers were employed in 10 public schools located in three school districts in suburban areas of northern Califomia." (Mitman, 1985, p. 151)
"The initial group of subjects in this study was composed of 42 undergraduate secondary education students majoring in a variety of disciplines. They were about to be placed in classrooms to student teach for their first semester. Thirty-five of these students also participated in the concluding part of the experiment at the end of the semester, following a 10-week student teaching experience." (Tiene and Buck, 1987, p. 262)
"The study was conducted in a school system of approximately 2,800 elementary school students attending 6 schools. All kindergarten and first grade teachers using intraclassroom ability grouping were asked toparticipate. Of 22 teachers invited to participate, 20 agreed and were subsequently observed." (Haskins, Walden, and Ramey, 1983, pp. 867-868)
 
Purposive Sampling In purposive sampling (sometimes referred to as purposeful, judgment or judgmental sampling) the researcher selects particular elements from the population that will be representative or informative about the topic. Based on the researcher's knowledge of the population, ajudgment is made about which cases should be selected to provide the best information to address the purpose of the research. For example, in research on effective teaching it may be most informative to observe "expert" or "master" teachers rather than all teachers. To study effective schools it may be most informative to interview key personnel, such as the principal and teachers who have been in the school a number of years. The use of "selected precincts" for political polls is a type of purposive sampling.
 
Purposive sampling: Selection of
particularly informative or useful
subjects.


SUBJECTS AND SAMPLING93


 
Purposive sampling is not widely used in quantitative studies. In qualitative research, on the other hand, some type of purposive sampling is almost always used. Purposive sampling is illustrated by the following excerpts. Further discussion of sampling for qualitative studies is included in Chapter 10.
 
Examples: Purposive Sampling
"Introductory psychology students (N 210) volunteered to take the Dogmatism Scale (Form E) for experimental credit. From the upper and lower quartiles on the Dogmatism Scale, 44 high and 44 low dogmatic subjects were selected for the experiment." (Rickards and Slife, 1987, pp. 636-637) Notice also that this is a convenience sample.
"Four second-grade and two first-grade teachers from public schools in the San Francisco Bay Area participated in the study. All were women with at least 10 years of teaching experience at the elementary level. Teachers were recruited to include as wide a range of backgrounds and approaches in the teaching of mathematics as possible. Some were recommended by their principals as being strong mathematics teachers who had been involved in various inservice and curriculum development activities. Others agreed to participate in the study because they were interested but did not consider themselves to be particularly outstanding mathematics teachers." (Putnam, 1987, pp. 17-18)
"Six schools were selected from the 26 in the district. Selection was governed by the need to capture the variability of retention practices within the district. For example, two schools with high-retaining and three with low-retaining kindergartens were selected, along with one school that had a developmental kindergarten and a transition (between kindergarten and first grade) class." (Smith and Shepard, 1988, p. 311)
 
Quota Sampling Quota sampling is used when the researcher is unable to take a probability sample but still wants a sample that is representative of the entire population. Different composite profiles of major groups in the population are identified, and then subjects are selected, nonrandomly, to represent each group. A type of quota sampling that is common in educational research is conducted to represent geographic areas or types of communities, such as urban, rural, and suburban. Typically, a state is divided into distinct geographic areas, and cases are selected to represent each area. As in availability and purposive sampling, there is a heavy reliance on the decisions of the researcher in selecting the sample, and appropriate caution should be used in interpreting the results.
 
Quota sampling: Nonrandom
sampling representative of a
target population.

94CHAPTER 4



HOW SUBJECTS AND SAMPLING AFFECT RESEARCH
In reading and interpreting research you will need to be conscious of how the sampling procedures might have affected the results and how the characteristics of the subjects affect the usefulness and the generalizability of the results.

Knowledge of Sampling Procedures
To understand how sampling may affect research it is essential to know the characteristics of different sampling procedures. This knowledge will help you interpret the sample that is used. You should first be able to identify the sampling procedure and then evaluate its adequacy in addressing the research problem and in supporting the conclusions. It will be helpful to know the strengths and weaknesses of each sampling procedure, as summarized in Table 4. 1.

Volunteer Samples
A continuing problem in educational research, as well as in most social science research, is the use of volunteers as subjects. It is well documented that volunteers differ from nonvolunteers in important ways. Volunteers tend to be better educated, higher socioeconomically, more intelligent, more in need of social approval, more sociable, more unconventional, less authoritarian, and less conforming than nonvolunteers. Obviously, volunteer samples may respond differently than nonvolunteers because of these characteristics.
One way volunteers are used is in survey research. The researcher typically sends questionnaires to a sample of individuals and tabulates the responses of those who return them. Often the percentage of the sample returning the questionnaire will be 50 to 60 percent or even lower. In this circumstance the sample is said to be biased in that the results may not be representative of the population. Thus, the nature of the results depends on the types of persons who respond, and generalizability to the target population is compromised. The specific effect that a biased sample has on the results depends on the nature of the study. For example, a study of the relationship between educational level and occupational success would be likely to show only a small relationship if only those who are most successful respond. Without some subjects who are not successful in the sample, success cannot be accurately related to the level of education. If a survey of teachers is conducted to ascertain their general knowledge and reading and writing skills, the results would probably be higher than the true case because of the tendency of volunteers to be better educated.



SUBJECTS AND SAMPLING95


 
Table 4.1 STRENGTHS AND WEAKNESSES OF SAMPLING METHODS

Method
of sampling
Strengths Weaknesses

Probability
Simple
random
  1. Usually representative of the population
  2. Easy to analyze and interpret results
  3. Easy to understand
  1. Requires numbering each element in the population
  2. Larger sampling error than in stratified sampling
Systematic
  1. 1, 2, and 3 above
  2. Simplicity of drawing sample
  1. Periodicity in list of population elements
Proportional
stratified
  1. 1, 2, and 3 of simple random
  2. Allows subgroup comparisons
  3. Usually more representative than simple random or systematic
  4. Fewer subjects needed
  5. Results represent population without weighting
  1. Requires subgroup identification of each population element
  2. Requires knowledge of the proportion of each subgroup in the population
  3. May be costly and difficult to prepare lists of population elements in each subgroup.
Disproportional
stratified
  1. 1, 2, 3, and 4 of proportional stratified
  2. Assures adequate numbers of elements in each subgroup
  1. 1, 2, and 3 of proportional stratified
  2. Requires proper weighting of subgroup to represent population
  3. Less efficient for estimating population characteristics
Cluster
  1. Low cost
  2. Requires lists of elements
  3. Efficient with large populations
  1. Less accurate than simple random, systematic, or stratified
  2. May be difficult to collect data from all elements in each cluster
  3. Requires that each population element be assigned to only one cluster

(continued)


96CHAPTER 4



Table 4.1 (continued)

Method
of sampling
Strengths Weaknesses

Nonprobability
Convenience
  1. Less costly
  2. Less time-consuming
  3. Ease of administration
  4. Usually assures high participation rate
  5. Generalization possible to similar subjects
  1. Difficult to generalize to other subjects
  2. Less representative of an identified population
  3. Results dependent on unique characteristics of the sample
Purposive
  1. 1, 2, 3, 4, and 5 of convenience
  2. Adds credibility to qualitative research
  3. Assures receipt of needed information
  1. 1, 2, and 3 of convenience
Quota
  1. 1, 2, 3, 4, and 5 of convenience
  2. More representative of population than convenience or purposive
  1. 1, 2, and 3 of convenience
  2. Usually more time consuming than convenience or purposive

Volunteers are commonly used in research because the availability of subjects is often limited by time and resources. There have been thousands of studies with teachers who volunteer their classes for research. Much research on school-age children requires written permission from parents, and this necessity can result in a biased sample. Suppose a researcher needed parents' permission to study their involvement in the education of their children. Chances are good that parents who are relatively involved would be most likely to agree to be in the study, affecting a description of the nature of parental involvement for "all" students.

Sample Size
An important consideration in judging the credibility of research is the size of the sample. In most studies there are restrictions that limit the number of subjects, although it is difficult to know when the sample is



SUBJECTS AND SAMPLING97


 
too small. Most researchers use general rules of thumb in their studies, such as having at least 30 subjects for correlational research, and at least 15 subjects in each group in an experiment. In surveys that sample a population, often a very small percentage of the population must be sampled, for example, less than 5 or even I percent. Of course if the survey sample is too small, it is likely that the results obtained cannot characterize the population. Formal statistical techniques can be applied to determine the number of subjects needed, but in most educational studies these techniques are not used.
In educational research a major consideration with sample size is concluding that a study with a relatively small sample that found no difference or no relationship is true. For example, suppose that you are studying the relationship between creativity and intelligence and, with a sample of 20 students, found that there was no relationship. Is it reasonable to conclude that in reality there is no relationship? Probably not, since a probable reason for not finding a relationship is because such a small sample was used. In addition to the small number of subjects, it is likely that there may not be many differences in either creativity or intelligence, and without such differences it is impossible to find that the two variables are related. That is, with a larger sample that has different creativity and intelligence scores, a relationship may exist. This problem, interpreting results that show no difference or relationship with small samples, is subtle but very important in educational research since so many studies have small samples. As we will see in Chapter 9, it is also possible to misinterpret what is reported as a "significant" difference or relationship with a very large sample. Also, a sample that is not properly drawn from the population is misleading, no matter what the size.

Subject Motivation
Sometimes subjects will be motivated to respond in certain ways. Clues for this phenomenon will be found in the description of how the subjects were selected. For example, if a researcher was interested in studying the effectiveness of computer simulations in teaching science, one approach to the problem would be to interview teachers who used computer simulations. The researcher might even want to select only those science teachers who had used the simulations more than two years. It is not hard to understand that the selected teachers, because they had been using the simulations, would be motivated to respond favorably toward them. The response would be consistent with the teachers' decision to use simulations. Psychology students may be motivated to give inaccurate responses in studies conducted by their psychology professor if


98CHAPTER 4



they do not like the professor, or they may respond more favorably if they want to help a professor they like.

Sampling Bias

In selecting a sample from a population there is always some degree of sampling error. This error is the discrepancy between the true value of a variable for the population and the value that is calculated from the sample, and it is expected and precisely estimated as part of sampling. A different type of error is due to sampling bias, a type of sampling error that is controlled or influenced by the researcher to result in misleading findings. Occasionally researchers will deliberately skew the sampling. The most obvious deliberate bias is selecting only those subjects that will respond in a particular way to support a point or result. For instance, if a researcher is measuring the values of college students and wants to show that the students are concerned about helping others and being involved in community service, bias would result if the researcher deliberately selected students in education or social work and ignored majors that might not be so altruistically oriented. Selecting friends or colleagues may also result in a biased sample. An even more flagrant type of bias occurs when a researcher discards some subjects because they have not responded as planned or keeps adding subjects until the desired result is obtained. Sampling bias also occurs nondeliberately, often because of inadequate knowledge of what is required to obtain an unbiased sample and the motivation to "prove" a desired result or point of view. In qualitative studies the researcher needs to be particularly careful about possible unintended bias if sampling changes during the study.
Bias can also result from selecting subjects from different populations and assigning them to different groups for an experiment or comparison. Suppose a researcher used graduate sociology students to receive a treatment in an experiment and graduate psychology students as a control group. Even if the samples were selected randomly from each population, differences in the populations, and consequently samples, in attitudes, values, knowledge, and other variables could explain why certain results were obtained.

CONSUMER TIPS: CRITERIA FOR EVALUATING SUBJECTS SECTIONS OF REPORTS AND SAMPLING PROCEDURES

1. The subjects in the study should be clearly described, and the description should be specific and detailed. Demographic characteris-

 

Sampling bias: Sampling error
caused by the researcher.



SUBJECTS AND SAMPLING99


 
tics, such as age, gender, socioeconomic status, ability, and grade level, should be indicated, as well as any unique characteristics, for example, gifted students, students enrolled in a psychology class, or volunteers.
2. The population should be clearly defined. It is especially important to provide a specific definition of the population in studies using probability sampling. Vague descriptions, such as "retired workers" or "high-ability students," should be avoided. The characteristics of each stratum in a stratified sampling procedure should also be included.
3. The method of sampling should be clearly described. The specific type of sampling procedure, such as simple random, stratified, cluster, or convenience, should be explicitly indicated in sufficient detail to enable other researchers to replicate the study.
4. The return rate should be indicated and analyzed. In studies that survey a population, the return rate of questionnaires should be indicated. If the return rate is less than 60 percent, the researcher should analyze the implications of excluding a significant portion of the population. This step is accomplished by comparing the nonrespondents to those who returned the questionnaires to determine if there are significant differences between the groups.
5. The selection of subjects should be free of bias. The procedures and criteria for selecting subjects should not result in systematic error. Bias is more likely when a researcher is "proving" something to be true, with convenience samples, and when volunteers are used as subjects.
6. Selection procedures should be appropriate for the problem being investigated. If the problem is to investigate science attitudes of middle school students, it would be inappropriate to use high school students as subjects. If the problem is to study the characteristics of effective teaching, the work of student teachers would probably not be very representative of effective teaching behaviors.
7. There should be an adequate number of subjects. If the sample is selected from a population, the sample size must be large enough to represent the population accurately. There must also be a sufficient number of subjects in each subgroup that is analyzed. Studies with small samples that report no differences or no relationships should be viewed with caution since a higher number or a better selection of subjects may result in meaningful differences or relationships. Studies that have a very large number of subjects may report "significant" differences or relationships that are of little practical utility.
8. Qualitative studies should have informative and knowledgeable subjects. Since the purpose of qualitative research is to understand a phenomenon in depth, it is important to select subjects that will provide the richest information. The researcher should indicate the criteria used to select subjects, the reasons why these particular individuals were selected, and the strategies used for selecting subjects during the study.

100CHAPTER 4



OUTLINE SUMMARY

  1. Subject selection.
    1. Participants from whom data are gathered.
    2. Population.
      1. Group to whom results are generalized.
      2. Described by delimiting variables.
    3. Sample.
  2. Procedures for selecting subjects.
    1. Probability sampling.
      1. Subjects selected from a larger population.
      2. Always some error in sampling.
      3. Simple random sampling.
        1. Every member of the population has the same chance of being selected.
        2. Every member of the population must be numbered.
      4. Systematic random sampling.
        1. Subjects are selected without numbering each member of the population.
        2. Periodicity may cause bias in the result.
      5. Stratified random sampling.
        1. Divides population into groups before sample selection.
        2. Often provides a more accurate sample.
        3. Desirable for comparing subgroups.
        4. Proportional or disproportional selection.
      6. Cluster sampling.
        1. Naturally occurring groups of subjects are selected at random.
        2. Usually less accurate.
      7. Nonprobability sampling.
        1. Very common and over time results in generalizable conclusions.
        2. Convenience samples.
        3. Purposive samples.
        4. Quota sampling.
  3. Subjects and sampling procedures affect research in several ways.
    1. Volunteer subjects.
    2. Sample size.
    3. Subject motivation.
    4. Sampling bias.
  4. Criteria for evaluating subjects sections of reports and sampling.
    1. Clearly defined subjects, population, and sampling design.
    2. Adequate and/or analyzed return rate.
    3. Selection should be free of bias.


SUBJECTS AND SAMPLING101


 
    1. Selection should be appropriate to the problem.
    2. Sample size should be adequate.
    3. Qualitative research should use the most knowledgeable and informative subjects.

STUDY QUESTIONS

  1. What is a sample and a population?
  2. Why is it important to define the population as specifically as possible?
  3. What is the difference between probability and nonprobability sampling?
  4. When should a researcher use stratified random sampling?
  5. How is cluster sampling different from stratified sampling?
  6. Why should readers of research be cautious of studies that use a convenience sample?
  7. What are some strengths and weaknesses of various types of sampling?
  8. How can volunteer subjects cause bias in a study?
  9. Why is sample size an important consideration in research that fails to find a "significant" difference or relationship?
  10. In what ways can sampling be biased?
  11. Give an example of a study that used both stratified and systematic sampling.
  12. What is the difference between a convenience and a purposive sample?
  13. What criteria should be used in judging the adequacy of a subjects section in a report or sampling procedure?

SAMPLE TEST QUESTIONS
Answers are provided in Appendix A.

  1. The sampling frame is most similar to the
    1. population.
    2. sample.
    3. participants.
    4. elements.
  2. We use the results obtained from a sample to

102CHAPTER 4



    1. generalize to the population.
    2. stratify the sample.
    3. select convenience samples.
    4. identify the subjects used in the study.
  1. It is important to have a complete description of the sample to be able to
    1. stratify the sample.
    2. describe the population.
    3. generalize the results.
    4. select informative subjects.
  2. Probability sampling is to systematic sampling as nonprobability sampling is to
    1. stratified sampling.
    2. proportional sampling.
    3. disproportional sampling.
    4. purposive sampling.
  3. Systematic sampling is preferred when
    1. stratified sampling is not possible.
    2. certain subjects need to be selected because of their position or special knowledge.
    3. it is not possible to number all members of the population.
    4. there is periodicity in a list of the population.
  4. A researcher decides to select a sample by taking simple random samples from three subgroups that have been identified from the population. What type of sampling was used?
    1. Proportional.
    2. Cluster.
    3. Convenience.
    4. Stratified.
  5. In qualitative research the sampling procedure is most likely to be
    1. purposive.
    2. cluster.
    3. quota.
    4. systematic.
  6. If your subjects have volunteered to participate in your study, what will you need to be careful about so that the research is credible?
    1. Sample size that is inadequate.


SUBJECTS AND SAMPLING103


 
    1. Sampling bias.
    2. Whether the sampling was proportional or disproportional.
    3. Whether the sampling was systematic.
  1. When a study has a small number of subjects and finds no relationship among the variables studied, it can typically be concluded that
    1. there is no relationship among the variables.
    2. there is a relationship among the variables.
    3. it is not possible to conclude that there is no relationship among the variables.
    4. the study is not very credible.
  2. Each of the following about the subjects should be indicated in a research report EXCEPT
    1. return rate of surveys.
    2. method of sampling.
    3. a clear description of the subjects.
    4. names of the subjects.