California State University, Dominguez Hills
University of Wisconsin, Parkside
Latest update: May 22, 1999
Faculty on the Site.

The difficulty of obtaining cooperative and representative samples is what leads so often to studies focussing on teachers (who are presumed captive because they are expected to respond to summonses by administrators), students (who are captive when their classes or used - the famous "college sophomore" studies), and others who are in some way captive to our demands to answer questions or fill out questionnaires.

• Difficulty in Knowing Extent to Which Subject Alert to Experimental Factors

  With any "captive" population, the researcher must always be concerned with the quality and degree of concentration afforded his/her efforts. Rosenthal's Self Fulfilling Prophecy study relied on memos placed in teachers' mail boxes. The memos provided them with a list of students in their classes for whom Rosenthal's tests purportedly indicated an intellectual growth spurt that year. There was no measurement of how much attention the teacher gave to the notice, or even if they noticed and/or remembered it. Yet it was assumed on the basis of those memos that teachers had knowledge of potential student intellectual spurts. Many teachers complained rancorously over this, and rightfully so.

• Difficulty in Finding a Measure that Reliably Differentiates Everyday Contexts

  Similar problems crop up when existing classrooms are used as part of the captive school setting to measure the difference between democratic and authoritarian classroom environments. In one dissertation, democratic and authoritaria were measured by a teacher questionnaire on which the researcher relied to create the experimental and control settings. At the end of the school year, actual classroom visitations revealed that there was no measureable difference between the chosen classroom environments.

  One plausible explanation for the difficulty encountered in defining contextual differences is that most of us vary over time. Few of us are "always" democratic or "always" authoritarian, so that, in actual practice, the total classroom environments are unlikely to be very different unless one practice is "extreme," in which case "democratic" or "authoritarian" is probably not the most salient characteristic about that classroom.

• Importance of Repeatedly Checking Key Assumptions

  Each of these problems could have been avoided by more demanding research efforts that checked for feedback on the study's assumptions. But there is often little time and less money to afford such luxuries. Researchers need to be aware of their own key assumptions, and they need to check those carefully to be sure that the conditions they have assumed actually exist. There is always enough time and money to check the key assumptions. The problem has been our all or nothing philosophy in failing to check at all when we cannot afford to check every detail.

  Martha Minow addresses the problem presented by unstated assumptions in Making All the Difference: Inclusion and Exclusion in American Law. The teachers in the Rosenthal study were angry at conclusions drawn about them, but they failed to stop the printing and the popularity of Rosenthal's book. Had Rosenthal openly stated the assumption that teachers had taken cognizance of his memos on intellectual growth spurts, the teachers might have been better able to negotiate a more reasonable statement of the findings. When assumptions are unstated, most of us find the discourse confused and confusing. Those with power, such as a publisher with money invested in a book, tend to win out over those with plausible validity claims, but no forum or means to make them heard (i.e. no potentially profitable book in the works). (For more information on how this inability to gain a forum in which to oppose research publication, refer to Susan Faludi's Backlash, in which she describes the false statistics perpetrated upon women by prominent professors writing on women's chances of finding mates beyond their thirties. Refer also to Steven Epstein's Impure Science on the study of the AIDS virus.)

• How do you decide which are the key assumptions in your study?

  Key assumptions are assumptions about factors on which yes, no, maybe answers are made, and subjects or treatments classified into categories on the basis of those answers. One good way to imagine this is to ask yourself, would your conclusions have been altered if the assumption had not held true. If you had checked the classroom environment, and found that you could not easily classified it as democratic or authoritarian, would you have needed to change your experimental conditions? If yes, that's a key assumption. And if you're planning to conclude that democratic teaching fosters more creativity than authoritarian teaching, then whether your classrooms are clear examples of democratic and authoritarian contexts is key.

  Maybe you feel inclined to weasel just a little, and tell yourself that the democratic classroom is usually democratic, you just couldn't tell clearly on this occasion. Well, if you can't tell, could the children, could the teacher? Do you really have an experimental class that is measurably different from your control class? At the very least, you would need to check the classroom several more times to see how well your classification of classrooms was working. Better to start over near the beginning, than to discover, as the writer of one dissertation did, at the end of the year, that her results showed no appreciable result. She was honest. She admitted the invalid assumption of classroom differentiation, and lost a year of doctoral research. Maybe she should have written her thesis on the methodological difficulties in measuring many of our educational variables. She certainly did have the data for that.

One procedure for obtaining samples that do not involve the coercion of a captive group (and the consequent uncertainties that entails for measurement) is that of using "informed respondents." Informed respondents are those who have some relationship to the research problem and who are consequently more likely to be informed on the related issues. If you are studying crime, corrections officers, police, social workers, teachers, juvenile justice workers are all likely to have had experience with crime in the course of their professional work. One way to get information is to issue a "Call to the Field." this can be done through a Web site, through local agencies, through local newsletters, through national newsletters, etc. Participation is voluntary in most cases, but such alternatives provide you with a forum: a place to announce that you need information and help in gathering it. Most professionals are glad to help in such endeavors. Increasingly the Web provides such a forum.

As an example, look at the National Criminal Justice Association "Call to the Field". You'll need to scroll down almost to the bottom of the page to find the call for help in gathering information on Assisting Victims of Campus Crime. Not only does the Web site offer you a forum from which to request help in gathering data. Its forms provide the means to do so.