This document contains Chapters 5, 7 and 8 CHAPTER 5 MAKING SYSTEMATIC OBSERVATIONS QUESTIONS TO PONDER 1. What is the reliability of a measure? 2. How does the concept of reliability apply to different types of measures? 3. What is meant by the accuracy of a measure? 4. How do the reliability and accuracy of a measure affect the generality of the results of a study? 5. What is the validity of a measure? 6. What are the ways you can assess the validity of a measure? 7. What is the relationship between the reliability and validity of a measure? 8. What are the defining characteristics of Stevens’s four scales of measurement? Do all measures fall neatly into one of the four categories? 9. What factors affect your choice of a scale of measurement? 10. What is ecological validity, and why should you be concerned about it? 11. What is meant by the adequacy of a dependent measure? 12. What is meant by the sensitivity of a dependent measure, and why should you be concerned about it? 13. What are range effects, and why should you be concerned about them? 14. When should you consider tailoring your dependent measures to the needs of your research participants? 15. How can you tailor your dependent measures? (Give examples.) 16. What are defining characteristics of the four types of dependent variables? 17. What are the advantages and disadvantages of each? 18. What is Q-sort methodology, and when is it used? 19. What do implicit measures reveal? 20. How can the act of measurement affect your subjects’ responses? 21. What are role attitude cues, and how might they affect the results of your study? 22. What are demand characteristics, and how can they affect the results of your study? 23. What is experimenter bias, and how can it affect the results of your study? 24. What measures can be taken to deal with reactivity in research? 25. What is a pilot study, and why should you conduct one? 26. What are manipulation checks, and why should you include them in your study? CHAPTER OUTLINE Deciding What to Observe Choosing Specific Variables for Your Study Research Tradition Theory Availability of New Techniques Availability of Equipment Choosing Your Measures Reliability of a Measure Accuracy of a Measure Validity of a Measure Acceptance as an Established Measure Scale of Measurement Variables and Scales of Measurement Choosing a Scale of Measurement Adequacy of a Dependent Measure Tailoring Your Measures to Your Research Participants Types of Dependent Variables and How to Use Them Choosing When to Observe The Reactive Nature of Psychological Measurement Reactivity in Research with Human Participants Demand Characteristics Other Influences The Role of the Experimenter Reactivity in Research with Animal Subjects Automating Your Experiments Detecting and Correcting Problems Conducting a Pilot Study Adding Manipulation Checks Summary Key Terms
reliability test–retest reliability parallel-forms reliability split-half reliability accuracy validity face validity content validity criterion-related validity concurrent validity predictive validity construct validity nominal scale ordinal scale interval scale ratio scale range effects behavioral measure physiological measure self-report measure Q-sort methodology Implicit Association Test (IAT) demand characteristics role attitude cues experimenter bias expectancy effects single-blind technique double-blind technique pilot study manipulation check
CHAPTER GOALS Chapter 5 introduces students to the general principles of systematic observation; specific techniques are discussed later in the text within the research design chapters (7 through 11). After reading this chapter, students should know how to select variables to include in a study; the characteristics of a measure (reliability, validity, accuracy, scale of measurement); how to evaluate the adequacy of a dependent measure; types of dependent measures; the problems created by reactivity, demand characteristics, and experimenter bias; and techniques to evaluate and deal with those problems. Your classroom lectures and discussions might focus on the following: 1. Creating operational definitions for variables. Although operational definitions are explained in Chapter 2, students can always use more practice operationalizing variables and should be reminded that an operational definition is a two-edged sword, making precise observation possible but at the expense of restricting the generality of findings. 2. How research tradition, theory, and invention of new methods, techniques, instruments, and equipment influence the choice of specific variables to include in a study. 3. How the reliability of a measure is defined and established, and how reliability relates to different types of measures (physical, population estimates, ratings by multiple observers, and psychological tests). 4. What the accuracy of a measure refers to, and why it is important to consider accuracy. 5. How the validity of a measure is established, and the kinds of validity different methods produce. 6. The relationships among reliability, accuracy, and validity. 7. What scales of measurement exist, and how the scale of measurement determines the information that can be obtained from a measure, and the statistics that can be legitimately applied to the data. 8. What factors should be evaluated when choosing a scale of measurement. 9. How the sensitivity of a dependent measure, range effects, and the nature of the subjects affect the usefulness of a dependent measure. 10. How and when to tailor your measures to your participants. 11. What broad types of dependent variables exist (behavioral, physiological, and self-report measures). 12. The reactive nature of psychological measurement including the effects of demand characteristics, subject attitudes, and experimenter bias, and techniques to deal with these problems. 13. How and why to automate experiments. 14. What a pilot study is, and why it is essential to conduct one before conducting the main experiment. 15. What manipulation checks are, and why you should include them in your research. IDEAS FOR CLASS ACTIVITIES Psychological Ruler This demonstration illustrates how psychological scales are constructed, while providing insight into the problems of validity, reliability, and accuracy of a measure. Provide students with the simple response sheet given immediately below. Psychological Ruler Data Sheet Observer_________________________ Date_____________ Page_____ of _____ Participant Height _____ |----------------------------------------|----------------------------------------| Very short Average Very tall _____ |----------------------------------------|----------------------------------------| Very short Average Very tall _____ |----------------------------------------|----------------------------------------| Very short Average Very tall _____ |----------------------------------------|----------------------------------------| Very short Average Very tall _____ |----------------------------------------|----------------------------------------| Very short Average Very tall _____ |----------------------------------------|----------------------------------------| Very short Average Very tall _____ |----------------------------------------|----------------------------------------| Very short Average Very tall _____ |----------------------------------------|----------------------------------------| Very short Average Very tall _____ |----------------------------------------|----------------------------------------| Very short Average Very tall _____ |----------------------------------------|----------------------------------------| Very short Average Very tall _____ |----------------------------------------|----------------------------------------| Very short Average Very tall _____ |----------------------------------------|----------------------------------------| Very short Average Very tall There are enough spaces on each response sheet for 10 observations. Each student should receive enough sheets to record 1 observation for each student in the class (for example, a class of 30 would require 3 sheets per student). The spaces under the label “Participant” are for identifying labels (numbers work well), one for each student observed. You could have your class simply number the students consecutively as they are “measured.” Data Collection Ask students to come, one at a time, to the front of the class. The rest of the students then rate the participant’s height by marking the appropriate point along the scale for that observation. (Be sure each participant rates himself or herself at this time as well!) When everyone in the class has been observed and rated, have the students actually measure each other’s heights using a ruler (measuring to the nearest centimeter using a meter stick works well and avoids fractional measurements). Data Analysis Have students convert their ratings to numbers by measuring in millimeters from the leftmost tick mark (very short) to the observation mark. Then have them collect all ratings for each subject and place them on a data coding sheet. They should create one column for each subject. Have them then average the scores in each column. Below this average, have them place the subject’s actual height as measured. At this point, your students will have two scores for each subject: the rated height and the measured height. Have your students plot the rated height (y-axis) against the measured height (x-axis). What usually emerges is a reasonable approximation of a straight line, with some degree of scatter about it. Have the students eyeball a best-fitting straight line to the data points. There are a number of things you can do at this point. You can show students how they could use the relationship they have uncovered to estimate actual height from the average ratings. This provides a nice springboard for discussing such issues as the reliability or repeatability of the measurements and how averaging over repeated observations can increase the reliability. It is also a nice example in which the external validity of a measure is established by comparison with an accepted measure (concurrent validity). You might point out that if rating scales can provide valid measures of objective quantities such as height, we should have some confidence that they can provide valid measures of psychological states for which objective physical measures are lacking. Research Examples Perhaps the best way students learn how variables are chosen, operationalized, and measured is by going over specific examples in the literature. Bring in a number of primary research articles, and have students identify the variables employed in each study. Have them provide both a general name for each variable (e.g., “anxiety”) and the specific operational definition used by the study (e.g., manifest anxiety scale). Note the potential difficulty of generalizing the results to the more general concept, and the possibility that different operational definitions might lead to different conclusions. You might then ask your students to identify the type of dependent measure(s) employed and the scale of measurement, and discuss with them how the scale of measurement relates to the statistics the authors of the study used in their analysis of the data. Role Playing To bring home the reactive nature of psychological measurement, including the influences of demand characteristics, participant attitudes, and experimenter bias, you can have your students play the roles of participant and experimenter in a blatantly biased “experiment.” This is particularly effective if students are given the chance to play the participant in both the experimental and control groups so they can contrast their treatments by the experimenter. In the experimental condition, the experimenter would practically help the participant to make the correct responses (giving hints, clearly explaining what to do, etc.). In the control condition, the experimenter would be almost obstructive (e.g., refusing to explain unclear instructions, being seemingly uninterested in the participant, etc.). After the demonstration, ask students what could be done to minimize these problems. CHAPTER 7 UNDERSTANDING ETHICAL ISSUES IN THE RESEARCH PROCESS QUESTIONS TO PONDER 1. What ethical issues does Watson and Rayner’s “Little Albert” study raise? 2. What ethical issues does Ax’s experiment raise? 3. What could you do to address some of the ethical issues raised in the two studies reviewed? 4. What is the Nuremberg Code, and how does it relate to current ethical guidelines? 5. What did the Declaration of Helsinki add to the Nuremberg Code? 6. What are the three principles laid out in the Belmont Report? 7. What are the main points of the APA code of research ethics? 8. What guidelines were instituted by the Department of Health and Human Services, and why were they necessary? 9. What are the ethical issues raised by using children as research participants? 10. What special ethical concerns face you if you conduct your research on the Internet? 11. What are the issues involved in obtaining informed consent in Internet research? 12. What are the issues surrounding privacy and confidentiality in Internet research? 13. What steps can be taken to protect Internet participants’ privacy? 14. What special issues are presented by using deception in Internet research? 15. What role does an institutional review board (IRB) play in the research process? 16. Why is IRB review important? 17. What are the IRB’s two roles? 18. What are the ethical guidelines you must follow when using animal subjects? 19. What is the composition of the institutional animal care and use committee, and why is review important? 20. How does a cost–benefit analysis enter into one’s decision to conduct a study using animal subjects? 21. What does the APA ethical code say about research integrity? 22. Why should we be concerned with research fraud? 23. What is the ORI and what does it do? 24. What constitutes research fraud, and why does it occur? 25. How prevalent is research fraud? 26. How can research fraud be dealt with? CHAPTER OUTLINE Ethical Research Practice With Human Participants John Watson and Little Albert Is It Fear or Is It Anger? Putting Ethical Considerations in Context The Evolution of Ethical Principles for Research With Human Participants The Nuremberg Code The Declaration of Helsinki The Belmont Report APA Ethical Guidelines Government Regulations Internet Research and Ethical Research Practice Ethical Guidelines, Your Research, and the Institutional Review Board Ethical Considerations When Using Animal Subjects The Institutional Animal Care and Use Committee Cost–Benefit Assessment: Should the Research Be Done? Treating Science Ethically: The Importance of Research Integrity and the Problem of Research Fraud What Constitutes Fraud in Research? The Prevalence of Research Fraud Explanations for Research Fraud Dealing With Research Fraud Summary Key Terms informed consent Nuremberg Code Declaration of Helsinki Belmont Report respect for persons beneficence justice Ethical Principles of Psychologists and Code of Conduct 2002 institutional review board (IRB) institutional animal care and use committee (IACUC) Office of Research Integrity (ORI)
CHAPTER GOALS Chapter 7 addresses three issues involving ethics: (1) ethical research practice with human participants, (2) ethical research practice with animal subjects, and (3) ethical research practice in the reporting of research findings. The chapter includes a section on the role of fraud and values in research. Students should learn what constitutes research fraud, why it occurs, and how it can be prevented. Some key points to hit during class are the following: 1. The history of the emergence of the APA ethical code. 2. The main points of the Nuremberg Code, Helsinki Report and the Belmont Report. 3. The summary of APA Ethical Principles that apply to human research participants. 4. The main points of the HHS guidelines. 5. The issues of informed consent, anonymity, confidentiality, and use of deception as they arise in the context of research conducted on the Internet. 6. The steps involved in submitting a research proposal to an institutional review board (IRB). 7. The major points of the APA ethical code for using animals in research. 8. The role of the institutional animal care and use committee (IACUC) in reviewing proposals for research involving animal subjects. 9. What constitutes fraud in science. 10. The prevalence of fraud in science and why it occurs. 11. What can be done to prevent fraud in science. IDEAS FOR CLASS ACTIVITIES Ethical Dilemmas One of the major themes developed in Chapter 7 is the importance of considering the impact of research on its subjects. Following are several research examples (some of which are included in the Study Guide) illustrating a wide range of research methods and subjects. Have students evaluate each of the examples for the following questions: 1. Overall, are there any ethical issues that should be raised concerning the research described? 2. What specific ethical problems can be identified in each example (e.g., invasion of privacy, deception, stress, debriefing, and so on)? 3. How could the ethical problems be remedied? That is, what steps could a researcher take to ensure that subjects are treated ethically? (For this question, tell students not to worry about how the validity of the study would be affected.) 4. What impact would any remedies for ethical issues have on the internal and external validity of the results of each research example? After each of the dilemmas, we have included a brief synopsis of the ethical issues raised. Keep in mind that these are not necessarily ethical problems. Rather, they are issues that should be raised to help students gain a greater sensitivity to ethical issues. 1. Middlemist, Knowles, and Matter (1976) conducted a study investigating whether or not invasions of personal space are physiologically, as well as psychologically, arousing. The experiment was run in a men’s lavatory in which the investigators closed off one or another urinal. Participants were forced to urinate either in the urinal next to a male confederate of the experimenter or in the urinal one away from the confederate. A second confederate positioned in a toilet stall adjacent to the urinals observed the subjects via periscope and recorded the latency to onset of urination and its duration. [There are several ethical issues to discuss with your class in this research example. First, the issue of invasion of privacy comes up. Should researchers be observing a behavior that our culture deems as private, even though that behavior is occurring in a semiprivate place (a public lavatory)? Second, the issue of informed consent should be raised. Participants did not give prior consent to be in the study. Ask your students how obtaining true informed consent would affect the validity of the results (prior consent would lead to a biased sample because only those participants who agree to be in the experiment could be included, and once participants know about the experiment their behavior may be affected). Have students think about how the problem of informed consent could be circumvented. One possibility is to obtain consent after the participants have been observed (perhaps by another experimenter outside the lavatory). Discuss the problems this could raise (biased sample). Finally, should the participants be debriefed? What consequences might this have for participants? Have students think about an effective debriefing procedure for this experiment.] 2. In a classic experiment on cognitive dissonance theory, Festinger and Carlsmith (1959) had participants engage in an extremely boring task. After doing this task, participants were asked if they would help out in the experiment with the next participant, because the regular assistant did not show up. Each accomplice-participant was asked to tell the next participant that the task was really interesting and exciting. For telling this little white lie, the accomplice-participant was promised payment (either $1 or $20, depending on the condition assigned). The experiment was actually designed to see if the amount of money promised affected the accomplice-participant’s attitude toward the boring task. At the end of the experiment, participants were asked to return the money. [Students normally focus on the fact that participants were asked to give back the money they had been paid. You could discuss the ethics of promising participants something and then reneging on that promise. Because participants were asked to return the money, there is probably no ethical violation. A second issue is the fact that participants were induced to lie. This may lead to subtle shifts in the participants’ self-perceptions (they may have come into the experiment believing they were honest and left with some doubts). Is it ethical to induce participants to perform a behavior our culture discourages? Have students discuss this issue. Also, discuss the issue of informed consent. How could it be obtained without reducing the validity of the results? Finally, have students design an effective debriefing session to reduce any negative effects of the experiment.] 3. In his classic study of obedience to authority, Milgram (1963) led participants to believe they were administering painful shocks to another participant (actually a confederate of the experimenter who was receiving no shocks) in a bogus learning experiment. Participants were told that each time the “learner” made a mistake, an electric shock of increasing intensity (15–450 volts) must be delivered. In some conditions, the ‘learner” screamed and pounded on the wall. Participants who expressed reluctance to continue were told that they “must go on.” The experiment actually examined whether participants would obey the experimenter’s orders to continue delivering shocks. [As you probably know, Milgram was criticized for the ethics of this famous study. Several ethical issues are raised in this example. First, participants were successfully deceived into believing they were administering shocks. This would be a good opportunity to discuss the possibility of using role playing as an alternative to the method used by Milgram. Have students consider whether the results obtained in a role-playing study would parallel Milgram’s (they most likely would not). Second, discuss the issue of informed consent. How could informed consent be obtained for this experiment while preserving the power of the situation? (Discuss the possibility of informing participants that the experiment would involve high levels of stress, without specifying the source. If this were done, raise the possibility of ending up with a biased sample, because only those participants who think they could endure the stress would remain in the experiment. Would including the caveat of “withdrawing at any time” in an informed consent form affect the results?) Third, discuss the shifts that might occur in participants’ self-perception. Finally, discuss the importance of following up with debriefing. Discuss the need for long-term follow-up (which Milgram included) in an experiment such as this.] 4. In a simulation study of plea bargaining, Gregory, Mowen, and Linder (1978) gave false feedback to undergraduate participants. Participants sat in a waiting room for an experiment to begin. In one condition, another participant who had just been in the experiment (actually a confederate of the experimenter) told the waiting participant that most of the answers to the test that would be taken are “B.” In a second condition, no information was given to the waiting participant. After the participant took the test, the experimenter accused the participant of cheating and said that it was a serious matter that would have to be presented to a review board for action. Participants were led to believe that the consequences of the accused cheating were severe. The participants were told that if they admitted cheating, they would simply lose credit for participating in the experiment. [The main point in this example is that participants were placed in a highly stressful situation in which they were deceived. Discussions could parallel those centering on the Milgram experiment. Issues of adequate informed consent, stress, debriefing, and so on should be discussed.] 5. In a field experiment conducted by Harari, Harari, and White (1985), male participants walking alone or in groups were exposed to a simulated rape. As participants walked along a path to a parking lot, a male and a female confederate of the experimenters acted out a simulated rape. (The male grabbed the female around the waist, put his hand over her mouth, and dragged her into some bushes. The female screamed for help.) Observers stationed at various points recorded the number of participants who offered help. Prior to actual intervention, participants were stopped and told of the true purpose of the experiment. The results showed that 85 percent of the participants walking in groups offered help to the victim, whereas 65 percent of the participants walking alone offered help. Support was found for the idea that individuals in groups are as likely, if not more likely, to help when the victim is clearly in need of help. [Because this is a field experiment, prior informed consent was not obtained from participants. Have the class discuss whether informed consent could have been obtained. You might point out that informed consent might be waived by an IRB if the researchers can demonstrate the potential importance of the research results. You can discuss the possibility of obtaining post hoc informed consent (participants are to be asked for consent after they participate in the experiment). Other issues to raise are the ethics of exposing participants to a highly stressful event and the unpredictability of the participants’ reactions to it (for example, what if one participant had a gun and used it?). Harari et al. make no mention of debriefing. Should participants be stopped and debriefed after the experiment? What about participants who choose not to help? How should they be debriefed? You can use this example to start a discussion of the ethical issues surrounding field experimentation.] 6. To test the effects of controllability on stress responses, Weiss (1971) administered electric shock to rats. Three rats were run together in a yoked procedure. One rat in the triad could avoid or escape the electric shock by making a response. A second rat received the same number of shocks of the same duration as the first rat but could neither avoid nor escape them. The third rat received no shocks. Immediately after the experimental session, the rats were removed from the test chambers and sacrificed. Their stomachs were then inspected for ulcers. The rats that lacked control over the shock had developed extensive lesioning, whereas neither the control rats (those receiving no shocks) nor those that could avoid or escape the shock showed much damage. [Some students may react to the use of electric shock in this (and other) studies. Use this example to reinforce the points about ethical treatment of animals in research. Discuss the cost–benefit ratio: Are the results of this study (and similar studies) important enough to warrant using electric shock? Who should make such a decision? Another aspect of your discussion could focus on the sacrificing of the animals to inspect their stomachs. Was there a need to sacrifice the animals (are there alternatives)? Were the rats sacrificed humanely? You can use this example to discuss whether the alternatives to using animals in research discussed in the text could be used in behavioral research.] References for Ethical Dilemmas Festinger, L., & Carlsmith, J. M. (1959). Cognitive consequences of forced compliance. Journal of Abnormal and Social Psychology, 58, 203–210. Gregory, W. L., Mowen, J. C., & Linder, D. E. (1978). Social psychology and plea bargaining: Applications, methodology and theory. Journal of Personality and Social Psychology, 36, 1521–1530. Harari, H., Harari, O., & White, R. V. (1985). The reaction to rape by American male bystanders. Journal of Social Psychology, 125, 653–658. Middlemist, R. D., Knowles, E. S., & Matter, C. F. (1976). Personal space invasions in the lavatory: Suggestive evidence for arousal. Journal of Personality and Social Psychology, 33, 541–546. Milgram, S. (1963). Behavioral study of obedience. Journal of Abnormal Psychology, 67, 371–378. Weiss, J. M. (1971). Effects of coping behavior in different warning signal conditions on stress pathology in rats. Journal of Comparative and Physiological Psychology, 77, 1–13. Simulating an IRB In Chapter 7, the institutional review board was discussed. For this exercise, assign students to research groups (which you could do early in the semester in preparation for this exercise). Have each research group come up with a research idea and develop a proposal for their study, specifying the subjects to be used, materials, procedures, informed consent, and strategies to protect participants. Once students have developed their proposals, have each research group assume the role of an institutional review board and evaluate the other groups’ research proposals. Have them screen each proposal according to the APA ethical guidelines outlined in Chapter 7. If any ethical problems are encountered, the IRBs should identify them and recommend remedies. After the proposals have been screened, you can lead a general class discussion of what ethical problems arose, and how they might be eliminated. You should also discuss how the steps taken to remedy any ethical problems affect the validity of the results of each study. Confronting Research Fraud Have students visit the Office of Research Integrity (ORI) Web site and take part in one of the online training exercises (under Programs and RIO Training). Although the materials available might change, there are usually exercises where students can read an scenario involving an instance of research fraud and answer a series of questions about the scenario. Have students complete one or more of these exercises and bring their responses to class for discussion. CHAPTER 8 USING NONEXPERIMENTAL RESEARCH QUESTIONS TO PONDER 1. What are the defining characteristics of observational research? 2. How are behavioral categories that are used in observational research developed? 3. What are the techniques used to make behavioral observations in observational research? 4. What is the distinction between recording single acts and behavior sequences? 5. What are the sampling techniques used to handle complexity when making behavioral observations? 6. Why should you evaluate interrater reliability? 7. What are the techniques used to evaluate interrater reliability, and when would each be used? 8. How do you deal with data from multiple observers? 9. What are the sources of bias in observational research, and how can the bias be reduced? 10. What is the difference between quantitative and qualitative data? 11. What are the problems inherent in collecting qualitative data? 12. Define naturalistic observation and unobtrusive observation. How are they used to study behavior? 13. What are some of the advantages and disadvantages of naturalistic observation? 14. What is ethnography, and what are the issues facing a field ethnographer? 15. How are ethnographic data recorded and analyzed? 16. What is sociometry, and when is it used? 17. How are the case history and archival research used? 18. What is content analysis, and what steps are taken when using it? 19. What is information extraction, and in what research situations would you consider using it? 20. What is meta-analysis, and what steps are involved in using it? 21. What are some of the issues facing you if you decide to do a meta-analysis? CHAPTER OUTLINE Conducting Observational Research An Example of Observational Research: Are Children Really Cruel? Developing Behavioral Categories Quantifying Behavior in an Observational Study Recording Single Events or Behavior Sequences Coping With Complexity Establishing the Reliability of Your Observations Sources of Bias in Observational Research Quantitative and Qualitative Approaches to Data Collection Nonexperimental Research Designs Naturalistic Observation Ethnography Sociometry The Case History Archival Research Content Analysis Meta-Analysis: A Tool for Comparing Results Across Studies Step 1: Identifying Relevant Variables Step 2: Locating Relevant Research to Review Step 3: Conducting the Meta-Analysis Drawbacks to Meta-Analysis Summary Key Terms behavioral categories interrater reliability Cohen’s Kappa intraclass correlation coefficient (rI) quantitative data qualitative data naturalistic observation ethnography participant observation nonparticipant observation sociometry sociogram case history archival research content analysis information extraction meta-analysis CHAPTER GOALS Chapter 8 introduces several types of nonexperimental research. The opening section of the chapter discusses issues related to observational research: developing behavioral categories, quantifying behavior, coping with complexity, and assessing interrater reliability. Students should learn that observation is fundamental to all forms of research, not just purely observational research. Next, students are introduced to various nonexperimental techniques (e.g., naturalistic observation, ethnography, participant and nonparticipant observation, sociometry, content analysis). The pros and cons of each technique should be reinforced. In class you can focus on the following: 1. How behavioral categories are developed, and the importance of clear category definitions. 2. How researchers cope with the complexities inherent in observational research by adopting different observation techniques. 3. The distinction between recording single events and behavior sequences, and why you might want to record behavior sequences. 4. How different sampling techniques (e.g., time sampling, individual sampling) are used to cope with complexity in observational research. 5. How percent agreement, Cohen’s Kappa, Pearson correlation, and the intraclass correlation coefficient can be used to evaluate interrater reliability. 6. How data from multiple observers are handled. 7. The sources of bias in observational research, and the techniques that are used to reduce bias. 8. The distinction between quantitative and qualitative data, and some of the special problems posed by collecting qualitative data. 9. The defining qualities of naturalistic observation and techniques for making unobtrusive observations, along with the advantages and disadvantages of this technique. 10. The defining characteristics of ethnography and the issues facing an ethnographer (e.g., participant versus nonparticipant observation, obtaining access to a field setting, gaining entry into a group). 11. How observations are made and data recorded and analyzed in ethnography. 12. The defining characteristics of sociometry, and how it can be used in observational research. 13. The defining characteristics of the case history method and archival research. 14. How a content analysis is done, including the distinction between recording and context units, and the requirements for doing high-quality content analyses. 15. How information extraction and data mining are used to content analyze large databases and how natural language processing is involved in these techniques. 16. How meta-analysis is used to evaluate the results from multiple studies in an area. 17. The steps involved in conducting meta-analysis. 18. The problems inherent in meta-analysis. 19. How the results from a meta-analysis compare to traditional literature reviews. IDEAS FOR CLASS ACTIVITIES Observational Research I To introduce your students to observational research, have them conduct a small-scale observational study. Note that this exercise might take two or three class periods because students will be designing their own measures, defining behaviors, and carrying out behavioral observations. It is a good idea to plan for this exercise in your schedule. Break up your class into small research groups (three or four students per group works well), and have each group choose behavior to observe (they may choose from the ideas in the Study Guide). Each group should come up with an operational definition of the behavior to be observed (for example, aggression is any act in which one child hits another), relevant behavioral categories, a coding sheet (patterned after the one shown in Chapter 8), an observation method, and a strategy for dealing with multiple observers and interrater reliability. It is a good idea for you to sit in with each group to give students direction and help them develop their measures. However, let them come up with the idea and decide how to develop it, even if there are some flaws in their strategy. Often, students learn more about research by making some mistakes. Later you can discuss with the class what problems occurred within each group, and how those problems could be rectified. After each group has chosen a behavior and developed its observation scheme, have the groups actually make some behavioral observations. Once the data have been collected, have each group summarize its data and evaluate interrater reliability using one of the methods discussed in Chapter 8. After all the groups have collected and summarized their data, have each group present its study to the class as a whole. Use this as an opportunity to point out important methodological issues to the class (e.g., operational definitions, how behavioral categories were defined, how behavior was observed, and any problems that were encountered). If problems were encountered, have the class analyze how those problems could have been avoided or how they might be handled in a second observational study. Observational Research II If you do not want to dedicate two or three class periods to carrying out the first exercise, you can still conduct an observational study by assigning the class as a whole to do a single study (perhaps one of the ideas in the Study Guide could be used). If you have a local zoo nearby, you can assign students to make observations of animal behavior. Perhaps subgroups of the class could observe different animals. The following observational study of primate behavior was suggested to us by our colleague Carol Lawton and is based on a project developed at the National Zoo in Washington, D.C. [Office of Education, National Zoological Park (1989), A curriculum for high school students, Washington, D.C., Smithsonian Institution]. This project gives students a good introduction to methods used in observational research. Students go to the local zoo and, working in pairs, make behavioral observations of monkeys. Materials A data coding sheet with columns representing the behavioral categories to be observed and rows representing time intervals is needed for this project. (A data coding sheet for this project is provided at the end of this section.) Students should choose four behavioral categories and label the columns on the data coding sheet accordingly. Some suggestions for behavioral categories are eating, grooming, climbing, playing, and attending to the observers. Students should also rate the activity level shown by the animals within each time interval. A column for this, subdivided into four smaller columns (labeled 1, 2, 3, and 4, respectively), is provided on the coding sheet. Use the following scale to rate activity level: 1 = little movement 2 = movement of limbs, head, or trunk with the animal remaining in one position 3 = slow-paced locomotion from one location to another 4 = fast-paced locomotion To signal observation periods, have students use portable tape recorders equipped with tapes that emit a beep every 30 seconds. Primate Observational Study Coding Sheet Observer_____________________________________________ Date____________ Time____________ Weather______________________________________ Species___________________________ Individual________________________________
TIME INTERVAL BEHAVIORS ACTIVITY LEVEL COMMENTS
Check behavior once per interval if it occurs Rate at the beep
1 2 3 4
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
Total
Frequencies
Procedure Students should spend about 10 minutes making casual observations of the monkeys. From these observations, the students should develop their four behavioral categories. Have students develop clear definitions for those behaviors. The four behaviors to be observed will be entered on the data sheet under the heading “Behaviors.” For each 30-second interval, each student will place a check mark in the appropriate column for each behavior that occurs within the interval. Activity ratings should be made at the beep by checking one of the four levels. For this project, students will work in pairs, each making an independent set of observations of the same monkey at the same time. Give students an initial 5-minute practice session to familiarize themselves with the task and to work out any discrepancies or clarify definitions. The main observation period should last for at least 15 minutes. Students can repeat their observations for a second subject and then compare observations across subjects (optional). Results Once all observations have been made, students should conduct the following data analyses: 1. Choose data from a subject (if more than one was observed) and compute interrater reliabilities using Cohen’s Kappa. A Cohen’s Kappa should be computed for each of the four behavioral categories and for the activity level measure. In a lab report, students could discuss any disagreements between observers. How could disagreement be reduced? 2. Have students compute frequencies with which each behavior occurred and an average activity level. Have them evaluate which behaviors were most frequent and any differences observed across subjects. 3. Have students think back on their observation period and record any difficulties that were encountered (e.g., complexity of the behaviors observed, difficulty making observations). Discuss with the class how these problems could be circumvented. Content Analysis A content analysis is a relatively easy and interesting research project for students to work on. Divide your class into several research groups (again, three or four students per group works well) and have them decide on something to content-analyze (e.g., television shows, magazine advertisements). Use the following set of guidelines for this project: 1. Choose something to content-analyze. Some possible areas are the following: a. Sexism in children’s literature. b. Aggression and violence in children’s television programming. c. Images of men and women in magazine or television advertisements. d. Sexual or violent themes presented in music videos. e. Racial portrayals in prime-time television. f. Gender differences in the content of personal advertisements. g. The content of “one-minute” speeches made by Republicans and Democrats on the floor of the U.S. Congress (access to C-SPAN on cable television is needed). 2. Develop a research hypothesis that will be the focus of the content analysis. For example, if a group chooses to content-analyze music videos, you might develop a hypothesis about the frequency of sexual content in videos by male and female artists or in videos shown on MTV and VH-1. If sexism in children’s literature is chosen, the hypothesis might focus on classics (e.g., Cinderella) versus modem stories. 3. Generate a list of appropriate behavioral categories, recording units, and context units for your content analysis. A quick literature review might help students develop their categories and the recording units. A coding sheet should be devised that includes the recording and context units to be used. 4. Conduct the content analysis. This will involve finding and analyzing representative samples of the material. Students must give some thought as to how they will select materials to content-analyze. 5. Analyze and interpret the data. Data analysis should focus on a. The frequencies within each behavioral category. b. The differences across “groups” relevant to the hypothesis (e.g., Did videos from MTV differ from those on VH-1? If so, how?). c. Measures of interrater reliability. d. Difficulties encountered when doing the content analysis (clearly defining recording units, finding materials, and so on). 6. Each research group should prepare a brief (10–15-minute) talk to present to the class. The presentations allow class members to share their experiences and learn what the other groups did for their content analyses. The talk should include the following: a. An outline of the topic under study. b. A description of how the recording/context units were developed. c. The method used to acquire materials for content analysis. d. A description of how the content analysis was done. e. A summary of the findings. f. A discussion of any difficulties encountered when doing the analysis. g. A discussion of the generality of the results. (Are there any methodological considerations that limit the conclusions you may draw from the results?) Ethnography I: Participant Observation In ethnography, a qualitative written record is made of observed behaviors. Have your students design and carry out an ethnographic study. They might, for example, study a group to which they belong (e.g., a bowling league, a church organization, a club) as a participant observer. Students should record the nature of the interactions that take place within the group, the hierarchical structure, if any, and perhaps even nonverbal cues (facial expressions, body language, and so on.). They should make careful notes of the behaviors they observe. Students should devise a method for observing behavior, making field notes, transcribing field notes, and analyzing their data. Ethnography II: Nonparticipant Observation Have students design and carry out a nonparticipant observation ethnographic study. Students should find a relatively crowded place (e.g., shopping mall, skating rink, bowling alley, sporting event, airport) and make careful observations of the people in that place. Both verbal (what people say to each other) and nonverbal (what people are doing) behaviors should be recorded. Students should work out a strategy for taking field notes and later making more complete notes at the end of the day. Students should also work out a strategy for remaining unobtrusive and for analyzing their data. Sociometry Have students carry out a sociometric study of a group. The group could be a class, a club, or a sports team. Students should list the names of all members of the group on a piece of paper. Then, using the coding categories that follow, a number is placed next to each person’s name indicating how they feel about that particular person. The number 1 must be used only once, and students should put a zero by their own name. Each person in the group should complete the sociometric measure. When all members of the group have completed the sociometric instrument, the results are to be summarized on a scoring sheet similar to the one shown in Figure 7-4 and then shown graphically on a sociogram. Sociometric Rating Instrument (Berg, 1998) Question/Choices Directions: On a separate sheet, write the name of everyone in your group or organization. Read the following paragraphs, and place their corresponding numbers in front of every name for which they apply. You may use the number 1 only once, and please place only a single number by each name. By your own name, please place a zero. My Very Best Friend 1. I would like to have this person as one of my very best friends. I would like to spend a great deal of time with this person. I think I could tell some of my problems and concerns to this person, and I would do everything I could to help this person with his or her problems and concerns. I will give a number 1 to my very best friend. My Other Friend(s) 2. I would enjoy working and doing things with this person. I would invite this person to a party in my home, and I would enjoy going places with this person and our other friends. I would like to talk and do a variety of things with this person and to be with this person often. I want this person to be one of my friends. I will give a number 2 to every person who is my friend. I Do Not Know This Person 3. I do not know this person very well. Maybe I would like this person if I got to know him or her; maybe I would not. I do not know whether I would like to spend time or work with this person. I will place a number 3 in front of the name of every person I do not know very well. I Do Not Care for This Person 4. I will greet this person when I see him or her around school or in a store, but I do not enjoy being around this person. I might spend some time with this person—if I had nothing to do, or if I had a social obligation to attend where this person also was in attendance. I do not care for this person very much. I will place a number 4 in front of the name of every person I do not care for very much. I Dislike This Person 5. I speak to this person only when it is necessary. I do not like to work or spend time with this person. I avoid serving on the same groups or committees with this person. I will place a number 5 in front of the name of every person I do not like. Exploring Meta-Analysis Find a recent literature review that includes a meta-analysis of the findings, and distribute copies to the class. Go over the paper with them and point out how meta-analysis is used to support the conclusions drawn by the author. Note which meta-analytic techniques were used, how the author handled the file drawer issue, and how the meta-analysis was conducted. Discuss with your class how the meta-analysis differs from a traditional literature review. Instructor Manual for Research Design and Methods: A Process Approach Kenneth Bordens, Bruce Barrington Abbott 9780078035456
Download
Close
Close
