Preview Extract
This document contains Chapters 9 to 11 CHAPTER 9 USING SURVEY RESEARCH QUESTIONS TO PONDER 1. What are some of the applications of survey research? 2. Why is it important to know about survey methods, even if you do not intend to conduct surveys? 3. How does a field survey differ from other observational methods? 4. What are anonymity and confidentiality, and why are they important? 5. What are the steps involved in designing a questionnaire? 6. How do open-ended and restricted items differ, and what are the advantages and disadvantages of each? 7. What are the ways in which questionnaire items can be formatted? 8. What are some of the factors that you should pay attention to when constructing questionnaire items? 9. How do you design effective rating scales? 10. Why is the first question on a questionnaire so important? 11. What does it mean that a questionnaire should have continuity? Why is continuity important? 12. What is a questionnaire’s navigational path, and why is it important? 13. What are the different ways of administering a questionnaire? 14. What are the advantages and disadvantages of the different ways of administering a questionnaire? 15. What is nonresponse bias, and what can you do to combat it? 16. How do social desirability effects affect your decision about how to administer a questionnaire? 17. What is meant by the reliability of a questionnaire, and why is it important? 18. How do you assess reliability with repeated administrations? 19. How do you assess reliability with a single administration? 20. What steps can be taken to increase reliability? 21. What is the validity of a questionnaire, and why is it important? 22. What are the different types of validity you should consider? 23. What factors can affect the validity of your questionnaire? 24. What is a representative sample, and why is it important to have one for a survey? 25. What is a biased sample, and how can a biased sample affect your results? 26. What is a random sample, and why is it important to do random sampling? 27. What is simple random sampling? 28. What are the various sampling techniques that represent modifications of simple random sampling? 29. Under what conditions would you use each of the sampling techniques discussed in the chapter? 30. What are the implications of using a nonrandom sample? 31. What is meant by an “economic sample”? 32. What is sampling error, and how do you know if you have an acceptable level? 33. How does the magnitude of the differences you expect to observe affect your decision about sample size? 34. What are some of the sample size issues you need to consider for different sampling techniques? CHAPTER OUTLINE Survey Research Designing Your Questionnaire Writing Questionnaire Items Assembling Your Questionnaire Administering Your Questionnaire Mail Surveys Internet Surveys Telephone Surveys Group-Administered Surveys Face-to-Face Interviews Mixed-Mode Surveys A Final Note on Survey Techniques Assessing the Reliability of Your Questionnaire Assessing Reliability by Repeated Administration Assessing Reliability With a Single Administration Increasing Reliability Assessing the Validity of Your Questionnaire Acquiring a Sample for Your Survey Representativeness Sampling Techniques Random and Nonrandom Sampling Revisited Sample Size Summary Key Terms open-ended item restricted item partially open-ended item mail survey nonresponse bias Internet survey telephone survey face-to-face interview mixed-mode survey representative sample biased sample simple random sampling stratified sampling proportionate sampling systematic sampling cluster sampling multistage sampling sampling error
CHAPTER GOALS Chapter 9 introduces the specific techniques and general logic of survey methodology beginning with questionnaire construction, then covering ways to administer the questionnaire (mail, Internet, telephone, group, face-to-face) and assess reliability and validity, and ending with sampling, including logic, issues, and specific sampling techniques. Your classroom discussions might focus on the following: 1. Why it is important to learn about survey research. 2. The steps involved in designing a questionnaire. 3. What demographics are and the distinction between predictor and criterion variables. 4. The types of questionnaire items and the advantages of each. 5. How to design rating scales. 6. How to write clear questionnaire items that minimize bias. 7. How to assemble the questionnaire and develop a navigational path to keep respondents interested in and clear about what they should be doing. 8. The two languages (verbal and graphic), and how they are used to assemble a questionnaire. 9. The types of surveys, their relative advantages and disadvantages, and how they should be conducted. 10. Techniques for assessing the reliability and validity of a questionnaire. 11. Sample representativeness. 12. Types of sampling techniques and how to use each. 13. Revisit the issue of random versus nonrandom sampling as it applies to survey research. 14. How to determine an appropriate sample size. IDEAS FOR CLASS ACTIVITIES Attitude Questionnaire The following is a laboratory exercise involving the construction and administration of an attitude questionnaire and the analysis of the resulting data. For simplicity, you might want to use a convenience sample for this exercise (e.g., students in the university cafeteria or library) while reminding students that a professional survey would use some form of random sampling procedure and that, because we used a convenience sample, the population to which their results apply is not really known. Attitude Questionnaire Purpose and Overview The purpose of this lab exercise is to introduce you to constructing a measurement instrument. For this exercise, you will construct and evaluate an attitude questionnaire. After administering your questionnaire, you will evaluate it for its reliability and for the internal consistency of items. Each lab group will decide on a topic about which attitudes will be tested. Topics might include issues relating to current events (e.g., attitudes toward recycling, the president’s policies), individual attitudes (e.g., sex roles), or any other topic that interests you. The only restriction is that you choose a nonsensitive topic (e.g., don’t develop a questionnaire on sexual or religious attitudes). II. Format of the Questionnaire Likert Items Your questionnaire will consist of at least 15 Likert items measuring the same attitude. A Likert item consists of a statement followed by a five-point rating scale on which the subject indicates the degree of agreement with the statement. For example:
Strongly Disagree Disagree Neutral Agree Strongly Agree
The president is doing a good job on the war on terror 1 2 3 4 5
Negatively and Positively Keyed Items To reduce the chances of a response set bias, you should have some of your items positively keyed and some negatively keyed. A positively keyed item is one where agreement means a positive attitude toward the issue being evaluated. For example, the preceding item is positively keyed because a high score (strongly agree) means that the subject is generally favorable toward how the president is handling the budget deficit. In contrast, a negatively keyed item is one where disagreement indicates a positive attitude. For example:
Strongly Disagree Disagree Neutral Agree Strongly Agree
The president is not doing a good job on the war on terror 1 2 3 4 5
Disagreement with this item indicates a generally favorable attitude toward how the president is handling budget cutting. Try, if you can, to have half of your items positively keyed and half negatively keyed. Demographic/Predictor Items In addition to the Liken items evaluating subjects’ attitudes, you should come up with a few items (four or five perhaps) that evaluate something about the subjects. These items should be ones that you feel will correlate with the attitude measured on your questionnaire. Some examples, relevant to the above example, might be Political party affiliation Educational level Occupation Income Whatever you choose to measure, keep in mind that these items must also be carefully constructed (see Chapter 9 in the text for information on how to construct questionnaire items). Design these with an eye toward correlating them with attitudes. Instructions/Informed Consent In addition to the questionnaire items, you will have to develop a set of instructions that generally explain how the questionnaire is set up and how to complete the specific types of items included on the questionnaire. Also, you will have to develop an appropriate informed consent form (see Figure 6-2 in the text for an example of an informed consent form). The informed consent form must contain the following: 1. A statement that the survey is being conducted as a requirement for your Research Methods class. 2. A statement about the topic of your survey and the requirements for participation, including a statement indicating that participation is voluntary, and that the participant is free to withdraw at any time. 3. A statement that all data will be held in confidence, and that the data will be used only within the Research Methods class for educational purposes. 4. An acknowledgment that the participant understands the nature and purpose of the study, that he/she is at least 18 years old, and that the participant freely consents to participate. III. Administering Your Questionnaire Each person in your lab group will administer the questionnaire to at least five people. Work out within your lab group precisely how the questionnaire is to be administered by members of your group. IV. Analysis Your questionnaire data will be evaluated along the following lines: 1. You will obtain an overall score on the questionnaire by summing over all items on your questionnaire. Before you do this, be sure to rekey your negatively keyed items so that they are positively keyed. That is, assign strongly disagree = 5, disagree = 4, and so on. 2. Descriptive statistics will be computed for the overall score as well as for each item separately. 3. The reliability of your questionnaire will be evaluated by applying coefficient alpha, which can be found in most statistical analysis software packages such as SPSS. CHAPTER 10 USING BETWEEN-SUBJECTS AND WITHIN-SUBJECTS EXPERIMENTAL DESIGNS REVIEW QUESTIONS 1. How do between-subjects, within-subjects, and single-subject experiments differ? 2. What are the sources of error variance in a between-subjects design, and how might error variance affect your results? 3. What steps can you take to deal with error variance in a between-subjects design? 4. How are statistics used to test the reliability of data from a between-subjects experiment? 5. How does a two-group, randomized design work? 6. What are some of the advantages and disadvantages of the two-group, randomized design? 7. How do parametric and nonparametric multigroup randomized designs work? 8. What is a matched-groups design, and when would you use one? 9. How does a matched-pairs design differ from a randomized two-group design? 10. What are some of the advantages and disadvantages of the matching strategy? 11. How does a within-subjects design differ from a between-subjects design? 12. What are the advantages of the within-subjects experimental design? 13. What are the disadvantages of the within-subjects experimental design? 14. How do carryover effects influence the interpretation of the results from a within- subjects experiment? 15. What are the sources of carryover effects in a within-subjects design? 16. Under what conditions will counterbalancing be effective or ineffective in dealing with carryover effects? 17. When do you use a Latin square design? 18. What strategies can be used to deal with carryover effects? 19. Do between-subjects and within-subjects designs applied to the same variables always produce the same functional relationship? Why or why not? 20. When should you consider using a within-subjects design instead of a between- subjects design? 21. When should you consider using a matched-groups design rather than a within- subjects design? 22. How do single-factor and multifactor experimental design differ? 23. What is a factorial design? 24. What are the advantages to using a multifactor experimental design? 25. What is a main effect? 26. What is an interaction, and how does it differ from main effects? 27. What is a higher-order factorial design? 28. What are the advantages and disadvantages of a higher-order factorial design? 29. What are some of the other group-based designs? 30. How do designs with more than one dependent variable work? 31. How does a confounding variable affect the validity of your results? 32. How can confounding variables be eliminated? CHAPTER OUTLINE Types of Experimental Design The Problem of Error Variance in Between-Subjects and Within-Subjects Designs Sources of Error Variance Handling Error Variance Between-Subjects Designs The Single-Factor Randomized-Groups Design Matched-Groups Designs Within-Subjects Designs An Example of a Within-Subjects Design: Does Caffeine Keep Us Going? Advantages and Disadvantages of the Within-Subjects Design Sources of Carryover Dealing With Carryover Effects When to Use a Within-Subjects Design Within-Subjects Versus Matched-Groups Designs Types of Within-Subjects Designs Factorial Designs: Designs With Two or More Independent Variables An Example of a Factorial Design: Can That Witness Really Not Remember an Important Event? Main Effects and Interactions Factorial Within-Subjects Designs Higher-Order Factorial Designs Other Group-Based Designs Designs With Two or More Dependent Variables Confounding and Experimental Design Summary Key Terms
between-subjects design within-subjects design single-subject design error variance randomized two-group design parametric design nonparametric design multiple control group design matched-groups design matched-pairs design carryover effect counterbalancing factorial design main effect interaction simple main effect higher-order factorial design
CHAPTER GOALS Chapter 10 is the first chapter on experimental designs, and therefore includes a brief overview of the three main types of experimental design: between-subjects, within-subjects, and single-subject. This is followed by a description of the logic underlying between-subjects designs including sources of variance and how between-subjects designs deal with them. The bulk of the chapter is taken up by descriptions of the various types of between-subjects and within-subjects designs, including single-factor randomized groups designs, matched groups and pairs designs, various within-subjects designs (including two-level and multilevel), factorial designs, and multivariate designs. The chapter closes with a discussion of confounding. When they have completed Chapter 10, students should know the following: 1. The defining characteristics of between-subjects, within-subjects, and single-subject designs. 2. The sources of error variance in experimental designs and how error variance is handled (reducing error variance, increasing effectiveness of the independent variable, and statistical analysis). 3. The characteristics of the single-factor, two-group randomized design. 4. The limitations of the single-factor, two-group design. 5. How the basic two-group randomized design can be expanded. 6. The distinction between parametric and nonparametric multigroup experiments. 7. The logic behind matching, and the strategies involved in the matched groups and matched pairs designs. 8. The defining characteristics of the within-subjects design, along with its advantages and disadvantages. 9. The problem of carryover effects, and the strategies used to deal with carryover. 10. When within-subjects designs should be used. 11. The single-factor within-subjects designs (two-level and multilevel designs). 12. The basic logic behind designs with two or more independent variables (factorial designs). 13. The distinction between main effects and interactions. 14. Higher order between-subjects and within-subjects designs. 15. What it means to say that a design is multivariate. 16. What confounding is, and some ways in which it can creep into a design. IDEAS FOR CLASS ACTIVITIES Imagery and Recall The following is a handout we give our students that provides the materials and method for a 2 x 2 factorial between-subjects design. The factors manipulated are (a) instructions to use imagery or rote memorization [Instruction Type] and (b) presentation of concrete or abstract words [Word Type]. The two variables are so powerful that they are almost certain to produce good results, including a significant interaction. This provides a nice opportunity to show students what an interaction is, and how it can be interpreted. Imagery and Recall Experiment For our next lab project, we will be conducting an experiment on the relationship between imagery and recall. Research has shown that imagery tends to facilitate memory processes. In this experiment we will test this idea. Design This lab project will illustrate a two-factor, completely between-subjects experimental design. There are two factors: Instruction Type (Imagery versus Rehearsal) and Word Type (Concrete and Abstract). Hence this experiment is a 2 x 2 factorial. Participants Each of you will be responsible for finding and running four participants (one in each of the experimental conditions). Materials The materials for this experiment will be an informed consent form, the instructions to be read to participants (see attached page), and two lists of words, one consisting of concrete nouns and the other consisting of abstract nouns (see attached lists). Additionally, each participant will be provided with a response sheet. Your class should develop an informed consent form following the guidelines in Chapter 7. Procedure Participants are to be approached and asked if they would mind taking part in a simple experiment on memory. If the participant agrees, find a quiet place to conduct the experiment. Give the participant an informed consent form, and read the form to him or her. After reading the informed consent form, have the participant sign and date it and return it to you. Next, read to the participant the appropriate instructions, and read the list of word pairs. The list should be read at a rate of one word pair every 15 seconds in all conditions. After reading the list to the participant, go back and read each stimulus word to the participant, and have him or her write down the appropriate response word on the response sheet. Read the list of stimulus words at a rate of one every 15 seconds. Note: Each “experimenter” must run one participant in each of the experimental conditions. (So each of you will run a participant in the imagery/high-imagery words, imagery/low-imagery words, rehearsal/high-imagery words, and rehearsal/low-imagery words conditions.) Instructions: Imagery Group This experiment concerns how well you can remember words. I will read you a list of word pairs (for example: water—bird). After I have read the entire list to you, I will give you the first word from each pair (for example, water) and you will give me the word that was paired with it (for example, bird). Also, I will be giving you the first word from each pair in an order that is different from the order in which the list will be read originally. So don’t try to remember the word pairs in order. When you are trying to learn the word pairs, try to form a mental image (picture) of the two words going together. For example, if the word pair was water—bird, you might form the image of a bird sitting in a pool of water. Try to form such an image for each word pair. Are there any questions? Are you ready to begin? Instructions: Rehearsal Group This experiment concerns how well you can remember words. I will read you a list of word pairs (for example: water—bird). After I have read you the entire list, I will give you the first word from each pair (for example, water) and you will give me the word that was paired with it (for example, bird). Also, I will be giving you the first word from each pair in an order that is different from the order in which the list will be read originally. So don’t try to remember the word pairs in order. When you are trying to learn the word pairs, say each word pair over and over to yourself. For example, if the word pair was water—bird, you would say “water—bird, water—bird, water—bird,” and so on, until I read you the next word pair. Rehearse each word pair in this way. Are there any questions? Are you ready to begin? HIGH-IMAGERY LIST
Stimulus Words Response Words
POTATO ROPE
BASKET FLOWER
BELL PRISON
STUDENT BOTTLE
TICKET COFFEE
BREAKFAST APPLE
LIBRARY DIAMOND
PALACE TOWER
PIE STAMP
SAILOR TONGUE
FUR MAGAZINE
FACTORY FURNITURE
LOW-IMAGERY LIST
Stimulus Words Response Words
ANGER BURDEN
BOND BREATH
LIBERTY EFFECT
AID HABIT
JUSTICE OWNER
ADVENTURE FAULT
SERIES HAPPINESS
PATTERN SECRET
SUM APPEARANCE
MASS EVIDENCE
CONTRACT SIN
EVENT RELIEF
Order of Stimulus Words for Recall
HIGH-IMAGERY LIST BREAKFAST FACTORY PALACE STUDENT SAILOR POTATO PIE TICKET FUR BASKET LIBRARY BELL LOW-IMAGERY LIST LIBERTY SUM JUSTICE AID ADVENTURE MASS CONTRACT EVENT ANGER PATTERN BOND SERIES
Experimental Design with Codes
High Imagery Low Imagery
Imagery instructions 1 2
Rehearsal instructions 3 4
Response Sheet After I read you the first word from each word pair, write the second word from the pair in the appropriate blank space below. 1._______________ 7._______________ 2._______________ 8._______________ 3._______________ 9._______________ 4._______________ 10._______________ 5._______________ 11._______________ 6._______________ 12._______________ Sexual Harassment Experiment The issue of sexual harassment has become a hot political issue and, consequently, a hot area of research. Following is a set of materials that you can use to conduct a simple 2 x 2 between-subjects factorial experiment. The two independent variables are the status differential between two parties involved in an alleged harassment incident (equal [two students] or unequal [student—professor]) and the frequency of the harassment incident (once or several times). Your class should develop an informed consent form following the guidelines in Chapter 7. Reproduce enough of the materials so that you can have your students run the experiment. Have students run an equal number of male and female participants. Sex will be included as a quasi-independent variable to determine if males and females perceive incidents of alleged sexual harassment differently. After the data are collected, have your students summarize them with descriptive statistics, and perform the appropriate inferential statistical test (a between-subjects ANOVA). Have each student prepare a lab report or a full APA-style paper reporting the methods, results, and conclusions. Variations You can easily modify these materials to include a third variable (e.g., gender of the victim, the severity of the harassment incident [a professor helping a student with her schoolwork only if she agrees to go on a date with him; showing a picture of a nude painting in an art history book]). You can add or delete measures. The possibilities are limitless. References Here are a few references to get you and your students started on a literature search: Bartling, C. A., & Eisenman, R. (1993). Sexual harassment proclivities in men and women. Bulletin of the Psychonomic Society, 31, 189–192. Brusik, K. (1992). Perceptions of sexual harassment in an academic context. Sex Roles, 27, 401–412. Cummings, K. M., & Armenta, M. (2002). Penalties for peer sexual harassment in an academic context: The influence of harasser gender, participant gender, severity of harassment, and the presence of bystanders. Sex Roles, 47, 273–280. Golden, J. H., Johnson, J. C., & Lopez, R. A. (2001). Sexual harassment in the workplace: Exploring the effects of attractiveness on perception of harassment. Sex Roles, 45, 767–784. Kanekar, S., & Dhir, V. (1993). Sex-related differences in perceptions of sexual harassment of women in India. Journal of Social Psychology, 133, 119–120. Marks, M. A., & Nelson, E. S. (1993). Sexual harassment on campus: Effects of professor gender on perception of sexually harassing behaviors. Sex Roles, 28, 207–217. Runtz, M. G., & O’Donnell, C. W. (2003). Students’ perceptions of sexual harassment: Is it harassment only if the offender is a man and the victim is a woman? Journal of Applied Social Psychology, 33, 963–982. Tata, J. (1993). The structure and phenomenon of sexual harassment: Impact of category of sexually harassing behavior, gender, and hierarchical level. Journal of Applied Social Psychology, 23, 199–211. Wuensch, K. L., & Moore, C. H. (2004). Effects of Physical Attractiveness on Evaluations of a Male Employee's Allegation of Sexual Harassment by His Female Employer. Journal of Social Psychology, 144, 207–217. Instructions This experiment is designed to investigate how people perceive incidents of alleged sexual harassment. Enclosed is a brief scenario for you to read, depicting an interaction between two individuals, one of whom claims sexual harassment. After reading the scenario, you will complete several measures relating to the scenario. Some of these measures ask you to circle a number on a rating scale. For these, simply circle the number on the scale that best reflects how you feel. Please follow the instructions given for completing each measure. Are there any questions? Before we begin, please provide the following information: 1. Your gender: _____ Male _____ Female 2. Your current standing at the university: _____ First year _____ Second year ____ Third year ____ Fourth year _____ Not a university student 3. Your age 4. Your occupation Experimental Condition: Equal Status, Single Incident Sandra is a 19-year-old college sophomore. The following incident happened to her: One day after class, a fellow student offered to help her get a better grade in the class, but only if she would agree to go out on a date with him. Sandra was offended by this student’s behavior and filed a grievance against him with the University Review Board for sexual harassment. The review board agreed to hear her case. Based on this scenario, please complete the following items: 1. To what extent does the behavior of Sandra’s fellow student constitute sexual harassment? (Circle the number that best reflects your judgment.)
0 1 2 3 4 5 6 7 8 9 10
Not at all Somewhat Very much
2. How reasonable do you think Sandra’s action was in this case? (Circle the number that best reflects your judgment.)
0 1 2 3 4 5 6 7 8 9 10
Not at all reasonable Somewhat reasonable Very reasonable
3. Please describe the reason(s) why you felt that this incident did or did not constitute sexual harassment. Experimental Condition: Unequal Status, Single Incident Sandra is a 19-year-old college sophomore. The following incident happened to her: One day after class, her professor offered to help her get a better grade in the class, but only if she would agree to go out on a date with him. Sandra was offended by this professor’s behavior and filed a grievance against him with the University Review Board for sexual harassment. The review board agreed to hear her case. Based on this scenario, please complete the following items: 1. To what extent does the behavior of Sandra’s professor constitute sexual harassment? (Circle the number that best reflects your judgment.)
0 1 2 3 4 5 6 7 8 9 10
Not at all Somewhat Very much
2. How reasonable do you think Sandra’s action was in this case? (Circle the number that best reflects your judgment.)
0 1 2 3 4 5 6 7 8 9 10
Not at all reasonable Somewhat reasonable Very reasonable
3. Please describe the reason(s) why you felt that this incident did or did not constitute sexual harassment. Experimental Condition: Unequal Status, Single Incident Sandra is a 19-year-old college sophomore. The following incident happened to her: One day after class, her professor offered to help her get a better grade in the class, but only if she would agree to go out on a date with him. Sandra was offended by this professor’s behavior and filed a grievance against him with the University Review Board for sexual harassment. The review board agreed to hear her case. Based on this scenario, please complete the following items: 1. To what extent does the behavior of Sandra’s professor constitute sexual harassment? (Circle the number that best reflects your judgment.)
0 1 2 3 4 5 6 7 8 9 10
Not at all Somewhat Very much
2. How reasonable do you think Sandra’s action was in this case? (Circle the number that best reflects your judgment.)
0 1 2 3 4 5 6 7 8 9 10
Not at all reasonable Somewhat reasonable Very reasonable
3. Please describe the reason(s) why you felt that this incident did or did not constitute sexual harassment. Experimental Condition: Equal Status, Several Incidents Sandra is a 19-year-old college sophomore. The following incident happened to her: On several occasions after class, a fellow student offered to help her get a better grade in the class, but only if she would agree to go out on a date with him. Sandra was offended by this student’s behavior and filed a grievance against him with the University Review Board for sexual harassment. The review board agreed to hear her ease. Based on this scenario, please complete the following items: 1. To what extent does the behavior of Sandra’s fellow student constitute sexual harassment? (Circle the number that best reflects your judgment.)
0 1 2 3 4 5 6 7 8 9 10
Not at all Somewhat Very much
2. How reasonable do you think Sandra’s action was in this case? (Circle the number that best reflects your judgment.
0 1 2 3 4 5 6 7 8 9 10
Not at all reasonable Somewhat reasonable Very reasonable
3. Please describe the reason(s) why you felt that this incident did or did not constitute sexual harassment. Experimental Condition: Unequal Status, Several Incidents Sandra is a 19-year-old college sophomore. The following incident happened to her: On several occasions after class, her professor offered to help her get a better grade in the class, but only if she would agree to go out on a date with him. Sandra was offended by this professor’s behavior and filed a grievance against him with the University Review Board for sexual harassment. The review board agreed to hear her case. Based on this scenario, please complete the following items: 1. To what extent does the behavior of Sandra’s professor constitute sexual harassment? (Circle the number that best reflects your judgment.)
0 1 2 3 4 5 6 7 8 9 10
Not at all Somewhat Very much
2. How reasonable do you think Sandra’s action was in this case? (Circle the number that best reflects your judgment.)
0 1 2 3 4 5 6 7 8 9 10
Not at all reasonable Somewhat reasonable Very reasonable
3. Please describe the reason(s) why you felt that this incident did or did not constitute sexual harassment. Two-Factor Within-Subjects Experiment The following describes a 2 x 2 factorial within-subjects experiment patterned after the classic Peterson and Peterson (1959) study of short-term memory. In the original experiment, subjects were given a consonant trigram, immediately followed by a three-digit number. Participants were instructed to successively subtract by 3 or 4, beginning with this number, until a light came on. The onset of the light was their cue to recall the trigram aloud to the experimenter. The retention interval (delay between receipt of the trigram and light onset) was manipulated within subjects between 3 and 18 seconds in 3-second increments. We modified this experiment by including only the highest and lowest retention intervals and adding a second factor, the nature of the trigram (triple consonants or three-letter word), to form a 2 x 2 factorial manipulation. Stimulus presentation and interval timing were handled by computer in our experiment, but a simple timer or stopwatch could be used instead. Effects of Retention Interval and Stimulus Meaningfulness on Short-Term Memory for Trigrams Participants Participants can be members of the class or people recruited by the students. If class members are used, each student should serve once as participant and once as experimenter. Materials Each experimenter should have a stopwatch and four 20-item lists: two of three-letter words and two of CCCs (consonant-consonant-consonant trigrams). Each word or CCC should be paired with a three-digit number. On the same line, there should be a space for the participant’s response to the item. Participants should be tested in isolated cubicles, if possible, to minimize disturbances and prevent participants from hearing items being presented to other participants. Have your students create a set of instructions to be read to the participants and an informed consent form. Each experimenter should then be given a copy of the instructions and one copy of the consent form for each participant he or she will test. The instructions should inform the participants that the experiment is being conducted to assess their ability to remember information for brief periods of time. The participant will be given a series of stimulus items (words or consonant trigrams), each followed by a three-digit number. When the participant hears the number, he or she should repeat the number, then subtract 3 from it and give the result, then subtract 3 from the result, and so on, and continue until the experimenter says “Stop.” The participant should then attempt to recall the stimulus item given at the beginning of the trial. Stimulus Lists
1. DOG FAN DRW TVC
2. PEN ELF YPP RST
3. HAT CAT NPT RHM
4. PIG WAR LXV YJL
5. FLY NET NMF NQG
6. MAN HEN PYQ PSM
7. BAT MAP DZQ QSW
8. CAR JAR NXL DPK
9. SEA APE NMD YSP
10. HAM EGG RMK FTP
11. BOY MAT FNC KPZ
12. LET POT LTG GTS
13. SKY RAT KHP VDB
14. GUN DEN DKW XMG
15. ANT RUG LCH KJQ
16. AGE ZOO NRY KGM
17. LOG NUT YJY MCT
18. SUN PAN JRY SYH
19. BAR ARM CFS VPW
20. JET TOE DFH XVJ
Procedure The experimenter and participant should be seated facing each other and should arrange the situation so that the participant cannot see the items on the lists (for example, by placing the lists in a clipboard). One word list and one CCC list are used for the 3-second retention interval; the other word and CCC lists are used for the 18-second retention interval. This yields four treatments: Word/3-second, Word/I 8-second, CCC/3-second, and CCC/l 8-second. The order in which these treatments are presented should be counterbalanced across participants. In addition, a given list should be associated with the 3-second interval for half the participants and with the 18-second interval for the other half to prevent confounding by list difficulty. The experimenter should read the informed consent form to the participant and then have the participant sign the form. Next, the experimenter should read the instructions and ask the participant whether he or she understands what is required. If the participant is confused, the experimenter should then explain the procedure until the participant understands it. The experimenter should then make sure the participant is ready to begin. When the participant is ready, the experimenter should read the first stimulus word or CCC from the first list, enunciating clearly, followed immediately by the number that is paired with the stimulus item. As the number is read, the stopwatch should be started. Receipt of the number is the participant’s cue to start subtracting out loud by 3s from the number. The participant should continue until the experimenter says “Stop” and then should attempt to recall the stimulus item. The experimenter should record the participant’s response and indicate whether it was correct or incorrect. For one of the word lists and one of the CCC lists, the experimenter should say “Stop” when 3 seconds have elapsed on the stopwatch. For the other two lists (word and CCC), the experimenter should say “Stop” when 18 seconds have elapsed. The stopwatch should then be stopped and reset. The experimenter should continue through the 20-item list and then give the participant a short “breather” before starting the next list to reduce the effects of fatigue. Analysis The results should be analyzed with a two-factor, within-subjects ANOVA. Have students create a line graph of the group means and discuss the results in class. Designing Experiments As a way of giving students practice with designing experiments, have them design an appropriate experiment to evaluate the following research situations (source articles are listed with each situation). Students, working alone or in groups, could then bring their designs to class and present them for critique. Have students do the following for each hypothesis: 1. Identify the most appropriate experimental design (e.g., single-factor or factorial). 2. Specify precisely what the independent variables will be, and how they will be manipulated. Also, have them specify the number of levels of each independent variable. 3. Specify exactly what the dependent variable(s) will be, and how the variable(s) will be quantified. 4. Specify the procedures that will be followed (e.g., how participants will be obtained, where the experiment will be run, etc.). 5. Indicate what, if anything, would be done to control extraneous variables and eliminate confounding variables. 6. Indicate if the experiment could be run as a within-subjects or matched-groups design. If so, what would the matching variables be and why? For within-subjects designs, have students indicate what they would do to address the problems of carryover effects, participant fatigue, habituation, etc. 7. Find the article from which the hypothesis was taken, and investigate how the researchers who did the experiment designed and executed the experiment. Have students compare their designed experiments with the real thing. A. You are interested in investigating the relationship between altruism and empathy. Specifically, you want to know whether focusing on the feelings of a person in need will lead to greater feelings of empathy and more willingness to help than focusing on that person’s thoughts. Oswald, P. A. (1996). The effects of cognitive and affective perspective taking on empathic concern and altruistic behavior. Journal of Social Psychology, 136, 613–623. B. You are interested in investigating the factors that contribute to the harshness of punishment recommended for criminals. Specifically, you are interested in whether the type of crime (person versus property), the severity of the crime, and whether the defendant had committed a similar crime in the past will influence how a transgressor is perceived and judged. Sanderson, C. A., Zanna, A. S., & Darley, J. M. (2000). Making the punishment fit the crime and the criminal: Attributions of dangerousness as a mediator of liability. Journal of Applied Social Psychology, 30, 1137–1159. C. There are many factors that might influence a person to donate to a public cause. In an experiment you are contemplating, you want to know if the amount of money requested, the fairness of the request, and who receives the money affect the amount a person is willing to donate. Ajzen, I., Rosenthal, L. H., & Brown, T. (2000). Effects of perceived fairness on willingness to pay. Journal of Applied Social Psychology, 30, 2439–2450. D. The Internet has become an outlet not just for information but for social relationships as well. It is well known that being ostracized from a group has negative consequences for a person. Imagine that you are interested in studying whether the manner in which a person is treated during a social interaction on the Internet affects his or her mood or self-esteem. You design an experiment to test the idea that individuals who are ostracized in an Internet social interaction will show poorer mood and lower self-esteem than those who are accepted. Williams, K. D., Cheung, C. K. T., & Choi, W. (2000). Cyberostracism: Effects of being ignored over the Internet. Journal of Personality and Social Psychology, 79, 748–762. E. You are interested in whether changing the format of a lineup will affect the accuracy of eyewitness identification in children and adults. Specifically, you want to know if individuals perform differently on three types of lineups: a standard simultaneous lineup (the witness views all individuals in the lineup at the same time and is asked whether he or she recognizes anyone), a fast elimination lineup (the witness is told to look at the lineup and pick out the person who looks most like the suspect), and a slow elimination lineup (the witness is told to eliminate individuals from the lineup who do not look like the suspect). You are interested in the rates of correct identifications (correctly identifying the suspect), false alarms (saying the suspect is in the lineup when he or she is not), misses (saying the suspect is not in the lineup when he or she is), and correct rejections (correctly saying that the suspect is not present in the lineup). Pozzulo, 1. D., & Lindsay, R. C. L. (1999). Elimination lineups: An improved identification procedure for child eyewitnesses. Journal of Applied Psychology, 84, 167–186. CHAPTER 11 USING SPECIALIZED RESEARCH DESIGNS QUESTIONS TO PONDER 1. What is a mixed design, and when is it used? 2. What is a nested design, and when is it used? 3. What are the various types of nesting that can be done? Why would you use each? 4. When should you consider using a design combining experimental and correlational variables? 5. What is a covariate, and when would you use one? 6. What is a quasi-independent variable, and when would you use one? 7. What are the advantages and disadvantages of including a quasi-independent variable in your research? 8. What are the characteristics of the time series and equivalent time samples designs? 9. What are the advantages and disadvantages of quasi-experimental designs? 10. How are problems of internal validity addressed in quasi-experimental designs? 11. What is a nonequivalent control group design, and when would you use one? 12. What are the defining characteristics of the pretest–posttest design, and what are the design’s strengths and weaknesses? 13. What is the Solomon four-group design, and why would you consider using it? 14. What are the defining qualities of the cross-sectional developmental design? 15. What are the advantages and disadvantages of the cross-sectional developmental design? 16. What are the defining qualities of the longitudinal developmental design? 17. What are the advantages and disadvantages of the longitudinal developmental design? 18. What is a cohort-sequential design, and when would you use one? 19. What are the advantages and disadvantages of the cohort-sequential developmental design? CHAPTER OUTLINE Combining Between-Subjects and Within-Subjects Designs The Mixed Design The Nested Design Combining Experimental and Correlational Designs Including a Covariate in Your Experimental Design Including Quasi-Independent Variables in an Experiment An Example of a Combined Design: Is Coffee a Physical or Psychological Stimulant? Quasi-Experimental Designs Time Series Designs Equivalent Time Samples Design Advantages and Disadvantages of Quasi Experiments Nonequivalent Control Group Design Pretest–Posttest Designs Problems With the Pretest–Posttest Design The Solomon Four-Group Design Eliminating the Pretest Developmental Designs The Cross-Sectional Design The Longitudinal Design The Cohort-Sequential Design Summary Key Terms
mixed design nested design covariate quasi-independent variable quasi-experimental design time series design interrupted time series design equivalent time samples design nonequivalent control group design pretest–posttest design Solomon four-group design cross-sectional design longitudinal design cohort-sequential design
CHAPTER GOALS This is the last of the chapters focusing on conventional experimental designs. Chapter 11 introduces students to a variety of specialized designs including mixed designs, covariate designs, quasi-experimental designs, pretest–posttest designs, and developmental designs. These designs are used when a research question cannot be adequately addressed with a straightforward experimental or correlational design. When covering the combined experimental/correlational designs, you should emphasize the fact that the correlational variable in such a design cannot be interpreted in the same way as the experimental variable (a causal role can be deduced only for the experimental variable). Because experimental and quasi-experimental variables look the same to ANOVA, even professional researchers sometimes fail to keep the distinction in mind, but it is important that they (and your students) do so. Major points to cover in your lectures might include the following: 1. What a mixed design is and when to use it. 2. What a nested design is, how nesting can be done, and when the design is used. 3. The difference between a mixed and a nested design. 4. How to combine experimental and correlational designs. 5. Including a covariate in an experimental design to statistically control the effects of extraneous variables. 6. How to interpret results relating to the experimental and correlational variables in the design combining experimental and correlational components. 7. The definition of a quasi-experimental variable. 8. The advantages and disadvantages of including quasi-independent variables in a design. 9. The defining characteristics of the time series and interrupted time series designs. 10. The defining characteristics of the equivalent time samples design. 11. The advantages and disadvantages of quasi-experimental designs. 12. What a nonequivalent control group design is used for. 13. What pretest–posttest designs are, what they are for, and how to deal with potential problems created by the pretest. 14. The characteristics of the cross-sectional developmental design along with its advantages and disadvantages. 15. The characteristics of the longitudinal developmental design along with its advantages and disadvantages. 16. The characteristics of the cohort-sequential developmental design along with its advantages and disadvantages IDEAS FOR CLASS ACTIVITIES Mixed Experiment The classic mixed experiment combines a between-subjects variable with a repeated measure across trials. In this experiment, we examine the effects of Memorization Technique and Number of Trials on recall of a list of words. Recall is assessed on each trial by using the paired-associates technique. Subjects Subjects should be randomly assigned to one of two groups, preferably in equal numbers. Materials If you have access to a memory drum or computer, you can use these devices to present the stimuli at a uniform rate. However, a simple watch that provides the seconds and 3-inch x 5-inch cards will perform satisfactorily. Print the stimulus word on one side of the card and both the stimulus word and the response word on the other side, separated by a dash. For the computer or memory drum, present the stimulus word, then the paired stimulus-response words at an even rate (see Procedure). Here is a list of 20 paired associates to use:
1. DOG—TREE 11. HAMMER-CARROT
2. POTATO—SHOE 12. PUMPKIN—HOUSE
3. BALLOON—PHONE 13. FLOWER—HAT
4. CLOCK—COMPUTER 14. WAGON—LAMP
5. ONION—LIPSTICK 15. STRING—PAPER
6. DESK—ROCK 16. GREASE—TOOTHBRUSH
7. BOOK—SHELF 17. BASEBALL—RACQUET
8. TRUCK—MILK 18. CHICKEN—BRIEFCASE
9. BAG—BOITLE 19. TROMBONE—HOSE
10. DRUM—PERFUME 20. SOAP—GLASSES
Procedure The between-subjects variable, Memorization Technique, has two levels corresponding to the two groups: Visualization and Repetition. Participants in the Visualization group receive the following instructions: In a moment, you will be shown a series of stimulus words. Each stimulus word will be paired with a response word. Try to form a vivid and silly visual image that incorporates both words. For example, if the two words were Horse—Man, you might picture a horse riding a man, complete with saddle. After you have seen all the stimulus-response pairs, you will then be shown the stimulus words one at a time. As you see each word, try to recall your visual image. Then tell your experimenter what you think the response word is. If you do not know, take a guess. When you have made your guess, the experimenter will show you the stimulus-response pair. Check to see whether your response was correct, and then form your visual image again. The instructions for the Repetition group are as follows: In a moment you will be shown a series of stimulus words. Each stimulus word will be paired with a response word. Repeat the stimulus-response pair to yourself until the experimenter removes the card. After you have seen all the stimulus-response pairs, you will then be shown the stimulus words one at a time. As you see each stimulus word, try to recall the response word. Then tell your experimenter what you think the response word is. If you do not know, take a guess. When you have made your guess, the experimenter will show you the stimulus-response pair. Check to see whether your response was correct, and then repeat the stimulus-response pair to yourself until the experimenter removes the card. Read the appropriate instructions to the participant, and then ask the participant if he or she understands what is expected. (If the participant is confused, explain the procedure until he or she knows what to do). Then show each stimulus-response pair to the participant at a rate of one every 10 seconds. This completes the first trial. On the second and subsequent trials, present the stimulus word, and wait until the participant gives a response or 10 seconds elapse. Record whether the response to that stimulus word was correct (if no response was given, record it as incorrect), and then show the participant the stimulus-response pair for 5 seconds. Then go on to the next stimulus word. Repeat this procedure for all 20 pairs. Then start over again, and repeat until you have completed five test trials (or six trials in all). Analysis Tally the number of correct responses on each trial for each group. Then use LabStat or another statistical analysis package to create a data file, and conduct a two-factor mixed ANOVA on the data. Create a graph with Trials on the x-axis and Number Correct on the y-axis, and then plot the number correct for each group on each test trial (use circles for the Visualization group, squares for the Repetition group). Combined Design Experiment A simple between-subjects experiment can be converted to one that includes a quasi-independent variable simply by separating the subjects according to some subject characteristic. If your class includes a relatively even mix of males and females, you might consider using gender as the quasi-independent variable (failing to find a gender difference may be as interesting as finding one, especially if the dependent variable is associated with sexual stereotyping). Use the appropriate two-factor ANOVA to analyze the data, and then give students an opportunity to interpret the results. This can be a good time to reinforce the inappropriateness of attributing a causal role to the quasi-independent variable. Developmental Research Although it is not usually feasible to have students conduct a developmental study in a Research Methods class, you can still reinforce the points made in the text by having each student find a journal article illustrating each of the three developmental designs. Have the students analyze each article according to the guidelines provided in Chapter. Have students pay special attention to the aspects of the study relevant to developmental designs (e.g., generation effects, subject mortality). Discuss with the class two or three representative articles, and show how each design was employed to evaluate age-related changes in behavior. Instructor Manual for Research Design and Methods: A Process Approach Kenneth Bordens, Bruce Barrington Abbott 9780078035456
