Testing for pairing order effects in IAT?


Author
Message
Kaarinen
Kaarinen
Esteemed Member (2.5K reputation)Esteemed Member (2.5K reputation)Esteemed Member (2.5K reputation)Esteemed Member (2.5K reputation)Esteemed Member (2.5K reputation)Esteemed Member (2.5K reputation)Esteemed Member (2.5K reputation)Esteemed Member (2.5K reputation)Esteemed Member (2.5K reputation)
Group: Forum Members
Posts: 15, Visits: 1

Hi,


This question doesn't concern just Inquisit but maybe someone would still like  to comment on it. My question concerns a ST-IAT, but I guess there's no diffence to a normal one. 


Since IAT is somewhat sensitive to which pairing is made first, Inquisit presents them in different order to every other subject, so the first one gets flowers paired with good and the second participant gets insects with good and the like. Right?


In my data which I've gathered with the demo (yes, I will buy the license as soon as I get the money), it seems there's a huge difference between the groups that got first paired with A or B. A few questions:


1) I checked this for the subjects by looking at the first block after attribute practices in my ST-IAT. So there's TargetAleft for the first subject, but TargetARight for the second etc. Am I interpreting this correctly?


2) What's the correct way of estimating the size of this effect? I was told that correlating does not work in this case, so should I just check the means and perform ANOVA for significance of these, or?



Thanks and best (apologies for asking simple things again, but i'm just making sure),


Kaarinen


Dave
Dave
Supreme Being (974K reputation)Supreme Being (974K reputation)Supreme Being (974K reputation)Supreme Being (974K reputation)Supreme Being (974K reputation)Supreme Being (974K reputation)Supreme Being (974K reputation)Supreme Being (974K reputation)Supreme Being (974K reputation)
Group: Administrators
Posts: 11K, Visits: 76K

Since IAT is somewhat sensitive to which pairing is made first, Inquisit presents them in different order to every other subject, so the first one gets flowers paired with good and the second participant gets insects with good and the like. Right?


As far as I can tell without looking at the actual script, this is correct.


In my data which I've gathered with the demo (yes, I will buy the license as soon as I get the money), it seems there's a huge difference between the groups that got first paired with A or B. A few questions:


1) I checked this for the subjects by looking at the first block after attribute practices in my ST-IAT. So there's TargetAleft for the first subject, but TargetARight for the second etc. Am I interpreting this correctly?


Hmm, not sure I understand the question. I would expect to see a notable difference when looking at a single block, since the block *is* different for the two subject groups. As you correctly noted, group A responded to a flower-good/insect-bad pairng while group B responded to the opposite insect-good/flower-bad combination in said block.


2) What's the correct way of estimating the size of this effect? I was told that correlating does not work in this case, so should I just check the means and perform ANOVA for significance of these, or?


Here's my take on this: In terms of ANOVA, 'pairing order' is just another two-level between-subjects factor. So, to test for any effects of 'pairing order', you  include it in your ANOVA and see if it produces any significant main or interaction effects.


~Dave


P.S.: I can't believe I've wasted my seminal, 500th forum post to an IAT-related question - bummer...:-)


Kaarinen
Kaarinen
Esteemed Member (2.5K reputation)Esteemed Member (2.5K reputation)Esteemed Member (2.5K reputation)Esteemed Member (2.5K reputation)Esteemed Member (2.5K reputation)Esteemed Member (2.5K reputation)Esteemed Member (2.5K reputation)Esteemed Member (2.5K reputation)Esteemed Member (2.5K reputation)
Group: Forum Members
Posts: 15, Visits: 1

Hi,


With question 1 I only meant that this is how I checked which pairing was done first in which case, but it seems to me now that Inquisit's just deciding the pairing order based on the subject ID. So with even numbered IDs the target's on the right first (after attribute practice), with odd numbered on the left. The reason I'm interested in this is because my data seems to show a huge difference between the two and I'm kinda puzzled over it. I just wanted to make sure I'm looking at the right thing.


Sorry about dragging you into IAT territory again ;).



Best,


Kaarinen


Keith Leavitt
Keith Leavitt
Esteemed Member (2K reputation)Esteemed Member (2K reputation)Esteemed Member (2K reputation)Esteemed Member (2K reputation)Esteemed Member (2K reputation)Esteemed Member (2K reputation)Esteemed Member (2K reputation)Esteemed Member (2K reputation)Esteemed Member (2K reputation)
Group: Forum Members
Posts: 27, Visits: 1

Hi Kaarinen,


  There are relaible order effects for the pairing in the IAT, which are well known and doccumented. However, the good news is that the regression coefficients don't seem to change. In other words, presenting the "consistent" blocks first generally produces a smaller "D" score, but does so in a systematic way, where subject ordering and relative effect size does not change. You should look on Tony Greenwald's website--there's a paper addressing this, but I don't remember exactly what the cite is.


Kaarinen
Kaarinen
Esteemed Member (2.5K reputation)Esteemed Member (2.5K reputation)Esteemed Member (2.5K reputation)Esteemed Member (2.5K reputation)Esteemed Member (2.5K reputation)Esteemed Member (2.5K reputation)Esteemed Member (2.5K reputation)Esteemed Member (2.5K reputation)Esteemed Member (2.5K reputation)
Group: Forum Members
Posts: 15, Visits: 1

Hi Keith,



Thanks for the reply. I'll try to find the paper you suggested. In my own, slightly unconventional ST-IAT study, I got a significant effect of -0.11 which was a nice finding since it supported the hypothesis  I had. However, upon closer inspection, I realised that the two different pairing conditions* had scores of -.25** and .02 (not significant). I checked the significance of the difference in means with ANOVA and a independent samples t-test, and it is significant at .001 level.


This kinda makes the result difficult to interpret, so I should search for information on this...



Best,


Kaarinen



*Whether the subject got the target paired with attribute A or B first. Here, this means every other participant since Inquisit makes this based on the subject ID (if I understood correctly).




Keith Leavitt
Keith Leavitt
Esteemed Member (2K reputation)Esteemed Member (2K reputation)Esteemed Member (2K reputation)Esteemed Member (2K reputation)Esteemed Member (2K reputation)Esteemed Member (2K reputation)Esteemed Member (2K reputation)Esteemed Member (2K reputation)Esteemed Member (2K reputation)
Group: Forum Members
Posts: 27, Visits: 1

It might be an issue with your "neutral" stimuli. As example, in the "attitude toward coworkers" IAT I did as part of my dissertation, my initial results were aweful (i was using "family members" as the "not coworkers" category, which ended up driving the task). Once I switched it to "other people at the same organization", it got much better). There are other folks out there with some interesting hard-learned lessons on IAT use (I've heard a lot of them). If you want to shoot me an email and describe your task, I might have some insights.



 keith.leavitt@usma.edu  



GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Reading This Topic

Explore
Messages
Mentions
Search