Counterbalancing on MTurk

Back

Counterbalancing on MTurk

Author	Message
darbyh	darbyh posted 7 Years Ago Topic Details Share Topic
Group: Forum Members Posts: 22, Visits: 77	Hi! I'm running a study using MTurk, and I'm wondering if there's a way to counterbalance subjects across four conditions on Inquisit. I'm using batches for each condition and have /groupassignment set to random for each batch, but I think this just randomizes, not counterbalances the subjects into conditions. I've read through other posts, but haven't been able to find a solution for this. Is this because it can't be done through Inquisit and must happen through MTurk? Thank you!
	Reply Like 64
Dave	Dave posted 7 Years Ago ANSWER Post Details Share Post
Group: Administrators Posts: 12K, Visits: 98K	+x darbyh - Thursday, March 2, 2017 Hi! I'm running a study using MTurk, and I'm wondering if there's a way to counterbalance subjects across four conditions on Inquisit. I'm using batches for each condition and have /groupassignment set to random for each batch, but I think this just randomizes, not counterbalances the subjects into conditions. I've read through other posts, but haven't been able to find a solution for this. Is this because it can't be done through Inquisit and must happen through MTurk? Thank you! /groupassignment = random will indeed randomly (think: coin flip) determine to which condition a given participant is assigned. To get better balance across conditions, you can set / groupassignment = groupnumber and either (1) configure the web experiment to generate sequential group numbers, or (2) for even more control read the group number from a query parameter supplied to the launch page via MTurk. In the latter case, you would essentially give out four different links (one per condition), and distribute each link to a quarter of participants (how exactly you do that would be up to you -- I don't know whether MTurk offers an easy way to achieve this, some online survey platforms however do). The four links would only differ in the numerical value of the query parameter, i.e. http://research.millisecond.com/yourusername/yourexperimentname.web?groupid=1 for condition #1, http://research.millisecond.com/yourusername/yourexperimentname.web?groupid=2 for condition #2, http://research.millisecond.com/yourusername/yourexperimentname.web?groupid=3 for condition #3, and http://research.millisecond.com/yourusername/yourexperimentname.web?groupid=4 for condition #4. Also see https://www.millisecond.com/support/docs/v5/html/howto/interopsurveys.htm , which covers the use of query parameters for sharing subject identifiers, but it works the same for group numbers.
	Reply Like 68
darbyh	darbyh posted 7 Years Ago ANSWER Post Details Share Post
Group: Forum Members Posts: 22, Visits: 77	+x Dave - Thursday, March 2, 2017 +x darbyh - Thursday, March 2, 2017 Hi! I'm running a study using MTurk, and I'm wondering if there's a way to counterbalance subjects across four conditions on Inquisit. I'm using batches for each condition and have /groupassignment set to random for each batch, but I think this just randomizes, not counterbalances the subjects into conditions. I've read through other posts, but haven't been able to find a solution for this. Is this because it can't be done through Inquisit and must happen through MTurk? Thank you! /groupassignment = random will indeed randomly (think: coin flip) determine to which condition a given participant is assigned. To get better balance across conditions, you can set / groupassignment = groupnumber and (1) configure the web experiment to generate sequential group numbers Hi Dave, Thank you so much, as usual. In the first option you gave, would I program the Inquisit script to generate sequential group numbers 1-4?
	Reply Like 83
Dave	Dave posted 7 Years Ago ANSWER Post Details Share Post
Group: Administrators Posts: 12K, Visits: 98K	+x darbyh - Friday, March 3, 2017 +x Dave - Thursday, March 2, 2017 +x darbyh - Thursday, March 2, 2017 Hi! I'm running a study using MTurk, and I'm wondering if there's a way to counterbalance subjects across four conditions on Inquisit. I'm using batches for each condition and have /groupassignment set to random for each batch, but I think this just randomizes, not counterbalances the subjects into conditions. I've read through other posts, but haven't been able to find a solution for this. Is this because it can't be done through Inquisit and must happen through MTurk? Thank you! /groupassignment = random will indeed randomly (think: coin flip) determine to which condition a given participant is assigned. To get better balance across conditions, you can set / groupassignment = groupnumber and (1) configure the web experiment to generate sequential group numbers Hi Dave, Thank you so much, as usual. In the first option you gave, would I program the Inquisit script to generate sequential group numbers 1-4? > In the first option you gave, would I program the Inquisit script to generate sequential group numbers 1-4? No, that's not something you do in the script. It's a simple setting in the web experiment's settings: Every time a participant visits the launch page, the page will give her/him a sequential groupnumber i.e. 1 for the 1st visitor, 2 for the 2nd visitor, 3 for the 3rd, 4 for the 4th, 5 for the 5th and so forth. Per https://www.millisecond.com/forums/Topic13856.aspx the person with id 1 gets the 1st condition, id 2 gets the 2nd condition, id 3 gets the 3rd condition, 4 gets the 4th condition, and then again 5 gets the 1st condition, id 6 gets the 2nd condition, and so forth. I.e. the condition assignment is groupnumber modulo the number of conditions. Attachments Untitled.jpg (523 views, 64.00 KB)
	Reply Like 70
darbyh	darbyh posted 7 Years Ago ANSWER Post Details Share Post
Group: Forum Members Posts: 22, Visits: 77	+x Dave - Friday, March 3, 2017 +x darbyh - Friday, March 3, 2017 +x Dave - Thursday, March 2, 2017 +x darbyh - Thursday, March 2, 2017 Hi! I'm running a study using MTurk, and I'm wondering if there's a way to counterbalance subjects across four conditions on Inquisit. I'm using batches for each condition and have /groupassignment set to random for each batch, but I think this just randomizes, not counterbalances the subjects into conditions. I've read through other posts, but haven't been able to find a solution for this. Is this because it can't be done through Inquisit and must happen through MTurk? Thank you! /groupassignment = random will indeed randomly (think: coin flip) determine to which condition a given participant is assigned. To get better balance across conditions, you can set / groupassignment = groupnumber and (1) configure the web experiment to generate sequential group numbers Hi Dave, Thank you so much, as usual. In the first option you gave, would I program the Inquisit script to generate sequential group numbers 1-4? > In the first option you gave, would I program the Inquisit script to generate sequential group numbers 1-4? No, that's not something you do in the script. It's a simple setting in the web experiment's settings: Every time a participant visits the launch page, the page will give her/him a sequential groupnumber i.e. 1 for the 1st visitor, 2 for the 2nd visitor, 3 for the 3rd, 4 for the 4th, 5 for the 5th and so forth. Per https://www.millisecond.com/forums/Topic13856.aspx the person with id 1 gets the 1st condition, id 2 gets the 2nd condition, id 3 gets the 3rd condition, 4 gets the 4th condition, and then again 5 gets the 1st condition, id 6 gets the 2nd condition, and so forth. I.e. the condition assignment is groupnumber modulo the number of conditions. Ohhh great. Thank you!
	Reply Like 67