Missing Data


Author
Message
nashby
nashby
Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)
Group: Forum Members
Posts: 78, Visits: 159
Well there are quite a few people who are emailing me saying they completed the task, they know all the details of the task, yet there is no data file for them there. If it was just one or two  I would say its a scam, but its actually a lot of them.

The attention check goes on to a next page that then does as you suggest already (e.g., you have failed to pay attention so you cannot take part in this task). No one has complained about this and there files still show up if the control quit or are kicked out.



seandr
seandr
Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)
Group: Administrators
Posts: 1.3K, Visits: 1K
Just to be clear, unless you've updated the script, the only choice is to click the invisible rectangle. You'll need to list "proceed2" as a response for that response to be recognized.

I'll look further at the logs to see if I can find more data to make sense of the feedback you're getting from respondents.

-Sean
nashby
nashby
Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)
Group: Forum Members
Posts: 78, Visits: 159
Yes I fixed it but again it has nothing to do with the issue I am having here with the data not being uploaded.
seandr
seandr
Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)
Group: Administrators
Posts: 1.3K, Visits: 1K
I'm slogging through the servers raw hit logs, and it appears some of the participants are also running your "T2" study, which captures the MTurk ID in the link. With those participants, I can use the ip address to examine their hits with respect to the "Bundle" study to see what connection they made with the server. Would it be possible to share some/all of the MTurk IDs for respondents who say they completed the whole study but for whom no data can be found? It would also be helpful to know which date they ran the test.

If you'd rather not post them here, you can email them to me at seandr at millisecond dot com.

Thanks,
Sean
Edited 6 Years Ago by seandr
seandr
seandr
Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)
Group: Administrators
Posts: 1.3K, Visits: 1K
P.S. It appears that T2 is registered with Inquisit 4.0.6, and Bundled is registered with 4.0.7. You could save workers the trouble of downloading the Inquisit control twice by setting these to the same version.

-Sean
nashby
nashby
Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)
Group: Forum Members
Posts: 78, Visits: 159
The T2 study is done now so no more people will be taking it. Didnt have any issues there, at least none were reported to me though that one wasn't run through my MTurk account and I think they just pay everyone regardless so that might be why there were no complaints.

 Here is a subset of workers which I am most confident actually did the study but whose data I cant seem to find. In terms of dates its from Nov1st to Today.

A1FTR8DJIKGD8H
A21A93NS7IJNZM
AQLO2RDFLX3DN
A34L4JED8UAIXC
A38Z4HGE16SX10
A19ED8FYO6CA36
A2JZQHB8N6LS7T

seandr
seandr
Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)
Group: Administrators
Posts: 1.3K, Visits: 1K
Ok, of those, only A34L4JED8UAIXC also ran T2. With that participant, Inquisit was for some reason unable to reach the server to verify the license. In that case, a message appears indicating the failure and giving the participant the option of running anyway even though data wouldn't be saved, which they seem to have done, since they did eventually arrive at the finish page. That would explain how they could have run the whole test without producing a data file.

I've identified 2 other cases where the license check similarly failed, although I can't map them back to the mturk IDs, unless you happen to know their subject id or ip address (not sure how you would). There may be other cases that I can't identify because they didn't reach the finish page.

I'm still investigating why the license check failed to reach the server. In the meantime, you could try setting the version back to 4.0.6 for this script, although that's just a shot in the dark, as I'm not aware of any recent changes to the code that could explain this.

P.S. As for the finish page timestamps missing from the logs, that appears to be a bug in the report. The hits are there in the underlying data tables. 


-Sean


nashby
nashby
Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)
Group: Forum Members
Posts: 78, Visits: 159
Thanks for looking into this.

So basically something pops up and they just ignore it and click through blindly. I think that part is going to be hard to avoid but I can at least put a warning about it up to try to prevent it. Do you think if they try to install and run again it would work on the next pass? Is there alternatively anything I could add to the script that could check if this is the case then kill the session if it is?
nashby
nashby
Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)Guru (6.4K reputation)
Group: Forum Members
Posts: 78, Visits: 159
Ah and yes a few reported that they got to the end but the finish page never popped up.
seandr
seandr
Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)Supreme Being (118K reputation)
Group: Administrators
Posts: 1.3K, Visits: 1K
I don't know if trying again would work, but you could instruct participants to try, especially if they have access to another computer. 

I'm planning to update 4.0.7 to remove the option of running the study anyway given its potential to cause this type of problem. That way, if the script registration fails, the participant doesn't waste their time and your money by completing the study. I'm kind of surprised this hasn't come up before.

The earliest I'd publish that update would be tomorrow (it's 11pm Seattle time right now). 

I'm also trying to find a computer that reproduces this issue. Once we can reproduce it, we should be able to fix it.

Regards,
Sean
Edited 6 Years Ago by seandr
GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Reading This Topic

Explore
Messages
Mentions
Search