By AKrishna - 3/23/2020
Hello all and I hope everyone is doing ok in the current troubled times!
In the spirit of social distancing, we recently ran an online study (called manikinliking2 for admin's reference) in which I noted some odd behavior that I haven't seen in the past. We used Prolific as a data collection platform and this time, there were around 12 participants (about 5% of the total sample) who submitted the study as completed on Prolific, stated that they had entered their Prolific ID during the experiment and accurately described details of the experimental procedure (proving that they took part in the study), yet who had no associated datasets. I've not noted this happening before, but it is possible that in previous studies, participants didn't message me to tell me.
I checked the web script logging for this experiment. Unfortunately, I'm not sure how to interpret the logging information, but the number of errors it displays seems far too low to cover all these missing participants. So the questions arise: was their data not logged and if so, why not? If their data was logged, why is it not showing up in our data files? Could this be an issue of delay? I downloaded the data files quite soon after the experiment finished - is there lag between data being saved on the server and that data being available to download for me?
On a possibly related note, a participant was kind enough to screenshot an error message they received. This participant did not complete the experiment as the script wouldn't start at all for them, but in case it helps diagnostics, here's the link:
https://i.ibb.co/6XLc06R/Capture.jpg
Thanks and stay safe, everyone!
|
By Dave - 3/23/2020
+xHello all and I hope everyone is doing ok in the current troubled times! In the spirit of social distancing, we recently ran an online study (called manikinliking2 for admin's reference) in which I noted some odd behavior that I haven't seen in the past. We used Prolific as a data collection platform and this time, there were around 12 participants (about 5% of the total sample) who submitted the study as completed on Prolific, stated that they had entered their Prolific ID during the experiment and accurately described details of the experimental procedure (proving that they took part in the study), yet who had no associated datasets. I've not noted this happening before, but it is possible that in previous studies, participants didn't message me to tell me. I checked the web script logging for this experiment. Unfortunately, I'm not sure how to interpret the logging information, but the number of errors it displays seems far too low to cover all these missing participants. So the questions arise: was their data not logged and if so, why not? If their data was logged, why is it not showing up in our data files? Could this be an issue of delay? I downloaded the data files quite soon after the experiment finished - is there lag between data being saved on the server and that data being available to download for me? On a possibly related note, a participant was kind enough to screenshot an error message they received. This participant did not complete the experiment as the script wouldn't start at all for them, but in case it helps diagnostics, here's the link: https://i.ibb.co/6XLc06R/Capture.jpgThanks and stay safe, everyone!
A delay in uploading would only occur if a participant were using an iOS device and put the experiment in question in offline mode. This seems unlikely.
More likely is that the data upload attempts simply failed, e.g. if a firewall were blocking the client connecting to the server. At the end of data collection on the client device, Inquisit will make multiple attempts to upload the data, and if these all fail, as a last resort will prompt the participant to save the data locally and contact the researchers. It's possible that participants overlooked this prompt / clicked it away without reading / noticing what it said.
The screenshot / error message you attached seems related but different. When executing an experiment, the Inquisit player checks whether the script in question is properly registered with the server, i.e. if there's a valid license. Here, too, if something on the client system or the client's network is blocking the Inquisit player from doing that, the license verification will fail and the experiment will not be executed.
I'll be happy to take a look a the logs to see if I spot anything else. For that, though, I'd have to know what to look for, so I'd need e.g. the IDs of the missing participants (if known), the approximate time during which they completed the experiment, and any other information you might have available (e.g. type of device used).
|
By AKrishna - 3/26/2020
I'm afraid I don't have any data at all from these individuals, so IDs are a no-go, as are devices. I'd give you the rough time, but I can't even tell when the participants actually took part and all of the participants were quite close to one another, so would that even help? I understand that working from no data is a bit of a tall order, so if you can't find anything, that's fine.
|
By Dave - 3/26/2020
+xI'm afraid I don't have any data at all from these individuals, so IDs are a no-go, as are devices. I'd give you the rough time, but I can't even tell when the participants actually took part and all of the participants were quite close to one another, so would that even help? I understand that working from no data is a bit of a tall order, so if you can't find anything, that's fine.
I had hoped you used their Prolific IDs as Inquisit subject IDs, or in the alternative passed on the Inquisit-generated subject ID to Prolific after completion. Then we would have something to go on at least, as there should be *some* trace of those participants in the logs. Without any of that, though, figuring out what may have happened for some or all of those participants unfortunately seems like a hopeless endeavour to me.
In the absence of more specific information, data upload failure and participants not noticing the "save data locally" prompt remains one of the likelier explanations.
|
|