API for Downloading


Author
Message
loganap
loganap
New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)
Group: Forum Members
Posts: 5, Visits: 13
Hi, my group routinely downloads bulk data files containing our Inquisit results from the millisecond.com online interface. I am responsible for maintaining a browser script that logs into the site and downloads items, but I've found that very minor updates to the site often break my script. It would be very helpful to have an API for more reliable access to our data files. Is there any plan to implement such an API? I found a request from a few years ago requesting this feature and I hoped it may have moved up the queue. https://www.millisecond.com/forums/Topic18845.aspx.
seandr
seandr
Supreme Being (131K reputation)Supreme Being (131K reputation)Supreme Being (131K reputation)Supreme Being (131K reputation)Supreme Being (131K reputation)Supreme Being (131K reputation)Supreme Being (131K reputation)Supreme Being (131K reputation)Supreme Being (131K reputation)
Group: Administrators
Posts: 1.3K, Visits: 1.1K
loganap - 9/20/2021
Hi, my group routinely downloads bulk data files containing our Inquisit results from the millisecond.com online interface. I am responsible for maintaining a browser script that logs into the site and downloads items, but I've found that very minor updates to the site often break my script. It would be very helpful to have an API for more reliable access to our data files. Is there any plan to implement such an API? I found a request from a few years ago requesting this feature and I hoped it may have moved up the queue. https://www.millisecond.com/forums/Topic18845.aspx.

Request noted.  Can you say a bit more about how you are processing data and what function(s) would be useful to you?

Note that we already have an API for forwarding incoming data files as they arrive to an HTTPS endpoint of your choosing. When you enable this feature, Millisecond's data service forwards a copy of each data file and metadata (via HTTP PUT or POST) to your endpoint. In this case, you only need to program your endpoint to receive and process the data files that we send to it.

As for a downloading API, we can't really do a simple GET that returns files in bulk because many thousands of large files might be involved that could take hours to zip up (or merge into an Excel or CSV) and stream for download. Another approach would be a LIST function that returns a list of files in a specified folder, and a GET function that downloads a single file. You can then iterate over the list and download the data file by file. 

If you ask me, the data forwarding approach is more elegant for moving files between servers, if that's your ultimate goal. 


loganap
loganap
New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)
Group: Forum Members
Posts: 5, Visits: 13
seandr - 9/20/2021
loganap - 9/20/2021
Hi, my group routinely downloads bulk data files containing our Inquisit results from the millisecond.com online interface. I am responsible for maintaining a browser script that logs into the site and downloads items, but I've found that very minor updates to the site often break my script. It would be very helpful to have an API for more reliable access to our data files. Is there any plan to implement such an API? I found a request from a few years ago requesting this feature and I hoped it may have moved up the queue. https://www.millisecond.com/forums/Topic18845.aspx.

Request noted.  Can you say a bit more about how you are processing data and what function(s) would be useful to you?

Note that we already have an API for forwarding incoming data files as they arrive to an HTTPS endpoint of your choosing. When you enable this feature, Millisecond's data service forwards a copy of each data file and metadata (via HTTP PUT or POST) to your endpoint. In this case, you only need to program your endpoint to receive and process the data files that we send to it.

As for a downloading API, we can't really do a simple GET that returns files in bulk because many thousands of large files might be involved that could take hours to zip up (or merge into an Excel or CSV) and stream for download. Another approach would be a LIST function that returns a list of files in a specified folder, and a GET function that downloads a single file. You can then iterate over the list and download the data file by file. 

If you ask me, the data forwarding approach is more elegant for moving files between servers, if that's your ultimate goal. 


Thank you, Sean. That might be just what I need! Can you point me to the documentation of this feature? I must have overlooked it in my searches.
loganap
loganap
New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)
Group: Forum Members
Posts: 5, Visits: 13
loganap - 9/20/2021
seandr - 9/20/2021
loganap - 9/20/2021
Hi, my group routinely downloads bulk data files containing our Inquisit results from the millisecond.com online interface. I am responsible for maintaining a browser script that logs into the site and downloads items, but I've found that very minor updates to the site often break my script. It would be very helpful to have an API for more reliable access to our data files. Is there any plan to implement such an API? I found a request from a few years ago requesting this feature and I hoped it may have moved up the queue. https://www.millisecond.com/forums/Topic18845.aspx.

Request noted.  Can you say a bit more about how you are processing data and what function(s) would be useful to you?

Note that we already have an API for forwarding incoming data files as they arrive to an HTTPS endpoint of your choosing. When you enable this feature, Millisecond's data service forwards a copy of each data file and metadata (via HTTP PUT or POST) to your endpoint. In this case, you only need to program your endpoint to receive and process the data files that we send to it.

As for a downloading API, we can't really do a simple GET that returns files in bulk because many thousands of large files might be involved that could take hours to zip up (or merge into an Excel or CSV) and stream for download. Another approach would be a LIST function that returns a list of files in a specified folder, and a GET function that downloads a single file. You can then iterate over the list and download the data file by file. 

If you ask me, the data forwarding approach is more elegant for moving files between servers, if that's your ultimate goal. 


Thank you, Sean. That might be just what I need! Can you point me to the documentation of this feature? I must have overlooked it in my searches.

Ah, found it. In case anyone needs it: https://www.millisecond.com/support/docs/v5/html/viewer.htm#articles/dataweb.htm
This can be done via the <data> tag in your experiment configuration.

Thanks again for pointing me in the right direction.
Dave
Dave
Supreme Being (968K reputation)Supreme Being (968K reputation)Supreme Being (968K reputation)Supreme Being (968K reputation)Supreme Being (968K reputation)Supreme Being (968K reputation)Supreme Being (968K reputation)Supreme Being (968K reputation)Supreme Being (968K reputation)
Group: Administrators
Posts: 11K, Visits: 74K
loganap - 9/20/2021
loganap - 9/20/2021
seandr - 9/20/2021
loganap - 9/20/2021
Hi, my group routinely downloads bulk data files containing our Inquisit results from the millisecond.com online interface. I am responsible for maintaining a browser script that logs into the site and downloads items, but I've found that very minor updates to the site often break my script. It would be very helpful to have an API for more reliable access to our data files. Is there any plan to implement such an API? I found a request from a few years ago requesting this feature and I hoped it may have moved up the queue. https://www.millisecond.com/forums/Topic18845.aspx.

Request noted.  Can you say a bit more about how you are processing data and what function(s) would be useful to you?

Note that we already have an API for forwarding incoming data files as they arrive to an HTTPS endpoint of your choosing. When you enable this feature, Millisecond's data service forwards a copy of each data file and metadata (via HTTP PUT or POST) to your endpoint. In this case, you only need to program your endpoint to receive and process the data files that we send to it.

As for a downloading API, we can't really do a simple GET that returns files in bulk because many thousands of large files might be involved that could take hours to zip up (or merge into an Excel or CSV) and stream for download. Another approach would be a LIST function that returns a list of files in a specified folder, and a GET function that downloads a single file. You can then iterate over the list and download the data file by file. 

If you ask me, the data forwarding approach is more elegant for moving files between servers, if that's your ultimate goal. 


Thank you, Sean. That might be just what I need! Can you point me to the documentation of this feature? I must have overlooked it in my searches.

Ah, found it. In case anyone needs it: https://www.millisecond.com/support/docs/v5/html/viewer.htm#articles/dataweb.htm
This can be done via the <data> tag in your experiment configuration.

Thanks again for pointing me in the right direction.

There's another, somewhat more convenient and more flexible option than using the <data> element. In your web experiment's settings, under "Advanced Settings", you can enable the data forwarding option, select the forwarding protocol (options are: Multipart Form POST, HTTP/REST POST or PUT, and AWS S3 forwarding), enter your endpoint's address, as well as the access credentials required by the endpoint (if any).



Two major advantages of this: You don't need to change anything in your scripts, they can use the existing / default <data options, and you can optionally keep a copy on the Millisecond servers as a backup.

loganap
loganap
New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)
Group: Forum Members
Posts: 5, Visits: 13
Dave - 9/20/2021
loganap - 9/20/2021
loganap - 9/20/2021
seandr - 9/20/2021
loganap - 9/20/2021
Hi, my group routinely downloads bulk data files containing our Inquisit results from the millisecond.com online interface. I am responsible for maintaining a browser script that logs into the site and downloads items, but I've found that very minor updates to the site often break my script. It would be very helpful to have an API for more reliable access to our data files. Is there any plan to implement such an API? I found a request from a few years ago requesting this feature and I hoped it may have moved up the queue. https://www.millisecond.com/forums/Topic18845.aspx.

Request noted.  Can you say a bit more about how you are processing data and what function(s) would be useful to you?

Note that we already have an API for forwarding incoming data files as they arrive to an HTTPS endpoint of your choosing. When you enable this feature, Millisecond's data service forwards a copy of each data file and metadata (via HTTP PUT or POST) to your endpoint. In this case, you only need to program your endpoint to receive and process the data files that we send to it.

As for a downloading API, we can't really do a simple GET that returns files in bulk because many thousands of large files might be involved that could take hours to zip up (or merge into an Excel or CSV) and stream for download. Another approach would be a LIST function that returns a list of files in a specified folder, and a GET function that downloads a single file. You can then iterate over the list and download the data file by file. 

If you ask me, the data forwarding approach is more elegant for moving files between servers, if that's your ultimate goal. 


Thank you, Sean. That might be just what I need! Can you point me to the documentation of this feature? I must have overlooked it in my searches.

Ah, found it. In case anyone needs it: https://www.millisecond.com/support/docs/v5/html/viewer.htm#articles/dataweb.htm
This can be done via the <data> tag in your experiment configuration.

Thanks again for pointing me in the right direction.

There's another, somewhat more convenient and more flexible option than using the <data> element. In your web experiment's settings, under "Advanced Settings", you can enable the data forwarding option, select the forwarding protocol (options are: Multipart Form POST, HTTP/REST POST or PUT, and AWS S3 forwarding), enter your endpoint's address, as well as the access credentials required by the endpoint (if any).



Two major advantages of this: You don't need to change anything in your scripts, they can use the existing / default <data options, and you can optionally keep a copy on the Millisecond servers as a backup.

Can anyone point to documentation on this feature? I'd like to configure our server to receive these requests but I'd like to see how the POST data will be sent. I didn't see it in the online docs. Thanks!

Dave
Dave
Supreme Being (968K reputation)Supreme Being (968K reputation)Supreme Being (968K reputation)Supreme Being (968K reputation)Supreme Being (968K reputation)Supreme Being (968K reputation)Supreme Being (968K reputation)Supreme Being (968K reputation)Supreme Being (968K reputation)
Group: Administrators
Posts: 11K, Visits: 74K
loganap - 9/24/2021
Dave - 9/20/2021
loganap - 9/20/2021
loganap - 9/20/2021
seandr - 9/20/2021
loganap - 9/20/2021
Hi, my group routinely downloads bulk data files containing our Inquisit results from the millisecond.com online interface. I am responsible for maintaining a browser script that logs into the site and downloads items, but I've found that very minor updates to the site often break my script. It would be very helpful to have an API for more reliable access to our data files. Is there any plan to implement such an API? I found a request from a few years ago requesting this feature and I hoped it may have moved up the queue. https://www.millisecond.com/forums/Topic18845.aspx.

Request noted.  Can you say a bit more about how you are processing data and what function(s) would be useful to you?

Note that we already have an API for forwarding incoming data files as they arrive to an HTTPS endpoint of your choosing. When you enable this feature, Millisecond's data service forwards a copy of each data file and metadata (via HTTP PUT or POST) to your endpoint. In this case, you only need to program your endpoint to receive and process the data files that we send to it.

As for a downloading API, we can't really do a simple GET that returns files in bulk because many thousands of large files might be involved that could take hours to zip up (or merge into an Excel or CSV) and stream for download. Another approach would be a LIST function that returns a list of files in a specified folder, and a GET function that downloads a single file. You can then iterate over the list and download the data file by file. 

If you ask me, the data forwarding approach is more elegant for moving files between servers, if that's your ultimate goal. 


Thank you, Sean. That might be just what I need! Can you point me to the documentation of this feature? I must have overlooked it in my searches.

Ah, found it. In case anyone needs it: https://www.millisecond.com/support/docs/v5/html/viewer.htm#articles/dataweb.htm
This can be done via the <data> tag in your experiment configuration.

Thanks again for pointing me in the right direction.

There's another, somewhat more convenient and more flexible option than using the <data> element. In your web experiment's settings, under "Advanced Settings", you can enable the data forwarding option, select the forwarding protocol (options are: Multipart Form POST, HTTP/REST POST or PUT, and AWS S3 forwarding), enter your endpoint's address, as well as the access credentials required by the endpoint (if any).



Two major advantages of this: You don't need to change anything in your scripts, they can use the existing / default <data options, and you can optionally keep a copy on the Millisecond servers as a backup.

Can anyone point to documentation on this feature? I'd like to configure our server to receive these requests but I'd like to see how the POST data will be sent. I didn't see it in the online docs. Thanks!

I'll outline it here. Five metadata fields are forwarded as query parameters to the endpoint. Those are:

ScriptPath (the script file name)
GroupId (the group number)
SubjectId (the subject ID)
SessionId (the session ID)
FileKey (the data file path & name)

The POST message's body is the actual data file.

Below screenshot should illustrate:



In essence, this is also what you'll find outlined here https://www.millisecond.com/products/inquisit6/webhosting.aspx under "Hosting Data Upload Service."

Edited Last Month by Dave
seandr
seandr
Supreme Being (131K reputation)Supreme Being (131K reputation)Supreme Being (131K reputation)Supreme Being (131K reputation)Supreme Being (131K reputation)Supreme Being (131K reputation)Supreme Being (131K reputation)Supreme Being (131K reputation)Supreme Being (131K reputation)
Group: Administrators
Posts: 1.3K, Visits: 1.1K
I've just updated the help topic on this, which was woefully out of date. It expands a bit on Dave's reply.

You can find details of various protocol options here:
https://www.millisecond.com/support/docs/v6/html/articles/dataweb.htm

loganap
loganap
New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)
Group: Forum Members
Posts: 5, Visits: 13
seandr - 9/24/2021
I've just updated the help topic on this, which was woefully out of date. It expands a bit on Dave's reply.

You can find details of various protocol options here:
https://www.millisecond.com/support/docs/v6/html/articles/dataweb.htm


Perfect, thanks

GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Reading This Topic

Explore
Messages
Mentions
Search