Problems uploading files to galaxy
Hi, I'm hoping you can help out with a problem we are experiencing uploading files to our own Galaxy instance. When a user tries to upload a file to Galaxy through their web browser, the job errors out: Traceback (most recent call last): File "/gpfs/fs1/sfw/cibem_galaxy/app/galaxy/tools/data_source/upload.py", line 430, in <module> __main__() File "/gpfs/fs1/sfw/cibem_galaxy/app/galaxy/tools/data_source/upload.py", line 407, in __main__ for line in open( sys.argv[3], 'r' ): IOError: [Errno 13] Permission denied: '/scratch/XXXXXX/XXXXXX/galaxy/tmp/tmpSlG2pC' The permissions on the referenced file: -rw-------+ 1 rlemley rlemley 317 Mar 7 13:42 tmpSlG2pC As the Galaxy daemons run as XXXXXX_galaxy, it does not have permission to read the file. I tried modifying external_chown_script.py so that XXXXXX_galaxy has permission to read that file, but that caused problems elsewhere so I backed out the changes. Any suggestions? Many thanks, -Rob Rob Lemley Systems Administrator - CIRC University of Rochester
Hi everyone, I didn’t see any response to my email below. Does anyone have any thoughts? I'm pretty stuck. Thanks again, -Rob Rob Lemley Systems Administrator – CIRC University of Rochester -----Original Message----- From: galaxy-dev [mailto:galaxy-dev-bounces@lists.galaxyproject.org] On Behalf Of Lemley, Rob Sent: Monday, March 7, 2016 2:03 PM To: galaxy-dev@lists.galaxyproject.org Subject: [galaxy-dev] Problems uploading files to galaxy Hi, I'm hoping you can help out with a problem we are experiencing uploading files to our own Galaxy instance. When a user tries to upload a file to Galaxy through their web browser, the job errors out: Traceback (most recent call last): File "/gpfs/fs1/sfw/cibem_galaxy/app/galaxy/tools/data_source/upload.py", line 430, in <module> __main__() File "/gpfs/fs1/sfw/cibem_galaxy/app/galaxy/tools/data_source/upload.py", line 407, in __main__ for line in open( sys.argv[3], 'r' ): IOError: [Errno 13] Permission denied: '/scratch/XXXXXX/XXXXXX/galaxy/tmp/tmpSlG2pC' The permissions on the referenced file: -rw-------+ 1 rlemley rlemley 317 Mar 7 13:42 tmpSlG2pC As the Galaxy daemons run as XXXXXX_galaxy, it does not have permission to read the file. I tried modifying external_chown_script.py so that XXXXXX_galaxy has permission to read that file, but that caused problems elsewhere so I backed out the changes. Any suggestions? Many thanks, -Rob Rob Lemley Systems Administrator - CIRC University of Rochester ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.galaxyproject.org_&d=BQIGaQ&c=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU&r=d-Zn2Sbc-XheRlOYkSFXPupZ8ZmuYq52isQBiI3MBWA&m=xTXPxT_d1oq2TJoB3W7P0xfhV52TZzeUjoGhvts8xNU&s=yW1eNcfXnR8Rut3-T62JBQMRJXblVpOZEA0w7DxiJ04&e= To search Galaxy mailing lists use the unified search at: https://urldefense.proofpoint.com/v2/url?u=http-3A__galaxyproject.org_search_mailinglists_&d=BQIGaQ&c=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU&r=d-Zn2Sbc-XheRlOYkSFXPupZ8ZmuYq52isQBiI3MBWA&m=xTXPxT_d1oq2TJoB3W7P0xfhV52TZzeUjoGhvts8xNU&s=sQNOB73SDBa-fat4oHkeYucMCsvheL3eXDp3qoUndLU&e=
If you are running jobs as "the real user" with drmaa (using the chown script) - you will probably want to setup a separate job destination using the local runner for upload jobs. The upload tool uses files that are created outside of Galaxy and these aren't modeled well. Does this help? -John On Fri, Mar 11, 2016 at 6:45 PM, Lemley, Rob <rob.lemley@rochester.edu> wrote:
Hi everyone,
I didn’t see any response to my email below. Does anyone have any thoughts? I'm pretty stuck.
Thanks again,
-Rob
Rob Lemley Systems Administrator – CIRC University of Rochester
-----Original Message----- From: galaxy-dev [mailto:galaxy-dev-bounces@lists.galaxyproject.org] On Behalf Of Lemley, Rob Sent: Monday, March 7, 2016 2:03 PM To: galaxy-dev@lists.galaxyproject.org Subject: [galaxy-dev] Problems uploading files to galaxy
Hi,
I'm hoping you can help out with a problem we are experiencing uploading files to our own Galaxy instance.
When a user tries to upload a file to Galaxy through their web browser, the job errors out:
Traceback (most recent call last): File "/gpfs/fs1/sfw/cibem_galaxy/app/galaxy/tools/data_source/upload.py", line 430, in <module> __main__() File "/gpfs/fs1/sfw/cibem_galaxy/app/galaxy/tools/data_source/upload.py", line 407, in __main__ for line in open( sys.argv[3], 'r' ): IOError: [Errno 13] Permission denied: '/scratch/XXXXXX/XXXXXX/galaxy/tmp/tmpSlG2pC'
The permissions on the referenced file: -rw-------+ 1 rlemley rlemley 317 Mar 7 13:42 tmpSlG2pC
As the Galaxy daemons run as XXXXXX_galaxy, it does not have permission to read the file.
I tried modifying external_chown_script.py so that XXXXXX_galaxy has permission to read that file, but that caused problems elsewhere so I backed out the changes.
Any suggestions?
Many thanks,
-Rob
Rob Lemley Systems Administrator - CIRC University of Rochester
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.galaxyproject.org_&d=BQIGaQ&c=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU&r=d-Zn2Sbc-XheRlOYkSFXPupZ8ZmuYq52isQBiI3MBWA&m=xTXPxT_d1oq2TJoB3W7P0xfhV52TZzeUjoGhvts8xNU&s=yW1eNcfXnR8Rut3-T62JBQMRJXblVpOZEA0w7DxiJ04&e=
To search Galaxy mailing lists use the unified search at: https://urldefense.proofpoint.com/v2/url?u=http-3A__galaxyproject.org_search_mailinglists_&d=BQIGaQ&c=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU&r=d-Zn2Sbc-XheRlOYkSFXPupZ8ZmuYq52isQBiI3MBWA&m=xTXPxT_d1oq2TJoB3W7P0xfhV52TZzeUjoGhvts8xNU&s=sQNOB73SDBa-fat4oHkeYucMCsvheL3eXDp3qoUndLU&e= ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
On Tue, Mar 15, 2016 at 2:21 PM, John Chilton <jmchilton@gmail.com> wrote:
If you are running jobs as "the real user" with drmaa (using the chown script) - you will probably want to setup a separate job destination using the local runner for upload jobs. The upload tool uses files that are created outside of Galaxy and these aren't modeled well.
Does this help?
-John
We found the opposite - if using "the real user", the Galaxy v16.01 would do a chown on the datafiles for DRMAA jobs (good) but also for local jobs (bad): http://dev.list.galaxyproject.org/external-chown-script-py-and-the-local-job... I will reply on that thread now with an update... Peter
Thanks Peter, that's really helpful. I'm glad we're not the only ones running into this. -Rob Rob Lemley Systems Administrator – CIRC University of Rochester -----Original Message----- From: Peter Cock [mailto:p.j.a.cock@googlemail.com] Sent: Tuesday, March 15, 2016 10:34 AM To: John Chilton <jmchilton@gmail.com> Cc: Lemley, Rob <rob.lemley@rochester.edu>; galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] Problems uploading files to galaxy On Tue, Mar 15, 2016 at 2:21 PM, John Chilton <jmchilton@gmail.com> wrote:
If you are running jobs as "the real user" with drmaa (using the chown script) - you will probably want to setup a separate job destination using the local runner for upload jobs. The upload tool uses files that are created outside of Galaxy and these aren't modeled well.
Does this help?
-John
We found the opposite - if using "the real user", the Galaxy v16.01 would do a chown on the datafiles for DRMAA jobs (good) but also for local jobs (bad): https://urldefense.proofpoint.com/v2/url?u=http-3A__dev.list.galaxyproject.org_external-2Dchown-2Dscript-2Dpy-2Dand-2Dthe-2Dlocal-2Djob-2Drunner-2Dtt4668738.html&d=BQIBaQ&c=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU&r=d-Zn2Sbc-XheRlOYkSFXPupZ8ZmuYq52isQBiI3MBWA&m=rMI9zRSmgepuzJAMtXaa8aFFhpcuYbKx2r_h3aC6wIg&s=IZ8BQgQLJhvOnBc_xzGzMYMVVETGgFUhcBM_tNlMXRE&e= I will reply on that thread now with an update... Peter
Hi John, Thanks for getting back to me. If I'm understanding the XML files correctly, the upload tool is already configured to run as a local job. From job_conf.xml: <tool id="upload1" destination="local" /> where "local" is our local job runner and "upload1" is defined in data_source/upload.xml. -Rob Rob Lemley Systems Administrator – CIRC University of Rochester -----Original Message----- From: John Chilton [mailto:jmchilton@gmail.com] Sent: Tuesday, March 15, 2016 10:21 AM To: Lemley, Rob <rob.lemley@rochester.edu> Cc: galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] Problems uploading files to galaxy If you are running jobs as "the real user" with drmaa (using the chown script) - you will probably want to setup a separate job destination using the local runner for upload jobs. The upload tool uses files that are created outside of Galaxy and these aren't modeled well. Does this help? -John On Fri, Mar 11, 2016 at 6:45 PM, Lemley, Rob <rob.lemley@rochester.edu> wrote:
Hi everyone,
I didn’t see any response to my email below. Does anyone have any thoughts? I'm pretty stuck.
Thanks again,
-Rob
Rob Lemley Systems Administrator – CIRC University of Rochester
-----Original Message----- From: galaxy-dev [mailto:galaxy-dev-bounces@lists.galaxyproject.org] On Behalf Of Lemley, Rob Sent: Monday, March 7, 2016 2:03 PM To: galaxy-dev@lists.galaxyproject.org Subject: [galaxy-dev] Problems uploading files to galaxy
Hi,
I'm hoping you can help out with a problem we are experiencing uploading files to our own Galaxy instance.
When a user tries to upload a file to Galaxy through their web browser, the job errors out:
Traceback (most recent call last): File "/gpfs/fs1/sfw/cibem_galaxy/app/galaxy/tools/data_source/upload.py", line 430, in <module> __main__() File "/gpfs/fs1/sfw/cibem_galaxy/app/galaxy/tools/data_source/upload.py", line 407, in __main__ for line in open( sys.argv[3], 'r' ): IOError: [Errno 13] Permission denied: '/scratch/XXXXXX/XXXXXX/galaxy/tmp/tmpSlG2pC'
The permissions on the referenced file: -rw-------+ 1 rlemley rlemley 317 Mar 7 13:42 tmpSlG2pC
As the Galaxy daemons run as XXXXXX_galaxy, it does not have permission to read the file.
I tried modifying external_chown_script.py so that XXXXXX_galaxy has permission to read that file, but that caused problems elsewhere so I backed out the changes.
Any suggestions?
Many thanks,
-Rob
Rob Lemley Systems Administrator - CIRC University of Rochester
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.galaxyproject.org_&d=BQIGaQ&c=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU&r=d-Zn2Sbc-XheRlOYkSFXPupZ8ZmuYq52isQBiI3MBWA&m=xTXPxT_d1oq2TJoB3W7P0xfhV52TZzeUjoGhvts8xNU&s=yW1eNcfXnR8Rut3-T62JBQMRJXblVpOZEA0w7DxiJ04&e=
To search Galaxy mailing lists use the unified search at: https://urldefense.proofpoint.com/v2/url?u=http-3A__galaxyproject.org_search_mailinglists_&d=BQIGaQ&c=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU&r=d-Zn2Sbc-XheRlOYkSFXPupZ8ZmuYq52isQBiI3MBWA&m=xTXPxT_d1oq2TJoB3W7P0xfhV52TZzeUjoGhvts8xNU&s=sQNOB73SDBa-fat4oHkeYucMCsvheL3eXDp3qoUndLU&e= ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.galaxyproject.org_&d=BQIFaQ&c=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU&r=d-Zn2Sbc-XheRlOYkSFXPupZ8ZmuYq52isQBiI3MBWA&m=CEkTHfFOPDidaSf14okOW1C-PgZzhlhbJVrEhanyil0&s=1Eql06w3_X1XFsEnMGdKCDlZus4pqc2vqwICDnPPNNs&e=
To search Galaxy mailing lists use the unified search at: https://urldefense.proofpoint.com/v2/url?u=http-3A__galaxyproject.org_search_mailinglists_&d=BQIFaQ&c=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU&r=d-Zn2Sbc-XheRlOYkSFXPupZ8ZmuYq52isQBiI3MBWA&m=CEkTHfFOPDidaSf14okOW1C-PgZzhlhbJVrEhanyil0&s=XiocBa1yVec6l9Im_dv0CZ93S_3tEfxU9tDAH5iHWnk&e=
participants (3)
-
John Chilton
-
Lemley, Rob
-
Peter Cock