Re: [galaxy-dev] Galaxy at Duke
Hi Alan, I'm CC'ing the Galaxy-Dev list on my response to your question. I talked with Jen Jackson on this. Her suggestion is that the symlink is most likely the problem somehow - this time. When in doubt, getting rid of these good place to start. So recommended starting solution to try - use full paths for everything. Also probably keep all on same partition for the best performance, but that would be hard to defend (there was some debate about this recently). Many thanks to Jen for her (as always) careful consideration. Dave C. On Mon, Nov 5, 2012 at 7:15 AM, Alan V. Cowles <alan.cowles@duke.edu> wrote:
Hey Dave,
Thanks for getting back in touch with me.
We are running our galaxy instance on a vm... with Centos (RHEL) 6.2, 2 vcpus, and 2gb vram, with a remote postgresql prod server as our database instance.
Our first issue was trying to get it so we could load sequence and larger files directly from an nfs mounted directory on the server, without uploading via the web (php vs large files problem.) We accomplished this via the data libraries interface but it's still not very user intuitive.
Our first demo user attempted to upload 3 large files and actually filled up the / partition on the server as galaxy was installed at /usr/local.
After scrapping and rebuilding that, we now have a several hundred GB nfs partition, mounted and /usr/local/galaxy is a symlink to a folder on this volume.
The issue we are currently seeing is during an upload, or even just linking files it runs for a while before erroring out and saying there is no information on the error. The file seems to be there and taking up the correct amount of disk space but galaxy has a red bar.
The galaxy server instance is running back grounded in a screen so we can always switch over and look at the output it shows to standard out as errors happen, and it seems that it has trouble changing permissions on the files in the nfs directory, even though, root squash is disabled, and the entire directory is 777 at the top level and everything underneath is owned galaxy:galaxy.
AC
-- <http://galaxyproject.org/wiki/GCC2012>http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ http://wiki.galaxyproject.org/
I tossed the symlink and mounted the san:/vol/galaxy_sata/data/galaxy to /usr/local/galaxy, though we use symlinks daily for practically all of our softwares and programs practically invisibly, but I am happy to try anything to get it going. AC On 11/06/2012 02:12 PM, Dave Clements wrote:
Hi Alan,
I'm CC'ing the Galaxy-Dev list on my response to your question.
I talked with Jen Jackson on this. Her suggestion is that the symlink is most likely the problem somehow - this time. When in doubt, getting rid of these good place to start. So recommended starting solution to try - use full paths for everything. Also probably keep all on same partition for the best performance, but that would be hard to defend (there was some debate about this recently).
Many thanks to Jen for her (as always) careful consideration.
Dave C.
On Mon, Nov 5, 2012 at 7:15 AM, Alan V. Cowles <alan.cowles@duke.edu <mailto:alan.cowles@duke.edu>> wrote:
Hey Dave,
Thanks for getting back in touch with me.
We are running our galaxy instance on a vm... with Centos (RHEL) 6.2, 2 vcpus, and 2gb vram, with a remote postgresql prod server as our database instance.
Our first issue was trying to get it so we could load sequence and larger files directly from an nfs mounted directory on the server, without uploading via the web (php vs large files problem.) We accomplished this via the data libraries interface but it's still not very user intuitive.
Our first demo user attempted to upload 3 large files and actually filled up the / partition on the server as galaxy was installed at /usr/local.
After scrapping and rebuilding that, we now have a several hundred GB nfs partition, mounted and /usr/local/galaxy is a symlink to a folder on this volume.
The issue we are currently seeing is during an upload, or even just linking files it runs for a while before erroring out and saying there is no information on the error. The file seems to be there and taking up the correct amount of disk space but galaxy has a red bar.
The galaxy server instance is running back grounded in a screen so we can always switch over and look at the output it shows to standard out as errors happen, and it seems that it has trouble changing permissions on the files in the nfs directory, even though, root squash is disabled, and the entire directory is 777 at the top level and everything underneath is owned galaxy:galaxy.
AC
-- http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ http://wiki.galaxyproject.org/
same error without the symlink, this time with actual syntax: galaxy.jobs ERROR 2012-11-06 14:41:06,596 (11) Failed to change ownership of /usr/local/galaxy/galaxy-dist/database/job_working_directory/000/11, failing Traceback (most recent call last): File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 293, in finish self.reclaim_ownership() File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 693, in reclaim_ownership self._change_ownership( self.galaxy_system_pwent[0], str( self.galaxy_system_pwent[3] ) ) File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 679, in _change_ownership assert p.returncode == 0 AssertionError galaxy.objectstore CRITICAL 2012-11-06 14:41:08,863 /usr/local/galaxy/galaxy-dist/database/job_working_directory/000/11 delete error [Errno 2] No such file or directory: '/usr/local/galaxy/galaxy-dist/database/job_working_directory/000/11' 152.3.172.61 - - [06/Nov/2012:14:41:07 -0400] "POST /library_common/library_item_updates HTTP/1.1" 200 - "http://galaxy.igsp.duke.edu:8080/library_common/browse_library?status=done&show_deleted=False&cntrller=library_admin&message=Added+7+datasets+to+the+library+__sq__Omics__sq__+%28each+is+selected%29.++Click+the+Go+button+at+the+bottom+of+this+page+to+edit+the+permissions+on+these+datasets+if+necessary.&id=5969b1f7201f12ae&created_ldda_ids=1462%2C1463%2C1464%2C1465%2C1466%2C1467%2C1468" "Mozilla/5.0 (X11; Linux x86_64; rv:16.0) Gecko/20100101 Firefox/16.0" On 11/06/2012 02:20 PM, Alan V. Cowles wrote:
I tossed the symlink and mounted the san:/vol/galaxy_sata/data/galaxy to /usr/local/galaxy, though we use symlinks daily for practically all of our softwares and programs practically invisibly, but I am happy to try anything to get it going.
AC
On 11/06/2012 02:12 PM, Dave Clements wrote:
Hi Alan,
I'm CC'ing the Galaxy-Dev list on my response to your question.
I talked with Jen Jackson on this. Her suggestion is that the symlink is most likely the problem somehow - this time. When in doubt, getting rid of these good place to start. So recommended starting solution to try - use full paths for everything. Also probably keep all on same partition for the best performance, but that would be hard to defend (there was some debate about this recently).
Many thanks to Jen for her (as always) careful consideration.
Dave C.
On Mon, Nov 5, 2012 at 7:15 AM, Alan V. Cowles <alan.cowles@duke.edu <mailto:alan.cowles@duke.edu>> wrote:
Hey Dave,
Thanks for getting back in touch with me.
We are running our galaxy instance on a vm... with Centos (RHEL) 6.2, 2 vcpus, and 2gb vram, with a remote postgresql prod server as our database instance.
Our first issue was trying to get it so we could load sequence and larger files directly from an nfs mounted directory on the server, without uploading via the web (php vs large files problem.) We accomplished this via the data libraries interface but it's still not very user intuitive.
Our first demo user attempted to upload 3 large files and actually filled up the / partition on the server as galaxy was installed at /usr/local.
After scrapping and rebuilding that, we now have a several hundred GB nfs partition, mounted and /usr/local/galaxy is a symlink to a folder on this volume.
The issue we are currently seeing is during an upload, or even just linking files it runs for a while before erroring out and saying there is no information on the error. The file seems to be there and taking up the correct amount of disk space but galaxy has a red bar.
The galaxy server instance is running back grounded in a screen so we can always switch over and look at the output it shows to standard out as errors happen, and it seems that it has trouble changing permissions on the files in the nfs directory, even though, root squash is disabled, and the entire directory is 777 at the top level and everything underneath is owned galaxy:galaxy.
AC
-- http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ http://wiki.galaxyproject.org/
On Nov 6, 2012, at 2:44 PM, Alan V. Cowles wrote:
same error without the symlink, this time with actual syntax:
galaxy.jobs ERROR 2012-11-06 14:41:06,596 (11) Failed to change ownership of /usr/local/galaxy/galaxy-dist/database/job_working_directory/000/11, failing
Hi Alan, You're using the "run jobs as the actual user" code? It looks like the job's working directory (/usr/local/galaxy/galaxy-dist/database/job_working_directory/000/11) does not exist after the job finishes running:
Traceback (most recent call last): File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 293, in finish self.reclaim_ownership() File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 693, in reclaim_ownership self._change_ownership( self.galaxy_system_pwent[0], str( self.galaxy_system_pwent[3] ) ) File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 679, in _change_ownership assert p.returncode == 0 AssertionError galaxy.objectstore CRITICAL 2012-11-06 14:41:08,863 /usr/local/galaxy/galaxy-dist/database/job_working_directory/000/11 delete error [Errno 2] No such file or directory: '/usr/local/galaxy/galaxy-dist/database/job_working_directory/000/11'
Which is why you're getting the errors about changing permissions. --nate
152.3.172.61 - - [06/Nov/2012:14:41:07 -0400] "POST /library_common/library_item_updates HTTP/1.1" 200 - "http://galaxy.igsp.duke.edu:8080/library_common/browse_library?status=done&show_deleted=False&cntrller=library_admin&message=Added+7+datasets+to+the+library+__sq__Omics__sq__+%28each+is+selected%29.++Click+the+Go+button+at+the+bottom+of+this+page+to+edit+the+permissions+on+these+datasets+if+necessary.&id=5969b1f7201f12ae&created_ldda_ids=1462%2C1463%2C1464%2C1465%2C1466%2C1467%2C1468" "Mozilla/5.0 (X11; Linux x86_64; rv:16.0) Gecko/20100101 Firefox/16.0"
On 11/06/2012 02:20 PM, Alan V. Cowles wrote:
I tossed the symlink and mounted the san:/vol/galaxy_sata/data/galaxy to /usr/local/galaxy, though we use symlinks daily for practically all of our softwares and programs practically invisibly, but I am happy to try anything to get it going.
AC
On 11/06/2012 02:12 PM, Dave Clements wrote:
Hi Alan,
I'm CC'ing the Galaxy-Dev list on my response to your question.
I talked with Jen Jackson on this. Her suggestion is that the symlink is most likely the problem somehow - this time. When in doubt, getting rid of these good place to start. So recommended starting solution to try - use full paths for everything. Also probably keep all on same partition for the best performance, but that would be hard to defend (there was some debate about this recently).
Many thanks to Jen for her (as always) careful consideration.
Dave C.
On Mon, Nov 5, 2012 at 7:15 AM, Alan V. Cowles <alan.cowles@duke.edu> wrote: Hey Dave,
Thanks for getting back in touch with me.
We are running our galaxy instance on a vm... with Centos (RHEL) 6.2, 2 vcpus, and 2gb vram, with a remote postgresql prod server as our database instance.
Our first issue was trying to get it so we could load sequence and larger files directly from an nfs mounted directory on the server, without uploading via the web (php vs large files problem.) We accomplished this via the data libraries interface but it's still not very user intuitive.
Our first demo user attempted to upload 3 large files and actually filled up the / partition on the server as galaxy was installed at /usr/local.
After scrapping and rebuilding that, we now have a several hundred GB nfs partition, mounted and /usr/local/galaxy is a symlink to a folder on this volume.
The issue we are currently seeing is during an upload, or even just linking files it runs for a while before erroring out and saying there is no information on the error. The file seems to be there and taking up the correct amount of disk space but galaxy has a red bar.
The galaxy server instance is running back grounded in a screen so we can always switch over and look at the output it shows to standard out as errors happen, and it seems that it has trouble changing permissions on the files in the nfs directory, even though, root squash is disabled, and the entire directory is 777 at the top level and everything underneath is owned galaxy:galaxy.
AC
-- http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ http://wiki.galaxyproject.org/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
yeah I'm not seeing the folder 11, I have a folder 2 created july 19... (galaxy_env)-bash-4.1$ pwd /usr/local/galaxy/galaxy-dist/database/job_working_directory/000 (galaxy_env)-bash-4.1$ ls -al total 12 drwxrwxr-x 3 galaxy galaxy 4096 Nov 6 14:41 . drwxrwxr-x 3 galaxy galaxy 4096 Jul 19 11:51 .. drwxrwxr-x 2 galaxy galaxy 4096 Jul 19 12:47 2 should I make 11? or is it supposed to do this automatically for the job? On 11/12/2012 10:54 AM, Nate Coraor wrote:
On Nov 6, 2012, at 2:44 PM, Alan V. Cowles wrote:
same error without the symlink, this time with actual syntax:
galaxy.jobs ERROR 2012-11-06 14:41:06,596 (11) Failed to change ownership of /usr/local/galaxy/galaxy-dist/database/job_working_directory/000/11, failing Hi Alan,
You're using the "run jobs as the actual user" code? It looks like the job's working directory (/usr/local/galaxy/galaxy-dist/database/job_working_directory/000/11) does not exist after the job finishes running:
Traceback (most recent call last): File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 293, in finish self.reclaim_ownership() File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 693, in reclaim_ownership self._change_ownership( self.galaxy_system_pwent[0], str( self.galaxy_system_pwent[3] ) ) File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 679, in _change_ownership assert p.returncode == 0 AssertionError galaxy.objectstore CRITICAL 2012-11-06 14:41:08,863 /usr/local/galaxy/galaxy-dist/database/job_working_directory/000/11 delete error [Errno 2] No such file or directory: '/usr/local/galaxy/galaxy-dist/database/job_working_directory/000/11' Which is why you're getting the errors about changing permissions.
--nate
152.3.172.61 - - [06/Nov/2012:14:41:07 -0400] "POST /library_common/library_item_updates HTTP/1.1" 200 - "http://galaxy.igsp.duke.edu:8080/library_common/browse_library?status=done&show_deleted=False&cntrller=library_admin&message=Added+7+datasets+to+the+library+__sq__Omics__sq__+%28each+is+selected%29.++Click+the+Go+button+at+the+bottom+of+this+page+to+edit+the+permissions+on+these+datasets+if+necessary.&id=5969b1f7201f12ae&created_ldda_ids=1462%2C1463%2C1464%2C1465%2C1466%2C1467%2C1468" "Mozilla/5.0 (X11; Linux x86_64; rv:16.0) Gecko/20100101 Firefox/16.0"
On 11/06/2012 02:20 PM, Alan V. Cowles wrote:
I tossed the symlink and mounted the san:/vol/galaxy_sata/data/galaxy to /usr/local/galaxy, though we use symlinks daily for practically all of our softwares and programs practically invisibly, but I am happy to try anything to get it going.
AC
On 11/06/2012 02:12 PM, Dave Clements wrote:
Hi Alan,
I'm CC'ing the Galaxy-Dev list on my response to your question.
I talked with Jen Jackson on this. Her suggestion is that the symlink is most likely the problem somehow - this time. When in doubt, getting rid of these good place to start. So recommended starting solution to try - use full paths for everything. Also probably keep all on same partition for the best performance, but that would be hard to defend (there was some debate about this recently).
Many thanks to Jen for her (as always) careful consideration.
Dave C.
On Mon, Nov 5, 2012 at 7:15 AM, Alan V. Cowles <alan.cowles@duke.edu> wrote: Hey Dave,
Thanks for getting back in touch with me.
We are running our galaxy instance on a vm... with Centos (RHEL) 6.2, 2 vcpus, and 2gb vram, with a remote postgresql prod server as our database instance.
Our first issue was trying to get it so we could load sequence and larger files directly from an nfs mounted directory on the server, without uploading via the web (php vs large files problem.) We accomplished this via the data libraries interface but it's still not very user intuitive.
Our first demo user attempted to upload 3 large files and actually filled up the / partition on the server as galaxy was installed at /usr/local.
After scrapping and rebuilding that, we now have a several hundred GB nfs partition, mounted and /usr/local/galaxy is a symlink to a folder on this volume.
The issue we are currently seeing is during an upload, or even just linking files it runs for a while before erroring out and saying there is no information on the error. The file seems to be there and taking up the correct amount of disk space but galaxy has a red bar.
The galaxy server instance is running back grounded in a screen so we can always switch over and look at the output it shows to standard out as errors happen, and it seems that it has trouble changing permissions on the files in the nfs directory, even though, root squash is disabled, and the entire directory is 777 at the top level and everything underneath is owned galaxy:galaxy.
AC
-- http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ http://wiki.galaxyproject.org/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
On Nov 12, 2012, at 11:17 AM, Alan V. Cowles wrote:
yeah I'm not seeing the folder 11, I have a folder 2 created july 19...
(galaxy_env)-bash-4.1$ pwd /usr/local/galaxy/galaxy-dist/database/job_working_directory/000
(galaxy_env)-bash-4.1$ ls -al total 12 drwxrwxr-x 3 galaxy galaxy 4096 Nov 6 14:41 . drwxrwxr-x 3 galaxy galaxy 4096 Jul 19 11:51 .. drwxrwxr-x 2 galaxy galaxy 4096 Jul 19 12:47 2
should I make 11? or is it supposed to do this automatically for the job?
A working directory will be created every time a new job runs. You can use the "cleanup_job" config option to control whether this directory is removed after job completion. --nate
On 11/12/2012 10:54 AM, Nate Coraor wrote:
On Nov 6, 2012, at 2:44 PM, Alan V. Cowles wrote:
same error without the symlink, this time with actual syntax:
galaxy.jobs ERROR 2012-11-06 14:41:06,596 (11) Failed to change ownership of /usr/local/galaxy/galaxy-dist/database/job_working_directory/000/11, failing Hi Alan,
You're using the "run jobs as the actual user" code? It looks like the job's working directory (/usr/local/galaxy/galaxy-dist/database/job_working_directory/000/11) does not exist after the job finishes running:
Traceback (most recent call last): File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 293, in finish self.reclaim_ownership() File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 693, in reclaim_ownership self._change_ownership( self.galaxy_system_pwent[0], str( self.galaxy_system_pwent[3] ) ) File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 679, in _change_ownership assert p.returncode == 0 AssertionError galaxy.objectstore CRITICAL 2012-11-06 14:41:08,863 /usr/local/galaxy/galaxy-dist/database/job_working_directory/000/11 delete error [Errno 2] No such file or directory: '/usr/local/galaxy/galaxy-dist/database/job_working_directory/000/11' Which is why you're getting the errors about changing permissions.
--nate
152.3.172.61 - - [06/Nov/2012:14:41:07 -0400] "POST /library_common/library_item_updates HTTP/1.1" 200 - "http://galaxy.igsp.duke.edu:8080/library_common/browse_library?status=done&show_deleted=False&cntrller=library_admin&message=Added+7+datasets+to+the+library+__sq__Omics__sq__+%28each+is+selected%29.++Click+the+Go+button+at+the+bottom+of+this+page+to+edit+the+permissions+on+these+datasets+if+necessary.&id=5969b1f7201f12ae&created_ldda_ids=1462%2C1463%2C1464%2C1465%2C1466%2C1467%2C1468" "Mozilla/5.0 (X11; Linux x86_64; rv:16.0) Gecko/20100101 Firefox/16.0"
On 11/06/2012 02:20 PM, Alan V. Cowles wrote:
I tossed the symlink and mounted the san:/vol/galaxy_sata/data/galaxy to /usr/local/galaxy, though we use symlinks daily for practically all of our softwares and programs practically invisibly, but I am happy to try anything to get it going.
AC
On 11/06/2012 02:12 PM, Dave Clements wrote:
Hi Alan,
I'm CC'ing the Galaxy-Dev list on my response to your question.
I talked with Jen Jackson on this. Her suggestion is that the symlink is most likely the problem somehow - this time. When in doubt, getting rid of these good place to start. So recommended starting solution to try - use full paths for everything. Also probably keep all on same partition for the best performance, but that would be hard to defend (there was some debate about this recently).
Many thanks to Jen for her (as always) careful consideration.
Dave C.
On Mon, Nov 5, 2012 at 7:15 AM, Alan V. Cowles <alan.cowles@duke.edu> wrote: Hey Dave,
Thanks for getting back in touch with me.
We are running our galaxy instance on a vm... with Centos (RHEL) 6.2, 2 vcpus, and 2gb vram, with a remote postgresql prod server as our database instance.
Our first issue was trying to get it so we could load sequence and larger files directly from an nfs mounted directory on the server, without uploading via the web (php vs large files problem.) We accomplished this via the data libraries interface but it's still not very user intuitive.
Our first demo user attempted to upload 3 large files and actually filled up the / partition on the server as galaxy was installed at /usr/local.
After scrapping and rebuilding that, we now have a several hundred GB nfs partition, mounted and /usr/local/galaxy is a symlink to a folder on this volume.
The issue we are currently seeing is during an upload, or even just linking files it runs for a while before erroring out and saying there is no information on the error. The file seems to be there and taking up the correct amount of disk space but galaxy has a red bar.
The galaxy server instance is running back grounded in a screen so we can always switch over and look at the output it shows to standard out as errors happen, and it seems that it has trouble changing permissions on the files in the nfs directory, even though, root squash is disabled, and the entire directory is 777 at the top level and everything underneath is owned galaxy:galaxy.
AC
-- http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ http://wiki.galaxyproject.org/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
So from what I am reading it is either not creating or prematurely deleting this working directory? I will mod that option and see if I can prevent it from deleting. AC On 11/12/2012 2:11 PM, Nate Coraor wrote:
On Nov 12, 2012, at 11:17 AM, Alan V. Cowles wrote:
yeah I'm not seeing the folder 11, I have a folder 2 created july 19...
(galaxy_env)-bash-4.1$ pwd /usr/local/galaxy/galaxy-dist/database/job_working_directory/000
(galaxy_env)-bash-4.1$ ls -al total 12 drwxrwxr-x 3 galaxy galaxy 4096 Nov 6 14:41 . drwxrwxr-x 3 galaxy galaxy 4096 Jul 19 11:51 .. drwxrwxr-x 2 galaxy galaxy 4096 Jul 19 12:47 2
should I make 11? or is it supposed to do this automatically for the job? A working directory will be created every time a new job runs. You can use the "cleanup_job" config option to control whether this directory is removed after job completion.
--nate
On 11/12/2012 10:54 AM, Nate Coraor wrote:
On Nov 6, 2012, at 2:44 PM, Alan V. Cowles wrote:
same error without the symlink, this time with actual syntax:
galaxy.jobs ERROR 2012-11-06 14:41:06,596 (11) Failed to change ownership of /usr/local/galaxy/galaxy-dist/database/job_working_directory/000/11, failing Hi Alan,
You're using the "run jobs as the actual user" code? It looks like the job's working directory (/usr/local/galaxy/galaxy-dist/database/job_working_directory/000/11) does not exist after the job finishes running:
Traceback (most recent call last): File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 293, in finish self.reclaim_ownership() File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 693, in reclaim_ownership self._change_ownership( self.galaxy_system_pwent[0], str( self.galaxy_system_pwent[3] ) ) File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 679, in _change_ownership assert p.returncode == 0 AssertionError galaxy.objectstore CRITICAL 2012-11-06 14:41:08,863 /usr/local/galaxy/galaxy-dist/database/job_working_directory/000/11 delete error [Errno 2] No such file or directory: '/usr/local/galaxy/galaxy-dist/database/job_working_directory/000/11' Which is why you're getting the errors about changing permissions.
--nate
152.3.172.61 - - [06/Nov/2012:14:41:07 -0400] "POST /library_common/library_item_updates HTTP/1.1" 200 - "http://galaxy.igsp.duke.edu:8080/library_common/browse_library?status=done&show_deleted=False&cntrller=library_admin&message=Added+7+datasets+to+the+library+__sq__Omics__sq__+%28each+is+selected%29.++Click+the+Go+button+at+the+bottom+of+this+page+to+edit+the+permissions+on+these+datasets+if+necessary.&id=5969b1f7201f12ae&created_ldda_ids=1462%2C1463%2C1464%2C1465%2C1466%2C1467%2C1468" "Mozilla/5.0 (X11; Linux x86_64; rv:16.0) Gecko/20100101 Firefox/16.0"
On 11/06/2012 02:20 PM, Alan V. Cowles wrote:
I tossed the symlink and mounted the san:/vol/galaxy_sata/data/galaxy to /usr/local/galaxy, though we use symlinks daily for practically all of our softwares and programs practically invisibly, but I am happy to try anything to get it going.
AC
On 11/06/2012 02:12 PM, Dave Clements wrote:
Hi Alan,
I'm CC'ing the Galaxy-Dev list on my response to your question.
I talked with Jen Jackson on this. Her suggestion is that the symlink is most likely the problem somehow - this time. When in doubt, getting rid of these good place to start. So recommended starting solution to try - use full paths for everything. Also probably keep all on same partition for the best performance, but that would be hard to defend (there was some debate about this recently).
Many thanks to Jen for her (as always) careful consideration.
Dave C.
On Mon, Nov 5, 2012 at 7:15 AM, Alan V. Cowles <alan.cowles@duke.edu> wrote: Hey Dave,
Thanks for getting back in touch with me.
We are running our galaxy instance on a vm... with Centos (RHEL) 6.2, 2 vcpus, and 2gb vram, with a remote postgresql prod server as our database instance.
Our first issue was trying to get it so we could load sequence and larger files directly from an nfs mounted directory on the server, without uploading via the web (php vs large files problem.) We accomplished this via the data libraries interface but it's still not very user intuitive.
Our first demo user attempted to upload 3 large files and actually filled up the / partition on the server as galaxy was installed at /usr/local.
After scrapping and rebuilding that, we now have a several hundred GB nfs partition, mounted and /usr/local/galaxy is a symlink to a folder on this volume.
The issue we are currently seeing is during an upload, or even just linking files it runs for a while before erroring out and saying there is no information on the error. The file seems to be there and taking up the correct amount of disk space but galaxy has a red bar.
The galaxy server instance is running back grounded in a screen so we can always switch over and look at the output it shows to standard out as errors happen, and it seems that it has trouble changing permissions on the files in the nfs directory, even though, root squash is disabled, and the entire directory is 777 at the top level and everything underneath is owned galaxy:galaxy.
AC
-- http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ http://wiki.galaxyproject.org/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Hi Alan, The file is first uploaded in /tmp and then copied to your NFS mounted share. I think it might be the last step that's causing trouble. Regarding NFS4: Make sure the settings in /etc/idmapd.conf are correct: especially the 'Domain' setting was causing trouble in our case if not set correctly. If it's not working, you can also try to change here the Nobody-user and -group to your galaxy user account, as a temporary fix, and if NFS is only being by Galaxy of course. See $man idpmapd.conf Good luck, Joachim Joachim Jacob, PhD Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib On 11/06/2012 08:12 PM, Dave Clements wrote:
Hi Alan,
I'm CC'ing the Galaxy-Dev list on my response to your question.
I talked with Jen Jackson on this. Her suggestion is that the symlink is most likely the problem somehow - this time. When in doubt, getting rid of these good place to start. So recommended starting solution to try - use full paths for everything. Also probably keep all on same partition for the best performance, but that would be hard to defend (there was some debate about this recently).
Many thanks to Jen for her (as always) careful consideration.
Dave C.
On Mon, Nov 5, 2012 at 7:15 AM, Alan V. Cowles <alan.cowles@duke.edu <mailto:alan.cowles@duke.edu>> wrote:
Hey Dave,
Thanks for getting back in touch with me.
We are running our galaxy instance on a vm... with Centos (RHEL) 6.2, 2 vcpus, and 2gb vram, with a remote postgresql prod server as our database instance.
Our first issue was trying to get it so we could load sequence and larger files directly from an nfs mounted directory on the server, without uploading via the web (php vs large files problem.) We accomplished this via the data libraries interface but it's still not very user intuitive.
Our first demo user attempted to upload 3 large files and actually filled up the / partition on the server as galaxy was installed at /usr/local.
After scrapping and rebuilding that, we now have a several hundred GB nfs partition, mounted and /usr/local/galaxy is a symlink to a folder on this volume.
The issue we are currently seeing is during an upload, or even just linking files it runs for a while before erroring out and saying there is no information on the error. The file seems to be there and taking up the correct amount of disk space but galaxy has a red bar.
The galaxy server instance is running back grounded in a screen so we can always switch over and look at the output it shows to standard out as errors happen, and it seems that it has trouble changing permissions on the files in the nfs directory, even though, root squash is disabled, and the entire directory is 777 at the top level and everything underneath is owned galaxy:galaxy.
AC
-- http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ http://wiki.galaxyproject.org/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
We are using NFS3 for our mounts as we have local private networks... so it shouldn't be the problem that you are referring to, and I don't think it's a lack of space in temp, as the files in some form or another are making it into the /data/001 folder... AC On 11/7/2012 3:53 AM, Joachim Jacob wrote:
Hi Alan,
The file is first uploaded in /tmp and then copied to your NFS mounted share. I think it might be the last step that's causing trouble.
Regarding NFS4: Make sure the settings in /etc/idmapd.conf are correct: especially the 'Domain' setting was causing trouble in our case if not set correctly. If it's not working, you can also try to change here the Nobody-user and -group to your galaxy user account, as a temporary fix, and if NFS is only being by Galaxy of course.
See $man idpmapd.conf
Good luck,
Joachim
Joachim Jacob, PhD
Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib
On 11/06/2012 08:12 PM, Dave Clements wrote:
Hi Alan,
I'm CC'ing the Galaxy-Dev list on my response to your question.
I talked with Jen Jackson on this. Her suggestion is that the symlink is most likely the problem somehow - this time. When in doubt, getting rid of these good place to start. So recommended starting solution to try - use full paths for everything. Also probably keep all on same partition for the best performance, but that would be hard to defend (there was some debate about this recently).
Many thanks to Jen for her (as always) careful consideration.
Dave C.
On Mon, Nov 5, 2012 at 7:15 AM, Alan V. Cowles <alan.cowles@duke.edu <mailto:alan.cowles@duke.edu>> wrote:
Hey Dave,
Thanks for getting back in touch with me.
We are running our galaxy instance on a vm... with Centos (RHEL) 6.2, 2 vcpus, and 2gb vram, with a remote postgresql prod server as our database instance.
Our first issue was trying to get it so we could load sequence and larger files directly from an nfs mounted directory on the server, without uploading via the web (php vs large files problem.) We accomplished this via the data libraries interface but it's still not very user intuitive.
Our first demo user attempted to upload 3 large files and actually filled up the / partition on the server as galaxy was installed at /usr/local.
After scrapping and rebuilding that, we now have a several hundred GB nfs partition, mounted and /usr/local/galaxy is a symlink to a folder on this volume.
The issue we are currently seeing is during an upload, or even just linking files it runs for a while before erroring out and saying there is no information on the error. The file seems to be there and taking up the correct amount of disk space but galaxy has a red bar.
The galaxy server instance is running back grounded in a screen so we can always switch over and look at the output it shows to standard out as errors happen, and it seems that it has trouble changing permissions on the files in the nfs directory, even though, root squash is disabled, and the entire directory is 777 at the top level and everything underneath is owned galaxy:galaxy.
AC
-- http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ http://wiki.galaxyproject.org/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
participants (4)
-
Alan V. Cowles -
Dave Clements -
Joachim Jacob -
Nate Coraor