Re: [galaxy-dev] [galaxy-bugs] /tmp directory
Thanks for the answer! I've copied this response to the dev list email instead, after realizing its probably best there. I'm also having some trouble getting our PBS system configured. I have pbs://hostname/ as my pbs parameter in the universe file, as well as everything else specified in the documentation. I'm getting no error in the interface, and the backend never actually gets the job. The error I see in the logs says: galaxy.jobs.runners.pbs ERROR 2009-10-30 13:40:27,762 Uncaught exception queueing job Traceback (most recent call last): File "/home/perin/galaxy-dist/lib/galaxy/jobs/runners/pbs.py", line 144, in run_next self.queue_job( obj ) File "/home/perin/galaxy-dist/lib/galaxy/jobs/runners/pbs.py", line 219, in queue_job script = pbs_symlink_template % (job_wrapper.galaxy_lib_dir, " ".join(job_wrapper.get_input_fnames() + output_fnames), self.app.config.pbs_stage_path, exec_dir, command_line) TypeError: sequence item 1: expected string, DatasetPath found I'm guessing this has something to do with my config parameters, and the queue its trying to submit to. the pbs://hostname/ value was used under the assumption that it would default to the 'default' queue on our cluster, so I'm wondering if I need to hard code those values into the connection string? I'm a little confused as to how the syntax translates to the actual required string. My guess is, for our values: pbs://variome.chop.edu/default Perhaps its related to the scrambling of the egg instead? I simply passed LIBTORQUE_DIR to the directory containing my libtorqure.so etc... ? Thanks in advance. Juan Perin On Fri, Oct 30, 2009 at 12:07 PM, Nate Coraor <nate@bx.psu.edu> wrote:
Juan Perin wrote:
I'm hosting galaxy locally and am trying to figure out how to change the
temporary upload location of files from the root / /tmp folder to something else. My /tmp partition is too small to handle large fastq files and this is causing me problems. I see where to change things in the universe file, but that's the final location, not the temporary space.
Hi Juan,
A quick solution is to set $TEMP in your shell environment. In the future, we'll force uploads to honor new_file_path in universe_wsgi.ini.
--nate
juan perin wrote:
Thanks for the answer! I've copied this response to the dev list email instead, after realizing its probably best there.
Agreed.
I'm also having some trouble getting our PBS system configured. I have pbs://hostname/ as my pbs parameter in the universe file, as well as everything else specified in the documentation. I'm getting no error in the interface, and the backend never actually gets the job. The error I see in the logs says:
galaxy.jobs.runners.pbs ERROR 2009-10-30 13:40:27,762 Uncaught exception queueing job Traceback (most recent call last): File "/home/perin/galaxy-dist/lib/galaxy/jobs/runners/pbs.py", line 144, in run_next self.queue_job( obj ) File "/home/perin/galaxy-dist/lib/galaxy/jobs/runners/pbs.py", line 219, in queue_job script = pbs_symlink_template % (job_wrapper.galaxy_lib_dir, " ".join(job_wrapper.get_input_fnames() + output_fnames), self.app.config.pbs_stage_path, exec_dir, command_line) TypeError: sequence item 1: expected string, DatasetPath found
Hi Juan, It looks like you're using file staging, and there was a bug in our file staging code (we don't use staging ourselves anymore, so it wasn't caught before). Here's the fix, you can pull it from our central repo or simply apply it manually: http://bitbucket.org/galaxy/galaxy-central/changeset/dac542cca894/ --nate
Actually, i did not want to use file staging. I am running from my home directory which is visible on all nodes, and has plenty of space for my needs. I'm guessing I did something in the config to activate file staging, in which case that would explain the error that shouldn't be. I didn't read closely enough, but I set: # The PBS options are described in detail in the Galaxy Configuration section of # the ClusteringGalaxy Wiki, and are only necessary when using file staging. pbs_application_server = variome pbs_stage_path = /home/perin/galaxy-dist/database/tmp pbs_dataset_server = variome which obviously caused issues as a result. Commenting this back cleared it up, but introduced another issue, where the nodes apparently don't see the binary it is trying to run. I was assuming that the binaries were located in the home directory I was running from, but now see that they are actually in /usr/local/bin/etc... I'm not sure where this is set, but assume it was done sometime during the setup.sh process. Is there a way I can specify this install location, or change it anywhere? I can copy the binaries over to the nodes, but would prefer to host them in a shared place. Is there a way to change Galaxy so that it looks in say "/share/apps/bin/" instead of /usr/local/bin ? Thanks! Juan On Fri, Oct 30, 2009 at 2:30 PM, Nate Coraor <nate@bx.psu.edu> wrote:
juan perin wrote:
Thanks for the answer! I've copied this response to the dev list email
instead, after realizing its probably best there.
Agreed.
I'm also having some trouble getting our PBS system configured. I have
pbs://hostname/ as my pbs parameter in the universe file, as well as everything else specified in the documentation. I'm getting no error in the interface, and the backend never actually gets the job. The error I see in the logs says:
galaxy.jobs.runners.pbs ERROR 2009-10-30 13:40:27,762 Uncaught exception queueing job Traceback (most recent call last): File "/home/perin/galaxy-dist/lib/galaxy/jobs/runners/pbs.py", line 144, in run_next self.queue_job( obj ) File "/home/perin/galaxy-dist/lib/galaxy/jobs/runners/pbs.py", line 219, in queue_job script = pbs_symlink_template % (job_wrapper.galaxy_lib_dir, " ".join(job_wrapper.get_input_fnames() + output_fnames), self.app.config.pbs_stage_path, exec_dir, command_line) TypeError: sequence item 1: expected string, DatasetPath found
Hi Juan,
It looks like you're using file staging, and there was a bug in our file staging code (we don't use staging ourselves anymore, so it wasn't caught before).
Here's the fix, you can pull it from our central repo or simply apply it manually:
http://bitbucket.org/galaxy/galaxy-central/changeset/dac542cca894/
--nate
juan perin wrote:
Actually, i did not want to use file staging. I am running from my home directory which is visible on all nodes, and has plenty of space for my needs. I'm guessing I did something in the config to activate file staging, in which case that would explain the error that shouldn't be. I didn't read closely enough, but I set:
# The PBS options are described in detail in the Galaxy Configuration section of # the ClusteringGalaxy Wiki, and are only necessary when using file staging. pbs_application_server = variome pbs_stage_path = /home/perin/galaxy-dist/database/tmp pbs_dataset_server = variome
which obviously caused issues as a result. Commenting this back cleared it up, but introduced another issue, where the nodes apparently don't see the binary it is trying to run. I was assuming that the binaries were located in the home directory I was running from, but now see that they are actually in /usr/local/bin/etc... I'm not sure where this is set, but assume it was done sometime during the setup.sh process. Is there a way I can specify this install location, or change it anywhere? I can copy the binaries over to the nodes, but would prefer to host them in a shared place. Is there a way to change Galaxy so that it looks in say "/share/apps/bin/" instead of /usr/local/bin ?
Do you mean the binaries for the dependencies? If so, they are expected to be found on the $PATH, so if that's consistent on the Galaxy server and the nodes, you should be okay. Otherwise, I'm not sure which binaries you're referring to.
I see that I was mistakenly assuming the FASTX toolkit was provided with Galaxy. I had installed it a while back and forgot. It now works fine having placed it properly where the nodes can see it. I'm now onto another issue. This time, I can't get bwa or bowtie to provide a valid alignment. I've tried several different fastq files, yet they all end up giving me the same output: "empty, format: sam, database:Info:" I'm using data that has worked fine for us outside of galaxy, however this won't run for me in the browser. I can't find any other error messages anywhere. I have tried two different human genome databases. One that was pre-indexed for bowtie from the bowtie website, and the other was an hg18 version i indexed using bowtie manually. Neither works for me. Any ideas on what I might be doing wrong, or what might be happening would be very much appreciated. Even if someone might point me in the right direction, as to what logs to be looking at. To note, the galaxy interface does not report an 'error' per se. It just gives me no results, and the output i pasted above. Thanks in advance. juan On Fri, Oct 30, 2009 at 4:32 PM, Nate Coraor <nate@bx.psu.edu> wrote:
juan perin wrote:
Actually, i did not want to use file staging. I am running from my home
directory which is visible on all nodes, and has plenty of space for my needs. I'm guessing I did something in the config to activate file staging, in which case that would explain the error that shouldn't be. I didn't read closely enough, but I set:
# The PBS options are described in detail in the Galaxy Configuration section of # the ClusteringGalaxy Wiki, and are only necessary when using file staging. pbs_application_server = variome pbs_stage_path = /home/perin/galaxy-dist/database/tmp pbs_dataset_server = variome
which obviously caused issues as a result. Commenting this back cleared it up, but introduced another issue, where the nodes apparently don't see the binary it is trying to run. I was assuming that the binaries were located in the home directory I was running from, but now see that they are actually in /usr/local/bin/etc... I'm not sure where this is set, but assume it was done sometime during the setup.sh process. Is there a way I can specify this install location, or change it anywhere? I can copy the binaries over to the nodes, but would prefer to host them in a shared place. Is there a way to change Galaxy so that it looks in say "/share/apps/bin/" instead of /usr/local/bin ?
Do you mean the binaries for the dependencies? If so, they are expected to be found on the $PATH, so if that's consistent on the Galaxy server and the nodes, you should be okay.
Otherwise, I'm not sure which binaries you're referring to.
Juan, I'm sure you are running up against errors with Bowtie and BWA, but we had to suppress the error output of the tools because it is also used for non-error output (and therefore stops the tool sometimes when it should not). First, check that the PATH variable has the right paths to your Bowtie and BWA installations. If that doesn't help, if you can share your history with me (kpvincent@bx.psu.edu ), I can try running your data without error output suppression, and see what I find. Alternatively, if you would rather do this yourself, it's actually very easy. In the tools/sr_mapping/bowtie_wrapper.py file, remove the "2> /dev/null" from the relevant command. The indexing command is on line 96 and the aligning command is on line 140 (paired-end data) or 142 (single-end). The same can be done for BWA (tools/sr_mapping/ bwa_wrapper.py): indexing--line 68, aligning--lines 97, 100, 101, and 103. Once you've done this, just run the tool as normal in the browser. Any output that goes to stderr will show up in the history. Thanks, Kelly On Nov 2, 2009, at 5:14 PM, juan perin wrote:
I see that I was mistakenly assuming the FASTX toolkit was provided with Galaxy. I had installed it a while back and forgot. It now works fine having placed it properly where the nodes can see it. I'm now onto another issue.
This time, I can't get bwa or bowtie to provide a valid alignment. I've tried several different fastq files, yet they all end up giving me the same output:
"empty, format: sam, database:Info:" I'm using data that has worked fine for us outside of galaxy, however this won't run for me in the browser. I can't find any other error messages anywhere. I have tried two different human genome databases. One that was pre-indexed for bowtie from the bowtie website, and the other was an hg18 version i indexed using bowtie manually. Neither works for me.
Any ideas on what I might be doing wrong, or what might be happening would be very much appreciated. Even if someone might point me in the right direction, as to what logs to be looking at. To note, the galaxy interface does not report an 'error' per se. It just gives me no results, and the output i pasted above.
Thanks in advance.
juan
On Fri, Oct 30, 2009 at 4:32 PM, Nate Coraor <nate@bx.psu.edu> wrote: juan perin wrote:
Actually, i did not want to use file staging. I am running from my home directory which is visible on all nodes, and has plenty of space for my needs. I'm guessing I did something in the config to activate file staging, in which case that would explain the error that shouldn't be. I didn't read closely enough, but I set:
# The PBS options are described in detail in the Galaxy Configuration section of # the ClusteringGalaxy Wiki, and are only necessary when using file staging. pbs_application_server = variome pbs_stage_path = /home/perin/galaxy-dist/database/tmp pbs_dataset_server = variome
which obviously caused issues as a result. Commenting this back cleared it up, but introduced another issue, where the nodes apparently don't see the binary it is trying to run. I was assuming that the binaries were located in the home directory I was running from, but now see that they are actually in /usr/local/bin/etc... I'm not sure where this is set, but assume it was done sometime during the setup.sh process. Is there a way I can specify this install location, or change it anywhere? I can copy the binaries over to the nodes, but would prefer to host them in a shared place. Is there a way to change Galaxy so that it looks in say "/share/apps/ bin/" instead of /usr/local/bin ?
Do you mean the binaries for the dependencies? If so, they are expected to be found on the $PATH, so if that's consistent on the Galaxy server and the nodes, you should be okay.
Otherwise, I'm not sure which binaries you're referring to.
_______________________________________________ galaxy-dev mailing list galaxy-dev@bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-dev
I've been working with a much older release of galaxy for a while, and it has worked great for the last few months. I noticed some progress with the NGS tools, so decided to attempt updating. My first mistake was in not knowing how to do so, so I essentially decided to start from the hg clone step and rebuild galaxy entirely in a new place on the same machine. I stupidly ran the database update script without listening to the warning about backing up the original galaxy db. So, my original instance won't work now... So, i'm trying to get the new instance working. Copied over my universe file and the custom .loc files from tool-data/ . everything seems to be ok, however I'm getting a python error that essentially repeats for anything I try to do, I'll paste the full output below for reference. I'm using python 2.4 Any ideas? Thanks in advance. URL: http://variome.chop.edu:8082/tool_runner?tool_id=bowtie_wrapper File '/opt/galaxy-dist/eggs/py2.4-noplatform/WebError-0.8a-py2.4.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/opt/galaxy-dist/eggs/py2.4-noplatform/Paste-1.6-py2.4.egg/paste/debug/prints.py', line 97 in __call__ status, headers, body = wsgilib.intercept_output( File '/opt/galaxy-dist/eggs/py2.4-noplatform/Paste-1.6-py2.4.egg/paste/wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/opt/galaxy-dist/eggs/py2.4-noplatform/Paste-1.6-py2.4.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/opt/galaxy-dist/eggs/py2.4-noplatform/Paste-1.6-py2.4.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/opt/galaxy-dist/lib/galaxy/web/framework/base.py', line 125 in __call__ body = method( trans, **kwargs ) File '/opt/galaxy-dist/lib/galaxy/web/controllers/tool_runner.py', line 61 in index return trans.fill_template( template, history=history, toolbox=toolbox, tool=tool, util=util, add_frame=add_frame, **vars ) File '/opt/galaxy-dist/lib/galaxy/web/framework/__init__.py', line 602 in fill_template return self.fill_template_mako( filename, **kwargs ) File '/opt/galaxy-dist/lib/galaxy/web/framework/__init__.py', line 613 in fill_template_mako return template.render( **data ) File '/opt/galaxy-dist/eggs/py2.4-noplatform/Mako-0.2.5-py2.4.egg/mako/template.py', line 133 in render return runtime._render(self, self.callable_, args, data) File '/opt/galaxy-dist/eggs/py2.4-noplatform/Mako-0.2.5-py2.4.egg/mako/runtime.py', line 364 in _render _render_context(template, callable_, context, *args, **_kwargs_for_callable(callable_, data)) File '/opt/galaxy-dist/eggs/py2.4-noplatform/Mako-0.2.5-py2.4.egg/mako/runtime.py', line 381 in _render_context _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) File '/opt/galaxy-dist/eggs/py2.4-noplatform/Mako-0.2.5-py2.4.egg/mako/runtime.py', line 414 in _exec_template callable_(context, *args, **kwargs) File '/opt/galaxy-dist/database/compiled_templates/tool_form.mako.py', line 103 in render_body __M_writer(unicode(util.object_to_string( tool_state.encode( tool, app ) ))) File '/opt/galaxy-dist/lib/galaxy/tools/__init__.py', line 216 in encode a = hmac_new( app.config.tool_secret, value ) File '/opt/galaxy-dist/lib/galaxy/util/hash_util.py', line 33 in hmac_new return hmac.new( key, value, sha1 ).hexdigest() File '/usr/lib64/python2.4/hmac.py', line 107 in new return HMAC(key, msg, digestmod) File '/usr/lib64/python2.4/hmac.py', line 42 in __init__ self.outer = digestmod.new() AttributeError: 'builtin_function_or_method' object has no attribute 'new'
Hi Juan, After some searching it seems that it is an issue with python2.4 and hashlib. We recently introduced a change that might be responsible for this (commit 3311) and we'll look into it. Thanks, Kanwei On Fri, Feb 5, 2010 at 12:47 PM, Juan Perin <juanperin@gmail.com> wrote:
I've been working with a much older release of galaxy for a while, and it has worked great for the last few months. I noticed some progress with the NGS tools, so decided to attempt updating. My first mistake was in not knowing how to do so, so I essentially decided to start from the hg clone step and rebuild galaxy entirely in a new place on the same machine. I stupidly ran the database update script without listening to the warning about backing up the original galaxy db. So, my original instance won't work now... So, i'm trying to get the new instance working. Copied over my universe file and the custom .loc files from tool-data/ . everything seems to be ok, however I'm getting a python error that essentially repeats for anything I try to do, I'll paste the full output below for reference. I'm using python 2.4 Any ideas? Thanks in advance. URL: http://variome.chop.edu:8082/tool_runner?tool_id=bowtie_wrapper File '/opt/galaxy-dist/eggs/py2.4-noplatform/WebError-0.8a-py2.4.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/opt/galaxy-dist/eggs/py2.4-noplatform/Paste-1.6-py2.4.egg/paste/debug/prints.py', line 97 in __call__ status, headers, body = wsgilib.intercept_output( File '/opt/galaxy-dist/eggs/py2.4-noplatform/Paste-1.6-py2.4.egg/paste/wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/opt/galaxy-dist/eggs/py2.4-noplatform/Paste-1.6-py2.4.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/opt/galaxy-dist/eggs/py2.4-noplatform/Paste-1.6-py2.4.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/opt/galaxy-dist/lib/galaxy/web/framework/base.py', line 125 in __call__ body = method( trans, **kwargs ) File '/opt/galaxy-dist/lib/galaxy/web/controllers/tool_runner.py', line 61 in index return trans.fill_template( template, history=history, toolbox=toolbox, tool=tool, util=util, add_frame=add_frame, **vars ) File '/opt/galaxy-dist/lib/galaxy/web/framework/__init__.py', line 602 in fill_template return self.fill_template_mako( filename, **kwargs ) File '/opt/galaxy-dist/lib/galaxy/web/framework/__init__.py', line 613 in fill_template_mako return template.render( **data ) File '/opt/galaxy-dist/eggs/py2.4-noplatform/Mako-0.2.5-py2.4.egg/mako/template.py', line 133 in render return runtime._render(self, self.callable_, args, data) File '/opt/galaxy-dist/eggs/py2.4-noplatform/Mako-0.2.5-py2.4.egg/mako/runtime.py', line 364 in _render _render_context(template, callable_, context, *args, **_kwargs_for_callable(callable_, data)) File '/opt/galaxy-dist/eggs/py2.4-noplatform/Mako-0.2.5-py2.4.egg/mako/runtime.py', line 381 in _render_context _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) File '/opt/galaxy-dist/eggs/py2.4-noplatform/Mako-0.2.5-py2.4.egg/mako/runtime.py', line 414 in _exec_template callable_(context, *args, **kwargs) File '/opt/galaxy-dist/database/compiled_templates/tool_form.mako.py', line 103 in render_body __M_writer(unicode(util.object_to_string( tool_state.encode( tool, app ) ))) File '/opt/galaxy-dist/lib/galaxy/tools/__init__.py', line 216 in encode a = hmac_new( app.config.tool_secret, value ) File '/opt/galaxy-dist/lib/galaxy/util/hash_util.py', line 33 in hmac_new return hmac.new( key, value, sha1 ).hexdigest() File '/usr/lib64/python2.4/hmac.py', line 107 in new return HMAC(key, msg, digestmod) File '/usr/lib64/python2.4/hmac.py', line 42 in __init__ self.outer = digestmod.new() AttributeError: 'builtin_function_or_method' object has no attribute 'new'
_______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev
participants (5)
-
juan perin -
Juan Perin
-
Kanwei Li -
Kelly Vincent -
Nate Coraor