- galaxy-dev - lists.galaxyproject.org

DRMAA: TypeError: check_tool_output() takes exactly 5 arguments (4 given)
by Peter Cock 19 Sep '12

19 Sep '12

Hi all (and in particular, Scott), I've just updated my development server and found the following error when running jobs on our SGE cluster via DRMMA: galaxy.jobs.runners.drmaa ERROR 2012-09-18 09:43:20,698 Job wrapper finish method failed Traceback (most recent call last): File "/mnt/galaxy/galaxy-central/lib/galaxy/jobs/runners/drmaa.py", line 371, in finish_job drm_job_state.job_wrapper.finish( stdout, stderr, exit_code ) File "/mnt/galaxy/galaxy-central/lib/galaxy/jobs/__init__.py", line 1048, in finish if ( self.check_tool_output( stdout, stderr, tool_exit_code ) ): TypeError: check_tool_output() takes exactly 5 arguments (4 given) This looks to have been introduced in this commit: https://bitbucket.org/galaxy/galaxy-central/changeset/f557b7b05fdd701cbf99e… There should be an additional jobs argument, proposed fix: $ hg diff lib/galaxy/jobs/__init__.py diff -r 4007494e37e1 lib/galaxy/jobs/__init__.py --- a/lib/galaxy/jobs/__init__.py Tue Sep 18 09:40:19 2012 +0100 +++ b/lib/galaxy/jobs/__init__.py Tue Sep 18 10:06:44 2012 +0100 @@ -1045,7 +1045,8 @@ # Check what the tool returned. If the stdout or stderr matched # regular expressions that indicate errors, then set an error. # The same goes if the tool's exit code was in a given range. - if ( self.check_tool_output( stdout, stderr, tool_exit_code ) ): + job = self.get_job() + if ( self.check_tool_output( stdout, stderr, tool_exit_code, job ) ): task.state = task.states.OK else: task.state = task.states.ERROR (Let me know if you want this as a pull request - it seems a lot of effort for a tiny change.) Regards, Peter

3 10

creating a galaxy dataset within a galaxy tool
by Dan Tenenbaum 18 Sep '12

18 Sep '12

I'd like to have a tool that does not actually return anything (is that possible?) but writes a file to the directory where Galaxy stores uploaded data files, and tells Galaxy about it (perhaps that involves writing to a database?). My tool is written in R. Here is my scenario: I want the user to upload a text file, then pass that text file, along with several parameters, (in something we'll call Tool1) which I use to create a serialized R object. That object is then used in several other tools. I don't want the user to have to download and then re-upload the serialized object. I want it to just be available after the user runs Tool1. I realize that workflows could solve my problem here, but I want to keep the UI as simple as possible. For Tool2..ToolN, I don't want to have to always ask the user for the parameters for Tool1 as well as the current tool. So, is there a way for my tool to make Galaxy aware of an additional dataset? Thanks Dan

2 2

error uploading to galaxy from InterMine
by Fengyuan Hu 18 Sep '12

18 Sep '12

Hi Dan, We are trying to export some sequence to galaxy main from FlyMine but get the error: *Error executing tool: Unable to create output dataset: object store is full* After switching to galaxy test, it works. The same error occurred in other mines, could anyone in your team take a look? Thanks Fengyuan

2 2

Fix for tophat2 output filter on fusions
by Jim Johnson 18 Sep '12

18 Sep '12

The filter on the tophat2 fusions output wasn't being evaluated correctly when settingsType is preSet since the param dict wouldn't then have an entry for 'fusion_search' $ hg diff tools/ngs_rna/tophat2_wrapper.xml diff -r 3f12146d6d81 tools/ngs_rna/tophat2_wrapper.xml --- a/tools/ngs_rna/tophat2_wrapper.xml Tue Sep 18 14:10:56 2012 -0400 +++ b/tools/ngs_rna/tophat2_wrapper.xml Tue Sep 18 15:47:47 2012 -0500 @@ -297,7 +297,7 @@ <outputs> <data format="tabular" name="fusions" label="${tool.name} on ${on_string}: fusions" from_work_dir="tophat_out/fusions.out"> - <filter>(params['fusion_search']['do_search'] == 'Yes')</filter> + <filter>(params['settingsType'] == 'full' and params['fusion_search']['do_search'] == 'Yes')</filter> </data> <data format="bed" name="insertions" label="${tool.name} on ${on_string}: insertions" from_work_dir="tophat_out/insertions.bed"> <actions>

2 1

Tophat2 Issues
by Lance Parsons 18 Sep '12

18 Sep '12

I've run into one bug and a one other smaller issue with the Tophat2 wrapper. The bug manifests itself when running the tool with "use Defaults" selected results in the following error: cp: cannot stat `/data/galaxy-dev/galaxy-dev/database/job_working_directory/000/281/tophat_out/fusions.out': No such file or directory It seems that somehow, the filter on the output dataset is not working properly when the value (params['fusion_search']['do_search']) is not explicitly set in the parameters. The second issue is more minor. It seems that the Tophat2 wrapper assumes that Tophat2 will use Bowtie2 (and not Bowtie). However, the actual Tophat2 program will only use Bowtie2 if it is found in the path. Otherwise it defaults to using bowtie which results in an error (Could not find Bowtie index files) since the wrapper points the tool to the bowtie2 index instead. To make things a bit more robust, I suggest adding bowtie2 as a requirement to the tophat2 wrapper. It also might be a good idea to add versions to those requirements. I've attached a small patch to do that (though it's fairly trivial). Thanks. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University

2 1

Create a job
by Alfredo Guilherme Silva Souza 18 Sep '12

18 Sep '12

Hello, I need to create a job to return a file for it, only need to create a job beyond what is already created during the execution of my tool because the data they need to return is the result of a process that runs in the background. Can someone help me? Hugs. -- *Alfredo Guilherme* * *

1 0

Trackster caching
by Fields, Christopher J 18 Sep '12

18 Sep '12

We're seeing an odd issue that seems to be related to Trackster's caching of track information, mainly that memory in the web server processes increases. It appears that tracks are being cached in memory per web process. If I create a new visualization (and monitor our web processed via top) we can see the %mem usage go up, so much so that it eventually crashes with an out-of-mem issue. This seems to happen only when new visualizations are created; when they are shared between users mem usage stays roughly the same. We can bump up the mem on the VM (we have more at our disposal), and we can monitor and restart the processes if they get too high, but is there a way to determine whether Trackster is the cause? Below is the top output (note the manager and handlers are still normal-ish range): PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 27777 galaxy 20 0 2637m 1.5g 2700 S 6.3 19.8 7:29.01 python ./scripts/paster.py serve universe_wsgi.ini --server-name=web2 --pid-file=web2.pid --log-file=web2.log --daemon 27759 galaxy 20 0 3624m 2.2g 2716 S 5.3 28.9 9:04.33 python ./scripts/paster.py serve universe_wsgi.ini --server-name=web1 --pid-file=web1.pid --log-file=web1.log --daemon 27749 galaxy 20 0 2621m 1.5g 2724 S 5.0 19.7 7:39.51 python ./scripts/paster.py serve universe_wsgi.ini --server-name=web0 --pid-file=web0.pid --log-file=web0.log --daemon 27808 galaxy 20 0 1616m 160m 2640 S 1.7 2.0 3:00.21 python ./scripts/paster.py serve universe_wsgi.ini --server-name=handler1 --pid-file=handler1.pid --log-file=handler1.log --da 27798 galaxy 20 0 1616m 159m 2652 S 1.0 2.0 2:59.03 python ./scripts/paster.py serve universe_wsgi.ini --server-name=handler0 --pid-file=handler0.pid --log-file=handler0.log --da 27789 galaxy 20 0 944m 88m 2400 S 0.0 1.1 0:36.40 python ./scripts/paster.py serve universe_wsgi.ini --server-name=manager --pid-file=manager.pid --log-file=manager.log --d chris

2 2

Cuffmerge
by Jennifer Jackson 18 Sep '12

18 Sep '12

Hi Ken, You will need to install Cuffmerge and then link it in the same way that the other tools were added. This wiki has details: a link to the dependencies (versions) and how to configure paths/ENV set up: http://wiki.g2.bx.psu.edu/Admin/Config/Tool%20Dependencies If you need more help, please let us know. Please be sure to keep the galaxy-dev list on the cc - this group is great at troubleshooting local install issues. Thanks! Jen Galaxy team On 9/18/12 8:40 AM, Kenneth R. Auerbach wrote: > Hi Jen, > > I have another question. Right now we have some RNA Analysis tools > (cufflinks, cuffdiff, and cuffcompare) but not cuffmerge. Can you tell > me how I would add it to the NGS:RNA Analysis group in our Galaxy > installation? > > Thank you. > Ken. -- Jennifer Jackson http://galaxyproject.org

1 0

Redirect log files?
by Fields, Christopher J 18 Sep '12

18 Sep '12

Simple question: is there a way within universe_wsgi.ini to point to a specific location where Galaxy log (and possibly PID) files can be dropped? I've searched for this but couldn't find an answer beyond locally modifying run.sh, but it's entirely possible I'm missing something obvious. We want to have the logs stored on the server side in /var/log or /var/spool, not on our NFS share. The way run.sh is currently grabbing server information it seems as if we need to have something like this in the [server:*] config section: [server:/var/log/galaxy/web0] which doesn't seem right. Maybe something like the following (per server)? [server:web0] use = egg:Paste#http port = 8080 host = 127.0.0.1 use_threadpool = true threadpool_workers = 7 log_dir=/var/log/galaxy/ or simply (for all log and/or PID files): log_dir=/var/log/galaxy/ chris

3 4

Re: [galaxy-dev] environment variables and paths for toolshed tools
by Jim Johnson 18 Sep '12

18 Sep '12

Greg, Pablo, I'm using a SnpEffect repository 'snpeff_with_dep' in http://testtoolshed.g2.bx.psu.edu/ to test using tool_dependencies.xml with the <set_environment> tag. ( You both write access, if you want to use this for correcting my errors or debugging. ) I'm getting an error in prepare_for_install: Modulegalaxy.web.controllers.admin_toolshed:*1174*in|prepare_for_install| |>>>|||tool_dependencies|| |{'set_environment': [{'type': 'environment variable', 'name': 'JAVA_JAR_PATH'}], 'snpEff/3.0f': {'readme': '\n ', 'version': '3.0f', 'type': 'package', 'name': 'snpEff'}}| ... Module_admin_tool_shed_repository_common_mako:*242*in|render_render_tool_dependency_section| |>> <http://localhost:18080/admin_toolshed/prepare_for_install?tool_shed_url=htt…>name*=*requirements_dict*[*'name'*]*| *TypeError: list indices must be integers, not str* It appears that the requirements_dict for the set_environment tag is enclosed in an array. I haven't yet looked at the repository code to see how that might happen. Pablo, The http://sourceforge.net/projects/snpeff/files/ repository should probably have versions packaged for download named like: http://sourceforge.net/projects/snpeff/files/snpEff_3_0.tgz $ tar ztvf snpEff_3_0.tgz drwxrwxrwx 0 0 0 0 Aug 23 12:17 snpEff_3_0/ -rwxrwxrwx 0 0 0 14298767 Aug 23 12:17 snpEff_3_0/SnpSift.jar ... That will allow versioning within galaxy, and successive versions of tool_dependencies.xml with only need a change to the action tag: download_by_url Thanks, JJ I'm trying to test a repository that use in tool_depen > Message: 18 > Date: Thu, 13 Sep 2012 11:46:32 -0400 > From: Greg Von Kuster<greg(a)bx.psu.edu> > To: David Hoover<hooverdm(a)helix.nih.gov> > Cc:galaxy-dev@lists.bx.psu.edu > Subject: Re: [galaxy-dev] environment variables and paths for toolshed > tools > Message-ID:<F75D6320-3AF8-4BB0-8F31-B0F44AA1306B(a)bx.psu.edu> > Content-Type: text/plain; charset="us-ascii" > > Hello David, > > I've just committed change set 7656:6aadac8026cb to the Galaxy central repository that provides the ability to do what you need here. It would be great if you could try things out and let me know if you run into any problems. > > I've been working with nikhil-joshi's deseq_and_sam2counts repository in the main Galaxy tool shed because his tools require this new feature as well. This example should provide you with the information you'll need to tweak your tool shed repository so that your tool dependencies are located where they get installed rather than attempting to move them to ${GALAXY_DATA_INDEX_DIR} or some other location. > > Here is the tool_dependency.xml file entry for locating a directory referred to by an environment variable name R_SCRIPT_PATH. > > > <?xml version="1.0"?> > <tool_dependency> > <set_environment version="1.0"> > <environment_variable name="R_SCRIPT_PATH" action="set_to">$REPOSITORY_INSTALL_DIR</environment_variable> > </set_environment> > </tool_dependency> > > > The <set_enviroment> tag, is still supported inside <package> tag sets, but when defined at the xml root level, it will locate dependencies included in the installed tool shed repository. Of course, the above tool_dependencies.xml file could of course also include entries for tool dependencies that are packages, For example: > > > <?xml version="1.0"?> > <tool_dependency> > <set_environment version="1.0"> > <environment_variable name="R_SCRIPT_PATH" action="set_to">$REPOSITORY_INSTALL_DIR</environment_variable> > </set_environment> > <package name="R" version="2.15.1"> > <install version="1.0"> > <actions> > <action type="download_by_url">http://CRAN.R-project.org/src/base/R-2/R-2.15.1.tar.gz</action> > <action type="shell_command">./configure --prefix=$INSTALL_DIR</action> > <action type="shell_command">make</action> > <action type="set_environment"> > <environment_variable name="PATH" action="prepend_to">$INSTALL_DIR/bin</environment_variable> > </action> > </actions> > </install> > <readme> > You need a FORTRAN compiler or perhaps f2c in addition to a C compiler to build R. > </readme> > </package> > </tool_dependency> > > These tool dependency definitions are handled as described in the following section of the tool shed wiki: > > http://wiki.g2.bx.psu.edu/Tool%20Shed#Automatic_third-party_tool_dependency… > > So in order for the dependencies to be handled when the repository is installed, they must be defined in at least one of the <requirement> tag sets in at least 1 tool config in the repository. So, the <requirements> tag set in the deseq.xml tool config file will look something like this (there are more requirement tags defined here than in the tool_dependencies.xml fiel above, but you should get the idea). > > > <requirements> > <requirement type="set_environment">R_SCRIPT_PATH</requirement> > <requirement type="package" version="2.15.1">R</requirement> > <requirement type="package" version="2.10">Bioconductor</requirement> > <requirement type="package" version="1.8.3">DESeq</requirement> > <requirement type="package" version="1.24.0">aroma.light</requirement> > <requirement type="package" version="0.20-6">lattice</requirement> > </requirements> > > > The <command> tag set in the deseq.xml tool config is also slightly altered. Here is the way it looks before using this new approach: > > <command interpreter="python"> > stderr_wrapper.py Rscript ${GALAXY_DATA_INDEX_DIR}/deseq.R $counts $column_types $comparison $top_table $diagnostic_html "$diagnostic_html.files_path" "$counts.name" > </command> > > To use this new feature, the command string now uses the R_SCRIPT_PATH environment variable ( notice the required backslash to escape the $ ) > > > <command interpreter="python"> > stderr_wrapper.py Rscript \$R_SCRIPT_PATH/deseq.R $counts $column_types $comparison $top_table $diagnostic_html "$diagnostic_html.files_path" "$counts.name" > </command> > > When installed from the tool shed, a tool dependency object named R_SCRIPT_PATH will be created and associated with the installed repository. The dependency will have a pointer to the env.sh file that is created to set the value of the R_SCRIPT_PATH environment variable. > > Let me know if you bump into any issues in getting this working for your tools. > > Thanks! > > Greg Von Kuster > > > On Sep 12, 2012, at 1:00 PM, David Hoover wrote: > >> Are there any environment variables that are honored by toolshed installed tools? I tried creating a tool that uses ${GALAXY_DATA_INDEX_DIR} or $GALAXY_HOME, then uploaded it the test toolshed, then installed it automatically. Neither of these resolved to what I expected. I don't want to hard-code the path in the xml tool file, but rather have a default location for other executables and jar files. How should this best be done? >> >> David Hoover >> Helix Systems Staff >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> >> http://lists.bx.psu.edu/

2 2