- galaxy-dev - lists.galaxyproject.org

2 RNAseq files in Main queue 1 week to map w/Top hat
by Theresa Stueve 09 Oct '13

09 Oct '13

Greetings, I have had two RNAseq files in queue to map with Top hat in Galaxy Main since last Thursday (History began Sept 19). I got word that NGS was stalled, my jobs were mistakenly believed to be moving afterwards when they weren't- but are now under review again. It has been a week now and the 2 mapping jobs are still waiting to run. Can you tell me whether I should delete the jobs and re-run them as new? Is there a problem with my trimmed dataset? My concern is that these jobs are still "believed to be moving" as was indicated by one of your galaxy stewards, but are actually still stalled in space. One week is a long tme to wait for local jobs, no? thank you very much for your help https://main.g2.bx.psu.edu/u/trstueve/h/rnaseqjegcsc3wkrun130924-1 <https://main.g2.bx.psu.edu/history/sharing#> Theresa Ryan Stueve 510-225-8894 (text better than voicemail for quick response) -- Theresa Ryan Stueve 510-225-8894 (text better than voicemail for quick response)

2 1

Migrating database from development back to stable?
by Guest, Simon 08 Oct '13

08 Oct '13

Dear Galaxy developers, Currently our production Galaxy runs the stable branch. However, I see one or two reasons to want to run the development (default) branch for a while. This will involved migrating the database to a later version, which is fine, as migration scripts exists. What I am wondering is whether it is guaranteed that I will be able to get back onto stable at a later date, i.e. there will be a database migration path from whatever development database version I end up on, onto a future stable database version. Can anyone comment? cheers, Simon

2 1

Path problems with installation on the Test ToolShed?
by Peter Cock 08 Oct '13

08 Oct '13

Title was: [galaxy-iuc] Platform-specific tool dependency support has been enhanced. On Fri, Oct 4, 2013 at 6:08 PM, Dave Bouvier <dave(a)bx.psu.edu> wrote: > Peter, > > Thank you for reporting that error, I've committed a fix in > 10791:6e063ca1960f. > > --Dave B. Hi Dave, The new platform specific actions seem to be working nicely for the BLAST+ binaries (see other thread), however there may be some lingering regressions due to being in the wrong directory after downloading files (further to the examples Bjoern and I raised last week). These 3 were all working in the past (I would have reported them earlier but they fell under the most recent round of "missing test results"). Thanks, Peter ---------------------------------------------------- http://testtoolshed.g2.bx.psu.edu/view/peterjc/nlstradamus/81cac85fc286 Tests that failed Tool id: nlstradamus Tool version: nlstradamus ... Fatal error: Exit code 127 () /bin/sh: 1: NLStradamus: not found ---------------------------------------------------- http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3/1e681e19e35c Tests that failed Tool id: effectiveT3 Tool version: effectiveT3 ... Fatal error: Exit code 1 () Contents of '/ToolDependencies/effectiveT3/1.0.1/peterjc/effectivet3/1e681e19e35c/module' is Main JAR was found: '/ToolDependencies/effectiveT3/1.0.1/peterjc/effectivet3/1e681e19e35c/TTSS_GUI-1.0.1.jar' Effective T3 model JAR file not found: '/ToolDependencies/effectiveT3/1.0.1/peterjc/effectivet3/1e681e19e35c/module/TTSS_ANIMAL-1.0.1.jar' ... Stderr: Fatal error: Exit code 1 () Contents of '/ToolDependencies/effectiveT3/1.0.1/peterjc/effectivet3/1e681e19e35c/module' is Main JAR was found: '/ToolDependencies/effectiveT3/1.0.1/peterjc/effectivet3/1e681e19e35c/TTSS_GUI-1.0.1.jar' Effective T3 model JAR file not found: '/ToolDependencies/effectiveT3/1.0.1/peterjc/effectivet3/1e681e19e35c/module/TTSS_PLANT-1.0.1.jar' ---------------------------------------------------- http://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod/0d39248a7a9f Tests that failed Tool id: clinod Tool version: clinod ... Stderr: Fatal error: Exit code 1 () Cannot find a Batchman executable! Batchman does not exist in /ToolDependencies/clinod/1.3/peterjc/clinod/0d39248a7a9f/batchman

2 2

Galaxy code run on main Tool Shed vs Test Tool Shed?
by Peter Cock 08 Oct '13

08 Oct '13

Splitting off from this thread: http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-October/016903.html On Mon, Oct 7, 2013 at 4:24 PM, Dave Bouvier <dave(a)bx.psu.edu> wrote: > Peter, > > The platform detection code is not yet in the stable branch, but is intended > to be included in the upcoming release. The automated testing framework does > run the default branch, though, so the nightly tests will make use of the > new definitions. > > --Dave B. Thank you for your reply Dave, but I hope there has been a miscommunication here. The "Test Tool Shed" is (as I understand it) effectively running galaxy-central (presumably updated daily or something like that), so it tests the latest and greatest Galaxy code-base and all the new Tool Shed features like the new <action> enhancements for tool dependency installation. That's fine as a live testbed for Galaxy and the Tool Shed itself (although not always ideal for us tool developers). I had been under the impression that the main "Tool Shed" (and the nightly tool tests run from it) was tracking the stable branch on galaxy-dist and therefore equivalent to what most Galaxy instances are running. Are you saying that for both the main "Tool Shed" and the "Test Tool Shed" you are using the galaxy-dist default branch with the bleeding edge code to run the tests? If so, this worries me as that means the main "Tool Shed" nightly tests are not indicative of what would happen once the tools are downloaded and installed to a galaxy-dist stable branch instance. e.g. If I was to release a tool update using the latest <action> work then I would expect it to work fine on the "Test Tool Shed", but not to work on a Galaxy instance running the current stable release. Therefore, I would hope/expect it to fail on the main "Tool Shed" (due to using features not yet in the stable release). (The <action> enhancement is just an example, this is a general question.) Could you clarify the situation please? Thanks, Peter

3 5

Adding a tool to Galaxy Main Server
by Hamid Reza Hassanzadeh 07 Oct '13

07 Oct '13

Hi everybody, We are going to add our gene prediction tool to Galaxy Main server, anybody know whom we should contact? Thanks -- Hamid Reza Hassanzadeh, PhD Student & Graduate Research Assistant, Bioinformatics Lab, Center for Bioinformatics and Computational Genomics Joint Georgia Tech and Emory Wallace H Coulter Department of Biomedical Engineering, Department of Computer Science at Georgia Institute of Technology office: KACB 1343

2 1

Dynamic job runner not being dispatched for a tool
by Ganote, Carrie L 07 Oct '13

07 Oct '13

Hi List, I'm a bit baffled by a new tool I've tried to add to Galaxy. The job runner does not dispatch it correctly. I added the tool via the toolshed, set up the executable on the cluster side, and attempted to hook it up to the dynamic job runner. The destination always ended up set as local. Works fine if the default runner is pbs, but if it's local by default, it will only dispatch locally. Attempting to work backwards, I copied the wrapper to a local tool directory, and pared everything down to a minimum to run. There must be something missing. My trinity_all dynamic definition works fine, Newbler doesn't. Is there anything obvious here? ~~~~~tool_conf.xml~~~~~~~~~~~~~~~~~~~~ <?xml version="1.0"?> <toolbox> <section name="Assembly" id="assembly"> <tool file="sr_assembly/runAssembly.xml" /> <tool file="ngs_rna/trinity_all.xml" /> </section>  <section name="Send Data" id="send"> <tool file="data_destination/epigraph.xml" /> <tool file="data_destination/epigraph_test.xml" /> <tool file="genomespace/genomespace_exporter.xml" /> </section> </toolbox> ~~~~~tools/sr_assembly/runAssembly.xml~~~~~~~~~~~~~~~~~~~~ <tool id="runAssembly" name="runAssembly" version="1.0.1"> <description>De novo assembly of Roche/454 reads using Newbler</description> <command> runAssembly $sanger_input -o $newbler_metrics </command> <inputs>  <param name="sanger_input" type="data" format="fasta,fastqsanger" label="SE Fasta/Fastq file"/>  <param name="walltime" type="select" label="How long will your job need"> <option selected="true" value="1">8 hrs</option> <option value="48">48 hrs</option> </param> </inputs> <outputs> <data name="newbler_metrics" format="txt" /> </outputs> </tool> ~~~~~job_conf.xml~~~~~~~~~~~~~~~~~~~~ <?xml version="1.0"?> <job_conf> <plugins workers="4"> <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" /> <plugin id="pbs" type="runner" load="galaxy.jobs.runners.pbs:PBSJobRunner" /> </plugins> <handlers default="handlers"> <handler id="handler0" tags="handlers"/> </handlers> <destinations default="local"> <destination id="local" runner="local"/> <destination id="pbs_normal" runner="pbs" tags="mycluster"> <param id="Resource_List">nodes=1:ppn=4,vmem=32gb,walltime=24:00:00</param> </destination> <destination id="newbler_d" runner="dynamic"> <param id="type">python</param> <param id="function">newbler</param> </destination> <destination id="trinity_d" runner="dynamic"> <param id="type">python</param> <param id="function">trinity_all</param> </destination> </destinations> <tools> <tool id="runAssembly" destination="newbler_d"/> <tool id="trinity_all" destination="trinity_d"/> </tools> <limits> </limits> </job_conf> ~~~~~lib/galaxy/jobs/rules/dynamicjobs.py~~~~~~~~~~~~~~~~~~~~ import os from galaxy.jobs import JobDestination def newbler(job): incoming = dict( [ ( p.name, p.value ) for p in job.parameters ] ) walltime_para = incoming["walltime"] walltime = int(walltime_para.replace("\"","")) params = {} string = "vmem=500gb,nodes=1:ppn=32,walltime=%d:00:00"%walltime params["Resource_List"] = string return JobDestination(runner="pbs", params=params) def trinity_all(job): incoming = dict( [ ( p.name, p.value ) for p in job.parameters ] ) walltime_para = incoming["walltime"] walltime = int(walltime_para.replace("\"","")) params = {} string = "vmem=500gb,nodes=1:ppn=32,walltime=%d:00:00"%walltime params["Resource_List"] = string return JobDestination(runner="pbs", params=params) I re-installed the Roche 454 toolsuite from the toolshed once more to see if it just needed a kick. I copied the tool id into my job_conf file to make sure I made no mistakes on the id. I tried to send it to dynamic job runner and to pbs runner. I pulled and merged this galaxy about a month ago, so it should be mostly up to date - but I've had so many issues that I may have to just re-install from scratch. Thanks a ton for any advice, Carrie Ganote

3 3

GA file format documentation
by Philip Mabon 07 Oct '13

07 Oct '13

Is there any documentation on how exported workflow with '.ga' extension are structured? We are looking for the ability to create our workflows without using the workflow editor thru the web portal. Thanks Takadonet

2 3

error checking active jobs : KeyError: 'destination'
by Ganote, Carrie L 07 Oct '13

07 Oct '13

Hi List, I'm having this same issue - if I dispatch a job to dynamic job runner, and it then hands the job to pbs, then it will flip out if I restart the server. In my case, though, there is no error in the job state - it is sitting in the Torque queue where it should be. As far as I can tell, galaxy successfully reloads the job back into the history. But it falters on recovering this destination for pbs. If I delete the queued jobs, it quiets down, but this is not a sustainable solution - I need to be able to reboot the server without all of my users' cluster jobs causing faults every second. Log: galaxy.jobs.runners.pbs DEBUG 2013-10-04 15:50:32,454 Set default PBS server to m1.mason.indiana.edu galaxy.jobs.runners DEBUG 2013-10-04 15:50:32,455 Starting 4 PBSRunner workers galaxy.jobs DEBUG 2013-10-04 15:50:32,459 Loaded job runner 'galaxy.jobs.runners.pbs:PBSJobRunner' as 'pbs' galaxy.jobs.handler DEBUG 2013-10-04 15:50:32,459 Loaded job runners plugins: local:pbs galaxy.jobs.handler INFO 2013-10-04 15:50:32,460 job handler stop queue started galaxy.jobs DEBUG 2013-10-04 15:50:32,478 (514) Working directory for job is: /N/dc/projects/galaxy/job_working_directory/000/514 galaxy.jobs.handler DEBUG 2013-10-04 15:50:32,478 recovering job 514 in pbs runner galaxy.jobs WARNING 2013-10-04 15:50:32,478 (514) Job runner URLs are deprecated, use destinations instead. galaxy.jobs.runners.pbs DEBUG 2013-10-04 15:50:32,479 (514/176938.m1.mason) is still in PBS queued state, adding to the PBS queue galaxy.jobs.handler INFO 2013-10-04 15:50:32,485 job handler queue started ... galaxy.jobs.runners ERROR 2013-10-04 15:50:33,456 Unhandled exception checking active jobs Traceback (most recent call last): File "/N/hd03/galaxy/Mason/galaxy-ncgas/lib/galaxy/jobs/runners/__init__.py", line 362, in monitor self.check_watched_items() File "/N/hd03/galaxy/Mason/galaxy-ncgas/lib/galaxy/jobs/runners/pbs.py", line 385, in check_watched_items ( failures, statuses ) = self.check_all_jobs() File "/N/hd03/galaxy/Mason/galaxy-ncgas/lib/galaxy/jobs/runners/pbs.py", line 466, in check_all_jobs pbs_server_name = self.__get_pbs_server(pbs_job_state.job_destination.params) File "/N/hd03/galaxy/Mason/galaxy-ncgas/lib/galaxy/jobs/runners/pbs.py", line 222, in __get_pbs_server return job_destination_params['destination'].split('@')[-1] KeyError: 'destination' Thanks so much for wisdom on this. Sincerely, Carrie Ganote

1 0

fastx_toolkit not recognizing fastq files
by UMD Bioinformatics 07 Oct '13

07 Oct '13

Hello, I've installed Galaxy on two separate machines. None of the fastx tools recognize my fastq files. They are not appearing in the drop down list on Galaxy for input options. I'm getting the same behavior on both machines. I've used the tool_shed to install fastx on one machine and I've done it manually on the other. fastx_tools work perfectly from the command line. Cheers Ian

2 2

GA file format documentation
by Philip Mabon 07 Oct '13

07 Oct '13

Is there any documentation on how exported workflow with '.ga' extension are structured? We are looking for the ability to create our workflows without using the workflow editor thru the web portal. Thanks

1 0