October 2013 - galaxy-dev - lists.galaxyproject.org

unable to import run or save-to-file published workflow after galaxy upgrade
by Jennifer Jackson 09 Oct '13

09 Oct '13

Hi Alan, I do not personally know the solution for this, but am moving it over to the mailing list for local installs so that our dev team or community can help provide some feedback. Thanks! Jen Galaxy team -------- Original Message -------- Subject: [galaxy-user] unable to import run or save-to-file published workflow after galaxy upgrade Date: Tue, 8 Oct 2013 23:44:32 +0000 From: McCulloch, Alan <alan.mcculloch(a)agresearch.co.nz> To: galaxy-user(a)lists.bx.psu.edu <galaxy-user(a)lists.bx.psu.edu> dear all, we've just upgraded our Galaxy server (Galaxy revision 7148:17d57db9a7c0, upgraded to revision 10422:a886bc3ae924 ), and have found that an NGS training workflow that one of us set up and published is no longer accessible - we now get a stack trace when we try to run it (screenshot attached) ; if we try to save to file, we "get page not found" from the browser, but no errors reported in web logs, or Galaxy log as far as we can see Is there any way we can recover the workflow ? How to avoid this happening in future upgrades ? This time it is not a major problem, however we would not want this to happen to production workflows. Should we save-to-file on all workflows as a precaution before an upgrade ? (Would that help ? ) The workflow included steps as below. Grateful for any suggestions. Cheers Alan McC First Part : checking GC content .Upload the sampling.fasta file or get it from the shared data in Galaxy. .Use geecee from EMBOSS .Remove beginning (Text manipulation), first line .Convert as tabular (Text manipulation) or cut (Text manipulation) .Histogram (Graph/Display Data)on col2 .Compute data (Text manipulation) (data4 convert; col2>0.2) .Count data (statistics on col3) If it is worth it we can remove the sequences which have a too low GC content 3 Second part : sequence length .Compute Sequence length (FASTA manipulation) .Summary Statistics (Statistics)on col2 .Filter by length(FASTA manipulation) ; 800 .Line/world/character count(Text manipulation) .Blastn against fungi db : yeast, only one hit to show (advanced options ) .Count the lines to know how many sequences have a hit . (Text manipulation) 4 Bonus track : removing sequences with low GC content .Select (filter and sort) on data 6 where matching = True .FASTA-to-Tabular (FASTA manipulation) the input file sampling.fasta (2 col for the title) .Join two datasets(Join, Subtract and Group) on the columns c1 .Tabular-to-FASTA (fasta manipulation) of the previous results. .You can see how many sequences you took away (click on the dataset name in History) and it should correspond to the number of True in Count . 5

1 0

Filter and Sort Errors
by Mersee Madison-Villar 09 Oct '13

09 Oct '13

Hello... I have tried to filter my data to contain only those results with e-values of less than 0.05. I am given output...however, this only goes to 0.001. When I sort these same data, I see results between 0.001 and 0.05. However, even if I try to filter off of the sorted results, I still get only values <= 0.001. Can this be fixed? My user information is under the email account "mjmv(a)brandeis.edu". M -- M.J. Madison-Villar, PhD NSF Funded Postdoctoral Research Fellow Mueller Lab Research Group Colorado State University, Fort Collins, CO Fax: (781) 316-0249

1 0

Intergrate custom script with multiple outputs
by UMD Bioinformatics 09 Oct '13

09 Oct '13

Hello, I'm trying to incorporate a python tool that can accommodate multiple input files and the number of output files cannot be determined prior to running the tool. I've been though the help section at http://wiki.galaxyproject.org/Admin/Tools/Multiple%20Output%20Files#Number_… but I'm not getting any results. The tool isn't even running and creating the proper output in the tool directory. I've felt with the naming convention requited by Galaxy and the tool works from the command line. I know that the issue isn't with my local Galaxy install because a generic tool I've installed that produces multiple outputs works fine. I've attached my code and the XML file. Any help would be appreciated. Cheers Ian

3 5

Launching CloudMan in Other AWS Regions than US East
by Ryota Yamanaka 09 Oct '13

09 Oct '13

Dear Galaxy Team, We would like to benchmark CloudMan with NGS data in Tokyo, and we are wondering if it is possible to launch it at AWS's Tokyo DC in order to facilitate large-size data transfer and low-latency response. However, CloudMan AMI is currently available only at Virginia DC according to Wiki: > ... set your AWS Region to US East (Virginia). This is the only > region Galaxy CloudMan is fully supported in at this time ... http://wiki.galaxyproject.org/CloudMan/AWS/GettingStarted 1) Do you plan to clone CloudMan AMI to other regions outside the US in the near future? 2) Or, is there any way we can build CloudMan in such regions, with or without your support? Sincerely, Ryota Yamanaka Genome Science, RCAST, The University of Tokyo

2 2

2 RNAseq files in Main queue 1 week to map w/Top hat
by Theresa Stueve 09 Oct '13

09 Oct '13

Greetings, I have had two RNAseq files in queue to map with Top hat in Galaxy Main since last Thursday (History began Sept 19). I got word that NGS was stalled, my jobs were mistakenly believed to be moving afterwards when they weren't- but are now under review again. It has been a week now and the 2 mapping jobs are still waiting to run. Can you tell me whether I should delete the jobs and re-run them as new? Is there a problem with my trimmed dataset? My concern is that these jobs are still "believed to be moving" as was indicated by one of your galaxy stewards, but are actually still stalled in space. One week is a long tme to wait for local jobs, no? thank you very much for your help https://main.g2.bx.psu.edu/u/trstueve/h/rnaseqjegcsc3wkrun130924-1 <https://main.g2.bx.psu.edu/history/sharing#> Theresa Ryan Stueve 510-225-8894 (text better than voicemail for quick response) -- Theresa Ryan Stueve 510-225-8894 (text better than voicemail for quick response)

2 1

Migrating database from development back to stable?
by Guest, Simon 08 Oct '13

08 Oct '13

Dear Galaxy developers, Currently our production Galaxy runs the stable branch. However, I see one or two reasons to want to run the development (default) branch for a while. This will involved migrating the database to a later version, which is fine, as migration scripts exists. What I am wondering is whether it is guaranteed that I will be able to get back onto stable at a later date, i.e. there will be a database migration path from whatever development database version I end up on, onto a future stable database version. Can anyone comment? cheers, Simon

2 1

Path problems with installation on the Test ToolShed?
by Peter Cock 08 Oct '13

08 Oct '13

Title was: [galaxy-iuc] Platform-specific tool dependency support has been enhanced. On Fri, Oct 4, 2013 at 6:08 PM, Dave Bouvier <dave(a)bx.psu.edu> wrote: > Peter, > > Thank you for reporting that error, I've committed a fix in > 10791:6e063ca1960f. > > --Dave B. Hi Dave, The new platform specific actions seem to be working nicely for the BLAST+ binaries (see other thread), however there may be some lingering regressions due to being in the wrong directory after downloading files (further to the examples Bjoern and I raised last week). These 3 were all working in the past (I would have reported them earlier but they fell under the most recent round of "missing test results"). Thanks, Peter ---------------------------------------------------- http://testtoolshed.g2.bx.psu.edu/view/peterjc/nlstradamus/81cac85fc286 Tests that failed Tool id: nlstradamus Tool version: nlstradamus ... Fatal error: Exit code 127 () /bin/sh: 1: NLStradamus: not found ---------------------------------------------------- http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3/1e681e19e35c Tests that failed Tool id: effectiveT3 Tool version: effectiveT3 ... Fatal error: Exit code 1 () Contents of '/ToolDependencies/effectiveT3/1.0.1/peterjc/effectivet3/1e681e19e35c/module' is Main JAR was found: '/ToolDependencies/effectiveT3/1.0.1/peterjc/effectivet3/1e681e19e35c/TTSS_GUI-1.0.1.jar' Effective T3 model JAR file not found: '/ToolDependencies/effectiveT3/1.0.1/peterjc/effectivet3/1e681e19e35c/module/TTSS_ANIMAL-1.0.1.jar' ... Stderr: Fatal error: Exit code 1 () Contents of '/ToolDependencies/effectiveT3/1.0.1/peterjc/effectivet3/1e681e19e35c/module' is Main JAR was found: '/ToolDependencies/effectiveT3/1.0.1/peterjc/effectivet3/1e681e19e35c/TTSS_GUI-1.0.1.jar' Effective T3 model JAR file not found: '/ToolDependencies/effectiveT3/1.0.1/peterjc/effectivet3/1e681e19e35c/module/TTSS_PLANT-1.0.1.jar' ---------------------------------------------------- http://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod/0d39248a7a9f Tests that failed Tool id: clinod Tool version: clinod ... Stderr: Fatal error: Exit code 1 () Cannot find a Batchman executable! Batchman does not exist in /ToolDependencies/clinod/1.3/peterjc/clinod/0d39248a7a9f/batchman

2 2

Galaxy code run on main Tool Shed vs Test Tool Shed?
by Peter Cock 08 Oct '13

08 Oct '13

Splitting off from this thread: http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-October/016903.html On Mon, Oct 7, 2013 at 4:24 PM, Dave Bouvier <dave(a)bx.psu.edu> wrote: > Peter, > > The platform detection code is not yet in the stable branch, but is intended > to be included in the upcoming release. The automated testing framework does > run the default branch, though, so the nightly tests will make use of the > new definitions. > > --Dave B. Thank you for your reply Dave, but I hope there has been a miscommunication here. The "Test Tool Shed" is (as I understand it) effectively running galaxy-central (presumably updated daily or something like that), so it tests the latest and greatest Galaxy code-base and all the new Tool Shed features like the new <action> enhancements for tool dependency installation. That's fine as a live testbed for Galaxy and the Tool Shed itself (although not always ideal for us tool developers). I had been under the impression that the main "Tool Shed" (and the nightly tool tests run from it) was tracking the stable branch on galaxy-dist and therefore equivalent to what most Galaxy instances are running. Are you saying that for both the main "Tool Shed" and the "Test Tool Shed" you are using the galaxy-dist default branch with the bleeding edge code to run the tests? If so, this worries me as that means the main "Tool Shed" nightly tests are not indicative of what would happen once the tools are downloaded and installed to a galaxy-dist stable branch instance. e.g. If I was to release a tool update using the latest <action> work then I would expect it to work fine on the "Test Tool Shed", but not to work on a Galaxy instance running the current stable release. Therefore, I would hope/expect it to fail on the main "Tool Shed" (due to using features not yet in the stable release). (The <action> enhancement is just an example, this is a general question.) Could you clarify the situation please? Thanks, Peter

3 5

Adding a tool to Galaxy Main Server
by Hamid Reza Hassanzadeh 07 Oct '13

07 Oct '13

Hi everybody, We are going to add our gene prediction tool to Galaxy Main server, anybody know whom we should contact? Thanks -- Hamid Reza Hassanzadeh, PhD Student & Graduate Research Assistant, Bioinformatics Lab, Center for Bioinformatics and Computational Genomics Joint Georgia Tech and Emory Wallace H Coulter Department of Biomedical Engineering, Department of Computer Science at Georgia Institute of Technology office: KACB 1343

2 1

Dynamic job runner not being dispatched for a tool
by Ganote, Carrie L 07 Oct '13

07 Oct '13

Hi List, I'm a bit baffled by a new tool I've tried to add to Galaxy. The job runner does not dispatch it correctly. I added the tool via the toolshed, set up the executable on the cluster side, and attempted to hook it up to the dynamic job runner. The destination always ended up set as local. Works fine if the default runner is pbs, but if it's local by default, it will only dispatch locally. Attempting to work backwards, I copied the wrapper to a local tool directory, and pared everything down to a minimum to run. There must be something missing. My trinity_all dynamic definition works fine, Newbler doesn't. Is there anything obvious here? ~~~~~tool_conf.xml~~~~~~~~~~~~~~~~~~~~ <?xml version="1.0"?> <toolbox> <section name="Assembly" id="assembly"> <tool file="sr_assembly/runAssembly.xml" /> <tool file="ngs_rna/trinity_all.xml" /> </section>  <section name="Send Data" id="send"> <tool file="data_destination/epigraph.xml" /> <tool file="data_destination/epigraph_test.xml" /> <tool file="genomespace/genomespace_exporter.xml" /> </section> </toolbox> ~~~~~tools/sr_assembly/runAssembly.xml~~~~~~~~~~~~~~~~~~~~ <tool id="runAssembly" name="runAssembly" version="1.0.1"> <description>De novo assembly of Roche/454 reads using Newbler</description> <command> runAssembly $sanger_input -o $newbler_metrics </command> <inputs>  <param name="sanger_input" type="data" format="fasta,fastqsanger" label="SE Fasta/Fastq file"/>  <param name="walltime" type="select" label="How long will your job need"> <option selected="true" value="1">8 hrs</option> <option value="48">48 hrs</option> </param> </inputs> <outputs> <data name="newbler_metrics" format="txt" /> </outputs> </tool> ~~~~~job_conf.xml~~~~~~~~~~~~~~~~~~~~ <?xml version="1.0"?> <job_conf> <plugins workers="4"> <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" /> <plugin id="pbs" type="runner" load="galaxy.jobs.runners.pbs:PBSJobRunner" /> </plugins> <handlers default="handlers"> <handler id="handler0" tags="handlers"/> </handlers> <destinations default="local"> <destination id="local" runner="local"/> <destination id="pbs_normal" runner="pbs" tags="mycluster"> <param id="Resource_List">nodes=1:ppn=4,vmem=32gb,walltime=24:00:00</param> </destination> <destination id="newbler_d" runner="dynamic"> <param id="type">python</param> <param id="function">newbler</param> </destination> <destination id="trinity_d" runner="dynamic"> <param id="type">python</param> <param id="function">trinity_all</param> </destination> </destinations> <tools> <tool id="runAssembly" destination="newbler_d"/> <tool id="trinity_all" destination="trinity_d"/> </tools> <limits> </limits> </job_conf> ~~~~~lib/galaxy/jobs/rules/dynamicjobs.py~~~~~~~~~~~~~~~~~~~~ import os from galaxy.jobs import JobDestination def newbler(job): incoming = dict( [ ( p.name, p.value ) for p in job.parameters ] ) walltime_para = incoming["walltime"] walltime = int(walltime_para.replace("\"","")) params = {} string = "vmem=500gb,nodes=1:ppn=32,walltime=%d:00:00"%walltime params["Resource_List"] = string return JobDestination(runner="pbs", params=params) def trinity_all(job): incoming = dict( [ ( p.name, p.value ) for p in job.parameters ] ) walltime_para = incoming["walltime"] walltime = int(walltime_para.replace("\"","")) params = {} string = "vmem=500gb,nodes=1:ppn=32,walltime=%d:00:00"%walltime params["Resource_List"] = string return JobDestination(runner="pbs", params=params) I re-installed the Roche 454 toolsuite from the toolshed once more to see if it just needed a kick. I copied the tool id into my job_conf file to make sure I made no mistakes on the id. I tried to send it to dynamic job runner and to pbs runner. I pulled and merged this galaxy about a month ago, so it should be mostly up to date - but I've had so many issues that I may have to just re-install from scratch. Thanks a ton for any advice, Carrie Ganote

3 3