- galaxy-dev - lists.galaxyproject.org

Migrating database from development back to stable?
by Guest, Simon 08 Oct '13

08 Oct '13

Dear Galaxy developers, Currently our production Galaxy runs the stable branch. However, I see one or two reasons to want to run the development (default) branch for a while. This will involved migrating the database to a later version, which is fine, as migration scripts exists. What I am wondering is whether it is guaranteed that I will be able to get back onto stable at a later date, i.e. there will be a database migration path from whatever development database version I end up on, onto a future stable database version. Can anyone comment? cheers, Simon

2 1

Path problems with installation on the Test ToolShed?
by Peter Cock 08 Oct '13

08 Oct '13

Title was: [galaxy-iuc] Platform-specific tool dependency support has been enhanced. On Fri, Oct 4, 2013 at 6:08 PM, Dave Bouvier <dave(a)bx.psu.edu> wrote: > Peter, > > Thank you for reporting that error, I've committed a fix in > 10791:6e063ca1960f. > > --Dave B. Hi Dave, The new platform specific actions seem to be working nicely for the BLAST+ binaries (see other thread), however there may be some lingering regressions due to being in the wrong directory after downloading files (further to the examples Bjoern and I raised last week). These 3 were all working in the past (I would have reported them earlier but they fell under the most recent round of "missing test results"). Thanks, Peter ---------------------------------------------------- http://testtoolshed.g2.bx.psu.edu/view/peterjc/nlstradamus/81cac85fc286 Tests that failed Tool id: nlstradamus Tool version: nlstradamus ... Fatal error: Exit code 127 () /bin/sh: 1: NLStradamus: not found ---------------------------------------------------- http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3/1e681e19e35c Tests that failed Tool id: effectiveT3 Tool version: effectiveT3 ... Fatal error: Exit code 1 () Contents of '/ToolDependencies/effectiveT3/1.0.1/peterjc/effectivet3/1e681e19e35c/module' is Main JAR was found: '/ToolDependencies/effectiveT3/1.0.1/peterjc/effectivet3/1e681e19e35c/TTSS_GUI-1.0.1.jar' Effective T3 model JAR file not found: '/ToolDependencies/effectiveT3/1.0.1/peterjc/effectivet3/1e681e19e35c/module/TTSS_ANIMAL-1.0.1.jar' ... Stderr: Fatal error: Exit code 1 () Contents of '/ToolDependencies/effectiveT3/1.0.1/peterjc/effectivet3/1e681e19e35c/module' is Main JAR was found: '/ToolDependencies/effectiveT3/1.0.1/peterjc/effectivet3/1e681e19e35c/TTSS_GUI-1.0.1.jar' Effective T3 model JAR file not found: '/ToolDependencies/effectiveT3/1.0.1/peterjc/effectivet3/1e681e19e35c/module/TTSS_PLANT-1.0.1.jar' ---------------------------------------------------- http://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod/0d39248a7a9f Tests that failed Tool id: clinod Tool version: clinod ... Stderr: Fatal error: Exit code 1 () Cannot find a Batchman executable! Batchman does not exist in /ToolDependencies/clinod/1.3/peterjc/clinod/0d39248a7a9f/batchman

2 2

Galaxy code run on main Tool Shed vs Test Tool Shed?
by Peter Cock 08 Oct '13

08 Oct '13

Splitting off from this thread: http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-October/016903.html On Mon, Oct 7, 2013 at 4:24 PM, Dave Bouvier <dave(a)bx.psu.edu> wrote: > Peter, > > The platform detection code is not yet in the stable branch, but is intended > to be included in the upcoming release. The automated testing framework does > run the default branch, though, so the nightly tests will make use of the > new definitions. > > --Dave B. Thank you for your reply Dave, but I hope there has been a miscommunication here. The "Test Tool Shed" is (as I understand it) effectively running galaxy-central (presumably updated daily or something like that), so it tests the latest and greatest Galaxy code-base and all the new Tool Shed features like the new <action> enhancements for tool dependency installation. That's fine as a live testbed for Galaxy and the Tool Shed itself (although not always ideal for us tool developers). I had been under the impression that the main "Tool Shed" (and the nightly tool tests run from it) was tracking the stable branch on galaxy-dist and therefore equivalent to what most Galaxy instances are running. Are you saying that for both the main "Tool Shed" and the "Test Tool Shed" you are using the galaxy-dist default branch with the bleeding edge code to run the tests? If so, this worries me as that means the main "Tool Shed" nightly tests are not indicative of what would happen once the tools are downloaded and installed to a galaxy-dist stable branch instance. e.g. If I was to release a tool update using the latest <action> work then I would expect it to work fine on the "Test Tool Shed", but not to work on a Galaxy instance running the current stable release. Therefore, I would hope/expect it to fail on the main "Tool Shed" (due to using features not yet in the stable release). (The <action> enhancement is just an example, this is a general question.) Could you clarify the situation please? Thanks, Peter

3 5

Adding a tool to Galaxy Main Server
by Hamid Reza Hassanzadeh 08 Oct '13

08 Oct '13

Hi everybody, We are going to add our gene prediction tool to Galaxy Main server, anybody know whom we should contact? Thanks -- Hamid Reza Hassanzadeh, PhD Student & Graduate Research Assistant, Bioinformatics Lab, Center for Bioinformatics and Computational Genomics Joint Georgia Tech and Emory Wallace H Coulter Department of Biomedical Engineering, Department of Computer Science at Georgia Institute of Technology office: KACB 1343

2 1

Dynamic job runner not being dispatched for a tool
by Ganote, Carrie L 07 Oct '13

07 Oct '13

Hi List, I'm a bit baffled by a new tool I've tried to add to Galaxy. The job runner does not dispatch it correctly. I added the tool via the toolshed, set up the executable on the cluster side, and attempted to hook it up to the dynamic job runner. The destination always ended up set as local. Works fine if the default runner is pbs, but if it's local by default, it will only dispatch locally. Attempting to work backwards, I copied the wrapper to a local tool directory, and pared everything down to a minimum to run. There must be something missing. My trinity_all dynamic definition works fine, Newbler doesn't. Is there anything obvious here? ~~~~~tool_conf.xml~~~~~~~~~~~~~~~~~~~~ <?xml version="1.0"?> <toolbox> <section name="Assembly" id="assembly"> <tool file="sr_assembly/runAssembly.xml" /> <tool file="ngs_rna/trinity_all.xml" /> </section>  <section name="Send Data" id="send"> <tool file="data_destination/epigraph.xml" /> <tool file="data_destination/epigraph_test.xml" /> <tool file="genomespace/genomespace_exporter.xml" /> </section> </toolbox> ~~~~~tools/sr_assembly/runAssembly.xml~~~~~~~~~~~~~~~~~~~~ <tool id="runAssembly" name="runAssembly" version="1.0.1"> <description>De novo assembly of Roche/454 reads using Newbler</description> <command> runAssembly $sanger_input -o $newbler_metrics </command> <inputs>  <param name="sanger_input" type="data" format="fasta,fastqsanger" label="SE Fasta/Fastq file"/>  <param name="walltime" type="select" label="How long will your job need"> <option selected="true" value="1">8 hrs</option> <option value="48">48 hrs</option> </param> </inputs> <outputs> <data name="newbler_metrics" format="txt" /> </outputs> </tool> ~~~~~job_conf.xml~~~~~~~~~~~~~~~~~~~~ <?xml version="1.0"?> <job_conf> <plugins workers="4"> <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" /> <plugin id="pbs" type="runner" load="galaxy.jobs.runners.pbs:PBSJobRunner" /> </plugins> <handlers default="handlers"> <handler id="handler0" tags="handlers"/> </handlers> <destinations default="local"> <destination id="local" runner="local"/> <destination id="pbs_normal" runner="pbs" tags="mycluster"> <param id="Resource_List">nodes=1:ppn=4,vmem=32gb,walltime=24:00:00</param> </destination> <destination id="newbler_d" runner="dynamic"> <param id="type">python</param> <param id="function">newbler</param> </destination> <destination id="trinity_d" runner="dynamic"> <param id="type">python</param> <param id="function">trinity_all</param> </destination> </destinations> <tools> <tool id="runAssembly" destination="newbler_d"/> <tool id="trinity_all" destination="trinity_d"/> </tools> <limits> </limits> </job_conf> ~~~~~lib/galaxy/jobs/rules/dynamicjobs.py~~~~~~~~~~~~~~~~~~~~ import os from galaxy.jobs import JobDestination def newbler(job): incoming = dict( [ ( p.name, p.value ) for p in job.parameters ] ) walltime_para = incoming["walltime"] walltime = int(walltime_para.replace("\"","")) params = {} string = "vmem=500gb,nodes=1:ppn=32,walltime=%d:00:00"%walltime params["Resource_List"] = string return JobDestination(runner="pbs", params=params) def trinity_all(job): incoming = dict( [ ( p.name, p.value ) for p in job.parameters ] ) walltime_para = incoming["walltime"] walltime = int(walltime_para.replace("\"","")) params = {} string = "vmem=500gb,nodes=1:ppn=32,walltime=%d:00:00"%walltime params["Resource_List"] = string return JobDestination(runner="pbs", params=params) I re-installed the Roche 454 toolsuite from the toolshed once more to see if it just needed a kick. I copied the tool id into my job_conf file to make sure I made no mistakes on the id. I tried to send it to dynamic job runner and to pbs runner. I pulled and merged this galaxy about a month ago, so it should be mostly up to date - but I've had so many issues that I may have to just re-install from scratch. Thanks a ton for any advice, Carrie Ganote

3 3

GA file format documentation
by Philip Mabon 07 Oct '13

07 Oct '13

Is there any documentation on how exported workflow with '.ga' extension are structured? We are looking for the ability to create our workflows without using the workflow editor thru the web portal. Thanks Takadonet

2 3

error checking active jobs : KeyError: 'destination'
by Ganote, Carrie L 07 Oct '13

07 Oct '13

Hi List, I'm having this same issue - if I dispatch a job to dynamic job runner, and it then hands the job to pbs, then it will flip out if I restart the server. In my case, though, there is no error in the job state - it is sitting in the Torque queue where it should be. As far as I can tell, galaxy successfully reloads the job back into the history. But it falters on recovering this destination for pbs. If I delete the queued jobs, it quiets down, but this is not a sustainable solution - I need to be able to reboot the server without all of my users' cluster jobs causing faults every second. Log: galaxy.jobs.runners.pbs DEBUG 2013-10-04 15:50:32,454 Set default PBS server to m1.mason.indiana.edu galaxy.jobs.runners DEBUG 2013-10-04 15:50:32,455 Starting 4 PBSRunner workers galaxy.jobs DEBUG 2013-10-04 15:50:32,459 Loaded job runner 'galaxy.jobs.runners.pbs:PBSJobRunner' as 'pbs' galaxy.jobs.handler DEBUG 2013-10-04 15:50:32,459 Loaded job runners plugins: local:pbs galaxy.jobs.handler INFO 2013-10-04 15:50:32,460 job handler stop queue started galaxy.jobs DEBUG 2013-10-04 15:50:32,478 (514) Working directory for job is: /N/dc/projects/galaxy/job_working_directory/000/514 galaxy.jobs.handler DEBUG 2013-10-04 15:50:32,478 recovering job 514 in pbs runner galaxy.jobs WARNING 2013-10-04 15:50:32,478 (514) Job runner URLs are deprecated, use destinations instead. galaxy.jobs.runners.pbs DEBUG 2013-10-04 15:50:32,479 (514/176938.m1.mason) is still in PBS queued state, adding to the PBS queue galaxy.jobs.handler INFO 2013-10-04 15:50:32,485 job handler queue started ... galaxy.jobs.runners ERROR 2013-10-04 15:50:33,456 Unhandled exception checking active jobs Traceback (most recent call last): File "/N/hd03/galaxy/Mason/galaxy-ncgas/lib/galaxy/jobs/runners/__init__.py", line 362, in monitor self.check_watched_items() File "/N/hd03/galaxy/Mason/galaxy-ncgas/lib/galaxy/jobs/runners/pbs.py", line 385, in check_watched_items ( failures, statuses ) = self.check_all_jobs() File "/N/hd03/galaxy/Mason/galaxy-ncgas/lib/galaxy/jobs/runners/pbs.py", line 466, in check_all_jobs pbs_server_name = self.__get_pbs_server(pbs_job_state.job_destination.params) File "/N/hd03/galaxy/Mason/galaxy-ncgas/lib/galaxy/jobs/runners/pbs.py", line 222, in __get_pbs_server return job_destination_params['destination'].split('@')[-1] KeyError: 'destination' Thanks so much for wisdom on this. Sincerely, Carrie Ganote

1 0

fastx_toolkit not recognizing fastq files
by UMD Bioinformatics 07 Oct '13

07 Oct '13

Hello, I've installed Galaxy on two separate machines. None of the fastx tools recognize my fastq files. They are not appearing in the drop down list on Galaxy for input options. I'm getting the same behavior on both machines. I've used the tool_shed to install fastx on one machine and I've done it manually on the other. fastx_tools work perfectly from the command line. Cheers Ian

2 2

GA file format documentation
by Philip Mabon 07 Oct '13

07 Oct '13

Is there any documentation on how exported workflow with '.ga' extension are structured? We are looking for the ability to create our workflows without using the workflow editor thru the web portal. Thanks

1 0

Re: [galaxy-dev] Functional tests broken, NameError: global name 'db_path' is not defined
by UMD Bioinformatics 07 Oct '13

07 Oct '13

Hello all, I'm having this same issue with Functional Test being broken. I've switched between 'stable' and 'default' and it doesn't seem to fix the issue. Its possible that I'm not switching properly? This is my first local install of Galaxy so I'm new to the structure, detailed help would be very appreciated. Cheers Ian $ sudo hg update default 0 files updated, 0 files merged, 0 files removed, 0 files unresolved $ hg heads | more changeset: 10421:a477486bf18e branch: stable tag: tip user: Nate Coraor <nate(a)bx.psu.edu> date: Thu Sep 26 11:02:58 2013 -0400 summary: Bugfix for tool-to-destination mapping, tool ids are lowercased but the mapping id was not lowercased. changeset: 10411:c42567f43aa7 user: greg date: Mon Aug 19 13:19:56 2013 -0400 summary: Filter invalid objects when generating the list of repository_dependencies objects that are associated with a tool shed repository installed into Galaxy. $ sudo hg update stable 194 files updated, 0 files merged, 48 files removed, 0 files unresolved $ hg heads | more changeset: 10421:a477486bf18e branch: stable tag: tip user: Nate Coraor <nate(a)bx.psu.edu> date: Thu Sep 26 11:02:58 2013 -0400 summary: Bugfix for tool-to-destination mapping, tool ids are lowercased but the mapping id was not lowercased. changeset: 10411:c42567f43aa7 user: greg date: Mon Aug 19 13:19:56 2013 -0400 summary: Filter invalid objects when generating the list of repository_dependencies objects that are associated with a tool shed repository installed into Galaxy. > Message: 16 > Date: Thu, 3 Oct 2013 12:06:16 +0100 > From: Peter Cock <p.j.a.cock(a)googlemail.com> > To: Galaxy Dev <galaxy-dev(a)bx.psu.edu> > Subject: [galaxy-dev] Functional tests broken, NameError: global name > 'db_path' is not defined > Message-ID: > <CAKVJ-_63+TJrLT1=CyxJeFpk1Z=6w9i=MkbaHdc-=dSFFCK+eg(a)mail.gmail.com> > Content-Type: text/plain; charset=ISO-8859-1 > > Hi all, > > Something has broken on the functional test framework. > > Attempting to run the tests fails, as does just listing the IDs: > > $ ./run_functional_tests.sh --list > /mnt/galaxy/galaxy-central/eggs/twill-0.9-py2.6.egg/twill/other_packages/_mechanize_dist/_auth.py:14: > DeprecationWarning: the md5 module is deprecated; use hashlib instead > import re, base64, urlparse, posixpath, md5, sha, sys, copy > /mnt/galaxy/galaxy-central/eggs/twill-0.9-py2.6.egg/twill/other_packages/_mechanize_dist/_auth.py:14: > DeprecationWarning: the sha module is deprecated; use the hashlib > module instead > import re, base64, urlparse, posixpath, md5, sha, sys, copy > /mnt/galaxy/galaxy-central/eggs/pysam-0.4.2_kanwei_b10f6e722e9a-py2.6-linux-x86_64-ucs4.egg/pysam/__init__.py:1: > RuntimeWarning: __builtin__.file size changed, may indicate binary > incompatibility > from csamtools import * > Traceback (most recent call last): > File "./scripts/functional_tests.py", line 457, in <module> > sys.exit( main() ) > File "./scripts/functional_tests.py", line 246, in main > database_connection = 'sqlite:///' + os.path.join( db_path, > 'universe.sqlite' ) > NameError: global name 'db_path' is not defined > 'run_functional_tests.sh help' for help > > > $ hg branch > stable > [galaxy@ppserver galaxy-central]$ hg heads | more > changeset: 11805:a3c018044538 > tag: tip > user: greg > date: Wed Oct 02 16:44:48 2013 -0400 > summary: Eliminate the problematic "Select one" option from select fields. > > changeset: 11761:a477486bf18e > branch: stable > parent: 11689:4b1c7ad464ea > user: Nate Coraor <nate(a)bx.psu.edu> > date: Thu Sep 26 11:02:58 2013 -0400 > summary: Bugfix for tool-to-destination mapping, tool ids are > lowercased but the mapping id was not lowercased. > > ... > > > Regards, > > Peter > > > ------------------------------ > > Message: 17 > Date: Thu, 03 Oct 2013 08:53:47 -0400 > From: Dave Bouvier <dave(a)bx.psu.edu> > To: Peter Cock <p.j.a.cock(a)googlemail.com>, Galaxy Dev > <galaxy-dev(a)bx.psu.edu> > Subject: Re: [galaxy-dev] Functional tests broken, NameError: global > name 'db_path' is not defined > Message-ID: <524D68DB.5060803(a)bx.psu.edu> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Peter, > > This bug was fixed on the default branch in 10427:33092bc258d7, or > whichever changeset that corresponds to in your repository. The relevant > changelog entry, if you want to find and apply it to your repository, is > as follows: > > changeset: 10427:33092bc258d7 > user: Carl Eberhard <carlfeberhard(a)gmail.com> > date: Wed Aug 21 10:00:09 2013 -0700 > summary: Functional tests: fix sqlite db_path var; fix api test for > hda update with nonsense key > > > --Dave B. > > On 10/03/2013 07:06 AM, Peter Cock wrote: >> Hi all, >> >> Something has broken on the functional test framework. >> >> Attempting to run the tests fails, as does just listing the IDs: >> >> $ ./run_functional_tests.sh --list >> /mnt/galaxy/galaxy-central/eggs/twill-0.9-py2.6.egg/twill/other_packages/_mechanize_dist/_auth.py:14: >> DeprecationWarning: the md5 module is deprecated; use hashlib instead >> import re, base64, urlparse, posixpath, md5, sha, sys, copy >> /mnt/galaxy/galaxy-central/eggs/twill-0.9-py2.6.egg/twill/other_packages/_mechanize_dist/_auth.py:14: >> DeprecationWarning: the sha module is deprecated; use the hashlib >> module instead >> import re, base64, urlparse, posixpath, md5, sha, sys, copy >> /mnt/galaxy/galaxy-central/eggs/pysam-0.4.2_kanwei_b10f6e722e9a-py2.6-linux-x86_64-ucs4.egg/pysam/__init__.py:1: >> RuntimeWarning: __builtin__.file size changed, may indicate binary >> incompatibility >> from csamtools import * >> Traceback (most recent call last): >> File "./scripts/functional_tests.py", line 457, in <module> >> sys.exit( main() ) >> File "./scripts/functional_tests.py", line 246, in main >> database_connection = 'sqlite:///' + os.path.join( db_path, >> 'universe.sqlite' ) >> NameError: global name 'db_path' is not defined >> 'run_functional_tests.sh help' for help >> >> >> $ hg branch >> stable >> [galaxy@ppserver galaxy-central]$ hg heads | more >> changeset: 11805:a3c018044538 >> tag: tip >> user: greg >> date: Wed Oct 02 16:44:48 2013 -0400 >> summary: Eliminate the problematic "Select one" option from select fields. >> >> changeset: 11761:a477486bf18e >> branch: stable >> parent: 11689:4b1c7ad464ea >> user: Nate Coraor <nate(a)bx.psu.edu> >> date: Thu Sep 26 11:02:58 2013 -0400 >> summary: Bugfix for tool-to-destination mapping, tool ids are >> lowercased but the mapping id was not lowercased. >> >> ... >> >> >> Regards, >> >> Peter >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> http://lists.bx.psu.edu/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ >> > > > ------------------------------ > > Message: 18 > Date: Thu, 3 Oct 2013 14:41:17 +0100 > From: Peter Cock <p.j.a.cock(a)googlemail.com> > To: Dave Bouvier <dave(a)bx.psu.edu> > Cc: Galaxy Dev <galaxy-dev(a)bx.psu.edu> > Subject: Re: [galaxy-dev] Functional tests broken, NameError: global > name 'db_path' is not defined > Message-ID: > <CAKVJ-_4rHBrLM_zK__cxWRJkwWChR3KhQ3mGgrkNsv06awG19w(a)mail.gmail.com> > Content-Type: text/plain; charset=ISO-8859-1 > > On Thu, Oct 3, 2013 at 1:53 PM, Dave Bouvier <dave(a)bx.psu.edu> wrote: >> Peter, >> >> This bug was fixed on the default branch in 10427:33092bc258d7, or whichever >> changeset that corresponds to in your repository. The relevant changelog >> entry, if you want to find and apply it to your repository, is as follows: >> >> changeset: 10427:33092bc258d7 >> user: Carl Eberhard <carlfeberhard(a)gmail.com> >> date: Wed Aug 21 10:00:09 2013 -0700 >> summary: Functional tests: fix sqlite db_path var; fix api test for hda >> update with nonsense key >> >> >> --Dave B. > > Thanks - from the date that ought to be on my development system > already, but I can check more specifically now. > > Peter > > > > > Message: 21 > Date: Thu, 3 Oct 2013 15:33:30 +0100 > From: Peter Cock <p.j.a.cock(a)googlemail.com> > To: Dave Bouvier <dave(a)bx.psu.edu> > Cc: Galaxy Dev <galaxy-dev(a)bx.psu.edu> > Subject: Re: [galaxy-dev] Functional tests broken, NameError: global > name 'db_path' is not defined > Message-ID: > <CAKVJ-_4-qw=Kz+5HstT6fHvVUbORbfx1DvsDpX3eiAOTYPgGFA(a)mail.gmail.com> > Content-Type: text/plain; charset=ISO-8859-1 > > On Thu, Oct 3, 2013 at 1:53 PM, Dave Bouvier <dave(a)bx.psu.edu> wrote: >> Peter, >> >> This bug was fixed on the default branch in 10427:33092bc258d7, or whichever >> changeset that corresponds to in your repository. The relevant changelog >> entry, if you want to find and apply it to your repository, is as follows: >> >> changeset: 10427:33092bc258d7 >> user: Carl Eberhard <carlfeberhard(a)gmail.com> >> date: Wed Aug 21 10:00:09 2013 -0700 >> summary: Functional tests: fix sqlite db_path var; fix api test for hda >> update with nonsense key >> > > Switching to the default branch did indeed fix it - so this needs > to be added to the stable branch (not there yet): > https://bitbucket.org/galaxy/galaxy-central/src/a477486bf18eafdd14dd7ba1e91… > > Thanks, > > Peter > >

1 0