- galaxy-dev - lists.galaxyproject.org

UnicodeDecodeError - suggested note regarding UTF8 encoding in postgres databases
by Guest, Simon 19 Sep '13

19 Sep '13

I've finally joined the happy throng suffering from Unicode errors. In my particular case, running the deseq tool from the main toolshed brought down my Galaxy server, and it wouldn't restart. The traceback in paster.log contained this Traceback (most recent call last): [snip] File "/home/galaxy-dev/galaxy/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 2903, in __init__ self._init_metadata() File "/home/galaxy-dev/galaxy/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 3306, in _init_metadata self.__buffer_rows() File "/home/galaxy-dev/galaxy/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 3326, in __buffer_rows self.__rowbuffer = collections.deque(self.cursor.fetchmany(size)) UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 162: ordinal not in range(128) The same error occurred each time I tried to restart the galaxy server until I restored the galaxy database from last night's postgres backup. The problem was my galaxy database in postgres was using SQL_ASCII encoding, which turns out to be a bad idea if you're ever going to write Unicode characters there. (I'm quite disappointed that this is the default encoding for postgres databases on CentOS 6.) The fix was to dump the galaxy database, recreate it with UTF8 encoding, and reload it. Now everything works, and that one tool doesn't crash my galaxy server. I thought it would be worth noting that postgres databases should be created with UTF8 encoding on this page: http://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer#Swi… I think the right thing to do is to ensure you set the default database encoding for postgres when you run initdb. You only get one chance to get this right. After that, a way to change the default database encoding to UTF8 is given here: https://gist.github.com/ffmike/877447 Scary stuff, but it worked well enough for me. Alternatively, if you didn't think about encodings when you ran initdb, you can think about it every time you run createdb. Hope this helps someone. cheers, Simon ======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================

1 0

Re: [galaxy-dev] Mercurial updates not working - caused by http_proxy?
by Guest, Simon 18 Sep '13

18 Sep '13

Well, having just hit this again myself, I found what was *actually* causing the problem, rather than my vague recollection of what caused it, for which, apologies. In my case just now, it was caused by having an HTTP proxy set via the environment variable http_proxy. Unsetting that fixed my problem. Pesky web proxy, bah! You should never trust random strangers on the net, by the way. ;-) cheers, Simon > -----Original Message----- > From: Guest, Simon > Sent: Thursday, 5 September 2013 1:38 p.m. > To: Alistair Chilcott > Cc: galaxy-dev(a)lists.bx.psu.edu > Subject: RE: Mercurial updates not working > > Hi Alistair, > > I have seen error output like this from mercurial when the client is > running a newer version than the server. > > Maybe you recently upgraded your machine, and your mercurial version is > now too recent. > > Hope that helps. Of course, it may be a complete red herring, YMMV. ;-) > > cheers, > Simon > > > -----Original Message----- > > From: galaxy-dev-bounces(a)lists.bx.psu.edu [mailto:galaxy-dev- > > bounces(a)lists.bx.psu.edu] On Behalf Of Alistair Chilcott > > Sent: Wednesday, 28 August 2013 12:33 p.m. > > To: galaxy-dev(a)lists.bx.psu.edu > > Subject: [galaxy-dev] Mercurial updates not working > > > > Hello all, > > > > Mercurial has recently started throwing errors: > > > > galaxy@hitmsbhpc1:~/galaxy-dist> hg incoming > > abort: error: _ssl.c:517: error:140770FC:SSL > > routines:SSL23_GET_SERVER_HELLO:unknown protocol > > > > It has previously worked with issues has anyone seen this before? > > > > I don't think anything has changed that would have caused this problem > > but googling error has proved to be unhelpful so far. > > > > Regards, > > > > Alistair > > > > > > > > ___________________________________________________________ > > Please keep all replies on the list by using "reply all" > > in your mail client. To manage your subscriptions to this and other > > Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ > > > > To search Galaxy mailing lists use the unified search at: > > http://galaxyproject.org/search/mailinglists/ ======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================

1 0

'galaxy-api-tools' project
by Carlos Borroto 18 Sep '13

18 Sep '13

Hi, I'm interested in developing a tool that would allow easy interaction with the Galaxy API. I started coding and this[1] is what I have so far. [1]https://github.com/cjav/galaxy-api-tools This is a pure "release early, release often" kind of project. In the current state many of the project's goals are missing and current implementation might significantly change. You can however already do something like this: $ galaxy-api-tools import --link --file-type fasta --library "API Imports" --folder R_TEST --description "R_TEST with description." --workflow 2491ee81eec9beee --extension .qual:qual454 --label QUAL:.qual --label FASTA:.fna /local/data/R_TEST/1.fna /local/data/R_TEST/2.fna This will import(only '--link' not copy, copy is the default) these two 454 fasta files with the matching quality files into data library "API Imports" and folder "R_TEST"(it will create both if they don't exists), it will them run a workflow where there are two inputs labeled as FASTA and QUAL assigning the correct file to each, the results will be available in a newly created history. I shamelessly copied/adapted code from '$GALAXY_SOURCE/scripts/api'. I'm releasing this code under BSD-Clause-2 which I think is one of the most open license I could find in the github dropdown menu. Hopefully this license won't have any incompatibility with Galaxy license which seems very open also. I would love to receive comments to help shape this tool. I'm very open to any kind of suggestions. Also, if there is any worry the name of the tool my cause any confusion on the source, please let me know and I'll be happy to change it. Best Carlos

1 1

which galaxy to use?
by Jennifer Jackson 17 Sep '13

17 Sep '13

Hi Peter, This is the correct list to post to (not owner, that is just for list subscription questions). http://wiki.galaxyproject.org/MailingLists Galaxy-dist is the stable release repository. Galaxy-central is the active development repository. Depending on how risk-tolerant your project is, either is a choice. You can also start with a dist base and pull from central individual changesets as desired, if you want to manage it that way. I'll let the developers comment about nglims. You may need to post that as a brand new question (new thread) with that in the subject line to get the best replies. Or you can start by searching with the keyword here (or 'sample tracking' and narrow down what you need to know: http://galaxyproject.org/search/getgalaxy/ These wiki's haven't been updated in a while, so I am not pointing you there first, but here they are for reference: http://wiki.galaxyproject.org/Admin/Sample%20Tracking/ Best, Jen Galaxy team On 9/17/13 11:10 AM, Peter Huang wrote: > > Hi Galaxy, > > I am trying to setup the production galaxy service at our place with > Rocks Cluster and NGLIMS. I am confused as which one to use: > galaxy-dist or galaxy-central. > > My question is could I use galaxy-dist with nglims? I try to find how > to do it, but couldn't. > > Any suggestions? > > Best, > > Peter > -- Jennifer Hillman-Jackson http://galaxyproject.org

2 1

tools in multiple sections can disappear from integrated_tool_panel.xml
by Richard Burhans 17 Sep '13

17 Sep '13

When Galaxy reads and updates the integrated_tool_panel.xml file, tools in multiple sections can be lost. 1) Install tool shed repository A into section A <section id="A" name="A" version=""> <tool id="toolshed.g2.bx.psu.edu/repos/some-user/repo-A/tool-A/ 1.0.0" /> </section> 2) Install tool shed repository B into section B <section id="A" name="A" version=""> <tool id="toolshed.g2.bx.psu.edu/repos/some-user/repo-A/tool-A/ 1.0.0" /> </section> <section id="B" name="B" version=""> <tool id="toolshed.g2.bx.psu.edu/repos/some-user/repo-B/tool-B/ 1.0.0" /> </section> 3) Edit integrated_tool_panel.xml to put a tool from repository B into section A <section id="A" name="A" version=""> <tool id="toolshed.g2.bx.psu.edu/repos/some-user/repo-A/tool-A/ 1.0.0" /> <tool id="toolshed.g2.bx.psu.edu/repos/some-user/repo-B/tool-B/ 1.0.0" /> </section> <section id="B" name="B" version=""> <tool id="toolshed.g2.bx.psu.edu/repos/some-user/repo-B/tool-B/ 1.0.0" /> </section> 4) When starting, tool-B will appear in both sections of the tool panel. Upon restart, tool-B will only appear in section B. This is due to a bug in the code which reads and re-writes the integrated_tool_panel.xml file. I've commited (96a2ab47b882) to galaxy-central a new version of lib/galaxy/tools/__init__.py which includes a workaround for this issue. Instead of my workaround, I would suggest some combination of: a) adjusting init_tools() and called methods to get this right b) redesigning the code and/or data structure used to read and write integrated_tool_panel.xml -rico

1 0

Re: [galaxy-dev] Made some additions
by Peter Cock 17 Sep '13

17 Sep '13

Hi Matt, Have a look here: http://wiki.galaxyproject.org/Tool%20Shed There were also some good overview talks at GCC2013, http://wiki.galaxyproject.org/Events/GCC2013 If your work would be better as an enhancement to an existing Tool Shed repository, then you could try to contact the author - there is a button to do this on the toolshed, but asking here by email would probably work too. Regards, Peter On Tue, Sep 17, 2013 at 5:42 PM, Matthew Paul <mrpaul(a)g.cofc.edu> wrote: > Awesome, I just looked at the repositories and it looks like my tools might > be a good fit. How do I go about adding them? > > Thanks > > > On Tue, Sep 17, 2013 at 12:28 PM, Peter Cock <p.j.a.cock(a)googlemail.com> > wrote: >> >> On Tue, Sep 17, 2013 at 4:16 PM, Matthew Paul <mrpaul(a)g.cofc.edu> wrote: >> > Hey guys, >> > >> > I have been working witha local Galaxy for a couple of months now on a >> > cluster. I have created a few tools and would like to know if you would >> > like >> > to include them in the central repository (or how I go about suggesting >> > so). >> >> The general idea is rather you would share them via the Galaxy Tool Shed, >> http://toolshed.g2.bx.psu.edu/ >> >> > I created a statisticss computer for de novo assembly builds. Returns >> > information on the spread of the contigs, the N50, and mean. >> > >> > I have also created a wrapper for trans-abyss (SE and PE) and my partner >> > created a wrapper for trimmomatic (SE and PE). >> >> Check the main and Test Tool Shed first in case someone else is also >> doing the same wrappers, but otherwise this sounds useful. >> http://toolshed.g2.bx.psu.edu/ >> http://testtoolshed.g2.bx.psu.edu/ >> >> Regards, >> >> Peter > >

1 0

Made some additions
by Matthew Paul 17 Sep '13

17 Sep '13

Hey guys, I have been working witha local Galaxy for a couple of months now on a cluster. I have created a few tools and would like to know if you would like to include them in the central repository (or how I go about suggesting so). I created a statisticss computer for de novo assembly builds. Returns information on the spread of the contigs, the N50, and mean. I have also created a wrapper for trans-abyss (SE and PE) and my partner created a wrapper for trimmomatic (SE and PE). Let me know what you think, Thanks

2 1

Re: [galaxy-dev] R-environment Patch [was: [galaxy-iuc] RFC: R, RPy2, DESeq2 example wrapper]
by John Chilton 17 Sep '13

17 Sep '13

On Tue, Sep 17, 2013 at 3:03 AM, Bjoern Gruening <bjoern.gruening(a)gmail.com> wrote: > Hi all, > > that is now implemented in: > > https://bitbucket.org/galaxy/galaxy-central/pull-request/219/implementation… > > Any comments are much appreciated. Fantastic work, I like this a lot. > Is it still to complicated? > > Greg and Dave can we refactor the set_env_var functionality so we can > set this automatically for the r-environment or even the > python-environment? Greg and/or Dave's comments on this point would be great. But, just to clarify this point the Python version does set its environment automatically: https://bitbucket.org/galaxy/galaxy-central/src/48fad71361b3075c85c7d365dd5… My vote is that you just add a similar command for R - even if the code for doing so is sort of verbose for right now - and then once the R/Ruby/Python (Perl?) stuff is added we revisit and refactor to remove duplicated code. Again, Greg or Dave should weigh in though :). Thanks Bjoern, great stuff! -John > > If that patch will be merged my plan is to write a short wiki page with > a 'best practice for R package development'. Furthermore, I will try to > create a small script that will obtain the dependency tree of all the > required tarballs. > > Speaking of tarballs, currently I just download them and host them in a > separate github repository. That is not ideal and I will start a new > discussion in one of my next mails about one central place. > > Thanks! > Bjoern > > > >> Cool work. I was thinking something like this would be needed >> (https://github.com/viking/Renv) but I like your approach better. >> >> The one think I would say is that, like Python with setup_venv, if >> this is best practices Galaxy should provide some high level support >> for this. I was thinking your package install could look something >> like this: >> >> <package name="deseq2" version="1.0.17"> >> <install version="1.0"> >> <actions> >> <action type="setup_r_environment"> >> <base_r changeset_revision="bae5c9880b71" >> name="package_r_3_0_1" owner="bgruening" >> toolshed="http://testtoolshed.g2.bx.psu.edu" /> >> <r_package>https://github.com/bgruening/download_store/raw/master/DESeq2-1_0_18/BiocGe…</r_package> >> <r_package>https://github.com/bgruening/download_store/raw/master/DESeq2-1_0_18/IRange…</r_package> >> <r_package>https://github.com/bgruening/download_store/raw/master/DESeq2-1_0_18/Genomi…</r_package> >> <r_package>https://github.com/bgruening/download_store/raw/master/DESeq2-1_0_18/Rcpp_0…</r_package> >> <r_package>https://github.com/bgruening/download_store/raw/master/DESeq2-1_0_18/RcppAr…</r_package> >> <r_package>https://github.com/bgruening/download_store/raw/master/DESeq2-1_0_18/locfit…</r_package> >> <r_package>https://github.com/bgruening/download_store/raw/master/DESeq2-1_0_18/Biobas…</r_package> >> <r_package>https://github.com/bgruening/download_store/raw/master/DESeq2-1_0_18/DBI_0.…</r_package> >> <r_package>https://github.com/bgruening/download_store/raw/master/DESeq2-1_0_18/RSQLit…</r_package> >> <r_package>https://github.com/bgruening/download_store/raw/master/DESeq2-1_0_18/Annota…</r_package> >> <r_package>https://github.com/bgruening/download_store/raw/master/DESeq2-1_0_18/xtable…</r_package> >> <r_package>https://github.com/bgruening/download_store/raw/master/DESeq2-1_0_18/XML_3.…</r_package> >> <r_package>https://github.com/bgruening/download_store/raw/master/DESeq2-1_0_18/annota…</r_package> >> <r_package>https://github.com/bgruening/download_store/raw/master/DESeq2-1_0_18/genefi…</r_package> >> <r_package>https://github.com/bgruening/download_store/raw/master/DESeq2-1_0_18/RColor…</r_package> >> <r_package>https://github.com/bgruening/download_store/raw/master/DESeq2-1_0_18/DESeq2…</r_package> >> </action> >>  >> </actions> >> </install> >> </package> >> >> Obviously, someone would need to implement the action type >> "setup_r_environment", but frankly I believe you have already done >> most (or at least a large amount) of the work in demonstrating how it >> should be implemented and that it is feasible. >> >> -John >> >> On Mon, Aug 5, 2013 at 11:10 AM, Bjoern Gruening >> <bjoern.gruening(a)gmail.com> wrote: >> > Dear Commissioners, >> > >> > I have spent some time in prototyping an R wrapper for DESeq2 and RPy2. >> > In the end I will skip RPy2 as dependency, because I will include a pure >> > R-script. Currently it's more or less a prototype. My main Aim was to >> > make R and RPy2 a first class citizen in the Tool Shed and to make it >> > easy to write tools in R in a reproducible way. >> > >> > The result looks really simple, imho: >> > http://testtoolshed.g2.bx.psu.edu/view/bgruening/deseq2 >> > >> > As Ross already investigated [1] ... to make the R and a R package >> > really reproducible is really hard or not possible at all with the R >> > package management system. What I did is to track the dependencies down >> > and upload the tarballs to some webspace and manually install them via >> > install.packages(). >> > >> > http://testtoolshed.g2.bx.psu.edu/view/bgruening/package_deseq2_1_0_17 >> > >> > Steps to write a R-wrapper: >> > - link to the package_r_3_0_1 >> > - (optionally link to package_rpy2_2_3_6) >> > - upload all dependencies of your R package to github/bitbucket ... >> > - write a wrapper that is fetching these packages and install them in >> > the right order to your own R package repository, not the basic R >> > installation >> > >> > What do you think, is that to complicated? If you agree witch such an >> > approach I will write a "best practice" wiki page. >> > >> > As a side note, this is using my 'complicated' R orphan tool dependency >> > with atlas dependency. But I tweaked it a little bit with the new tool >> > shed features to fail silently and fallback to a non-atlas mode. So OSX >> > users, should be able to install it, finally :) >> > >> > Thanks! >> > Bjoern >> > >> > [1] Subject: "Re: [galaxy-iuc] [Bioc-devel] finding specific (outdated!) >> > bioc package archives?" >> > >> > _______________________________________________ >> > galaxy-iuc mailing list >> > galaxy-iuc(a)lists.bx.psu.edu >> > http://lists.bx.psu.edu/listinfo/galaxy-iuc > > >

2 1

no access to ftp on email change
by Vandeweyer Geert 17 Sep '13

17 Sep '13

Hi, Galaxy provides the option for users to change their login email. However, if they do, they loose access to their ftp folder. Is there an option to allow galaxy to rename these folders? Best, Geert

1 0

Issue with Bioblend's run_workflow import_inputs_to_history
by Carlos Borroto 17 Sep '13

17 Sep '13

Hi, I'm enjoying a lot using Bioblend to develop CLI toolbox to interact with Galaxy API. I recently hit an issue regarding how to deal with inputs after running a workflow. bioblend.galaxy.workflows.run_workflow()[1] has an argument import_inputs_to_history. From the documentation: "import_inputs_to_history (bool) – If True, used workflow inputs will be imported into the history. If False, only workflow outputs will be visible in the given history." As far as I can tell this will add this key to the payload delivery to Galaxy: payload['no_add_to_history'] = True However this is the behavior I'm experiencing. When set to 'False' inputs are still imported into the history, but instead of the usual history serial id number they have ':', even worse they are imported in duplicates. I was expecting no presence at all. When set to 'True' they get the expected history serial number, but once again they are imported in duplicates. I'm not sure is this is a problem with Bioblend or Galaxy API. I also was guessing this list might be a good as any place to discuss issues with Bioblend. [1]http://bioblend.readthedocs.org/en/latest/api_docs/galaxy/all.html#module… Thanks, Carlos

1 0