June 2013 - galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] [galaxy-user] How do I add Ray to Galaxy Central in the tool shed ?
by Sébastien Boisvert 13 Jun '13

13 Jun '13

Hi, On 12/06/13 05:50 PM, Jennifer Jackson wrote: > Hi Sébastien, > > This all looks fine, except for the part about including this in the > distribution. As James stated, we are moving to a model where eventually > all tools will be sourced from the Tool Shed. That is where you should > create your repository. So if someone installs galaxy-dist, the person will have to pick up each tool in the tool shed. Is that correct ? Will the schema of XML files remain the same ? > Whether or not to have the repo binaries > self-install & config or not will be up to you. You mean that some apps will self-install and other will require the intervention of a system administrator. > > Going forward, you probably will want to start up a new thread for new > questions and have discussions on the galaxy-dev(a)bx.psu.edu mailing > list. This will give your questions the most exposure to the development > community. > http://wiki.galaxyproject.org/MailingLists#The_lists > You are right, I should have posted to -dev in the first place. Sorry. So my deliverable to add Ray in galaxy is 2 XML files. Thanks. > Take care, > > Jen > Galaxy team > > On 6/12/13 8:05 AM, Sébastien Boisvert wrote: >> On 06/06/13 02:32 PM, James Taylor wrote: >>>> I want to add Ray (a scalable de novo assembler for genomes and >>>> metagenomes) >>>> to Galaxy. >>> >>> And I really want you to do this! >>> >>>> I will also have to write a wrapper for Ray to prepare the command >>>> line from >>>> the options provided by the >>>> Galaxy API. >>>> >>>> >>>> But where is stored the executable (in my case, where is sdtored Ray) ? >>>> >>>> Does Galaxy include the specs to build all the tools available in >>>> Galaxy-Central ? >>> >>> No, we are gradually moving all the tools out of galaxy-central into >>> the Tool Shed. You probably want to look at this page: >>> >>> http://wiki.galaxyproject.org/ToolShedToolFeatures#Automatic_third-party_to… >> >> >> From what I understand, I need to write 2 XML files in order to allow >> people to add Ray to their Galaxy instances: >> >> * ray.xml >> * tool_dependencies.xml >> >> These two files are self-contained -- that is Galaxy will install >> everything using tool_dependencies.xml (in my case, >> Open-MPI and Ray will be installed from source using the GNU compiler). >> >> >> Can I put these files directly in >> >> galaxy-dist/tools/sr_assembly/ray/ >> >> ? >> >> I suppose these 2 XML files will be distributed in galaxy-dist, but >> like Velvet, Ray won't be enabled by default in tool_conf.xml >> as assembly is more costly than mapping. >> >> >> Then, I need to add this to galaxy-dist/tool_conf.xml: >> >> <section name="NGS: Assembly" id="ngs_assembly"> >> <label text="Ray" id="ray"/> >> <tool file="sr_assembly/ray/ray.xml" /> >> </section> >> >> >> And finally, I just need to restart Galaxy. >> >> >> Is that correct ? >> >>> >>> Which describes how you can add a recipe to the toolshed that will >>> install the Ray binaries. >>> >> >> ___________________________________________________________ >> The Galaxy User list should be used for the discussion of >> Galaxy analysis and other features on the public server >> at usegalaxy.org. Please keep all replies on the list by >> using "reply all" in your mail client. For discussion of >> local Galaxy instances and the Galaxy source code, please >> use the Galaxy Development list: >> >> http://lists.bx.psu.edu/listinfo/galaxy-dev >> >> To manage your subscriptions to this and other Galaxy lists, >> please use the interface at: >> >> http://lists.bx.psu.edu/ >> >> To search Galaxy mailing lists use the unified search at: >> >> http://galaxyproject.org/search/mailinglists/ >

2 1

Galaxy wrappers for NCBI BLAST+ BoF
by Peter Cock 13 Jun '13

13 Jun '13

(Apologies is anyone who go this twice, the Galaxy mailing list has been offline.) Hello all, As many of you know I wrote the original BLAST+ wrappers for Galaxy, which are now on the Tool Shed here: * http://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus * http://toolshed.g2.bx.psu.edu/view/devteam/blast_datatypes Over the years other people have started to make contributions (e.g. Edward Kirton worked on BLAST databases as a datatype), and this is starting to become a team effort. I'd like that to continue. I've already been in contact with by email with those interested people I knew about to setup a developers meeting during GCC2013, but realise it might be nice to have this formally listed as a BoF - and to make sure we have an allocated time slot as there seems to be a lot going on at the Oslo meeting: http://wiki.galaxyproject.org/Events/GCC2013/BoF/GalaxyBlast Not everyone will be at the conference, but don't worry - like most open source projects most of this will be done online. People using BLAST+ within Galaxy could come along too and tell us about what they think we should prioritise, but I am mainly hoping to get together with people who have or will be actively involved in the wrapper development and testing. Thanks all, Peter P.S. I will only have intermittent email access from now until the GCC2013 meeting.

1 0

Re: [galaxy-dev] Portable tool identifiers in workflows
by Peter Cock 13 Jun '13

13 Jun '13

Forwarding back to the list which had been offline and seemed to be missing these two emails... ---------- Forwarded message ---------- From: Greg Von Kuster <greg(a)bx.psu.edu> Date: Thu, Jun 13, 2013 at 6:34 PM Subject: Re: Portable tool identifiers in workflows To: Peter Cock <p.j.a.cock(a)googlemail.com> Cc: Galaxy Dev <galaxy-dev(a)bx.psu.edu>, Björn Grüning <bjoern.gruening(a)pharmazie.uni-freiburg.de> Hello Peter, Unfortunately, workflows are currently not very compatible with the tool shed - there is a Trello card that details the issues here: https://trello.com/card/workflow-issues/506338ce32ae458f6d15e4b3/692 See my inline comments below. On Jun 13, 2013, at 7:13 AM, Peter Cock <p.j.a.cock(a)googlemail.com> wrote: > Hi Greg, > > I've recent uploaded a couple of workflows to the Tool Shed, > http://toolshed.g2.bx.psu.edu/view/peterjc/venn_list/ > http://toolshed.g2.bx.psu.edu/view/peterjc/rxlr_venn_workflow > > The *.ga files where created by saving the workflow from our > local Galaxy instance. I then wrote a repository_dependencies.xml > file, a README file, made a tar ball, and uploaded this to the > Tool Shed. > > Perhaps unusually the tools were installed manually rather than > via the Tool Shed. This is basically because our Galaxy instance > pre-dates the new hg based Tool Shed, and I was already installing > locally developed tools this way. > > This has a potential implication in the form of the tool identifiers, > drawn to my attention by Björn: > > On Thu, Jun 13, 2013 at 10:54 AM, Björn Grüning wrote: >> >> ... I'm just asking because your >> tool-ids in the ga file are not toolshed-tool-ids. So I'm curious if >> that can work. >> >> I will bug Greg. > > So, does Galaxy handle old fashioned tool identifiers and the > newer toolshed-tool-identifiers gracefully in a situation like this? Unfortunately this does not currently work - this is related to the 3rd item in the above Trello card: Workflows do not consider tool version lineage when inspecting the Galaxy tools_by_id in-memory dictionary, so tools that should be discovered for a workflow are not. > > Do I need to update the *.ga files to use toolshed-tool-ids? If > I did that, would it break the workflow on any Galaxy instance > (like ours) where the tools were installed manually? Yes, you would currently need to update the workflows to use toolshed-tool-ids, and your statement regarding the workflow not working with older tools is correct. This again is related to the 3rd item in the above Trello card: Workflows do not consider tool version lineage when inspecting the Galaxy tools_by_id in-memory dictionary, so tools that should be discovered for a workflow are not. > > Unfortunately from the point of view of testing, these first two > workflows both require a Tool Shed repository dependency > which is non-auto-installable (propriety licensing). > > Thanks, > > Peter >

1 0

Re: [galaxy-dev] blastn: Error: NCBI C++ Exception
by Peter Cock 13 Jun '13

13 Jun '13

Resending as the mailing list was down: On Thu, Jun 13, 2013 at 6:02 PM, Nicola Soranzo <soranzo(a)crs4.it> wrote: >> >> There are two issues here, one that we would like to be able to >> handle multi-threading better via Galaxy configuration on a >> per-tool level, but for now it must often be hard coded or >> done via tool-specific environment variables. > > Regarding the configuration of multi-threading for BLAST+ tools, instead > of hardcoding "-num_threads 8" in the XML files, I'd like to do > implement something like this: > > http://toolshed.g2.bx.psu.edu/repos/jjohnson/cdhit/rev/cca0838c1597 > > What do you think? Should I prepare a pull request to discuss at > GCC2013? That's a sensible idea, something like $BLASTTHREADS as an environment variable? I'd still prefer something built into Galaxy like $THREADS or $GALAXYTHREADS or whatever which can be sets with a default value and adjusted in the per-tool job runner setup (e.g. send BLAST jobs to this cluster queue with 16 threads). This is definitely a good general topic for the tool authors and/or BLAST wrapping BoF sessions at the conference: http://wiki.galaxyproject.org/Events/GCC2013/BoF/ToolDevelopers http://wiki.galaxyproject.org/Events/GCC2013/BoF/GalaxyBlast Peter

1 0

Question about the UCSC external viewer
by Malcolm Tobias 13 Jun '13

13 Jun '13

All, Let me try this again. My previous attempts have been bouncing. I've succeeded in getting in getting the "display at UCSC main" link to display, but I'm getting a strange error when I'm redirected to the page. I can't cut-and-paste the error, so hopefully the attached screen shot wont get eaten by the mailing list. From the galaxy log, I'm seeing the UCSC viewer request data: 128.114.119.133 - - [13/Jun/2013:08:47:22 -0500] "GET /display_application/6505e875ddb66fd2/ucsc_bam/main/None/data/galaxy_6505e875ddb66fd2.bam HTTP/1.0" 200 1130428 "-" "genome.ucsc.edu/net.c" but I don't see anything like "6505e875ddb66fd2" under display applications: [galaxy@login002 ~]$ ls -l galaxy-dist/display_applications/ total 48 drwxr-xr-x 2 galaxy galaxy 8192 May 15 12:12 ensembl drwxr-xr-x 2 galaxy galaxy 8192 May 15 12:12 gbrowse drwxr-xr-x 2 galaxy galaxy 8192 May 15 12:12 igb drwxr-xr-x 2 galaxy galaxy 8192 May 15 12:12 igv drwxr-xr-x 2 galaxy galaxy 8192 May 15 12:12 rviewer drwxr-xr-x 2 galaxy galaxy 8192 May 15 12:12 ucsc While the UCSC viewer is requesing data from the "display_application" subdirectory, I only have a "display_applications" subdirectory. I'm not seeing anything like "6505e875ddb66fd2" under galaxy-dist, so maybe I'm not doing some pre-processing needed before being redirected to UCSC? [galaxy@login002 ~]$ find galaxy-dist -name '*6505*' -print [galaxy@login002 ~]$ Any suggestions on what might be going wrong? Cheers, Malcolm -- Malcolm Tobias 314.362.1594

1 0

Re: [galaxy-dev] blastn: Error: NCBI C++ Exception
by Nicola Soranzo 13 Jun '13

13 Jun '13

Il giorno gio, 13/06/2013 alle 18.06 +0100, Peter Cock ha scritto: > On Thu, Jun 13, 2013 at 6:02 PM, Nicola Soranzo <soranzo(a)crs4.it> wrote: > >> > >> There are two issues here, one that we would like to be able to > >> handle multi-threading better via Galaxy configuration on a > >> per-tool level, but for now it must often be hard coded or > >> done via tool-specific environment variables. > > > > Regarding the configuration of multi-threading for BLAST+ tools, instead > > of hardcoding "-num_threads 8" in the XML files, I'd like to do > > implement something like this: > > > > http://toolshed.g2.bx.psu.edu/repos/jjohnson/cdhit/rev/cca0838c1597 > > > > What do you think? Should I prepare a pull request to discuss at > > GCC2013? > > That's a sensible idea, something like $BLASTTHREADS as > an environment variable? I was thinking something like BLAST_SITE_OPTIONS='-num_threads 8', which could also include other parameters (e.g. memory limits) in the future. > I'd still prefer something built into Galaxy like $THREADS or > $GALAXYTHREADS or whatever which can be sets with a > default value and adjusted in the per-tool job runner setup > (e.g. send BLAST jobs to this cluster queue with 16 threads). > > This is definitely a good general topic for the tool authors > and/or BLAST wrapping BoF sessions at the conference: > http://wiki.galaxyproject.org/Events/GCC2013/BoF/ToolDevelopers > http://wiki.galaxyproject.org/Events/GCC2013/BoF/GalaxyBlast > > Peter

1 0

blastn: Error: NCBI C++ Exception
by Berner, Thomas 13 Jun '13

13 Jun '13

Hi guys, another error appeared after we installed NCBI BLAST + to our local Galaxy instance and wanted to test it. After uploading some testdata to our history we tried to run blastn on it and got the following error: Fatal error: Matched on Error: Error: NCBI C++ Exception: "/opt/galaxy/galaxy-dist/database/tmp/tmphx2DHp/ncbi-blast-2.2.26+-src/c++/src/corelib/ncbiobj.cpp", line 689: Critical: ncbi::CObject::ThrowNullPointerException() - Attempt to access NULL pointer. With some testing around we figure out the problem: ./blastn -query ~/test.fasta -db nt -task blastn -evalue 0.001 -out ~/out2.txt -outfmt "6 std sallseqid score nident positive gaps ppos qframe sframe qseq sseq qlen slen" -num_threads 8 Doesn't work until we replaced the "num_threads 8" like it is written in the original "ncbi_blastn_wrapper.xml" with "num_threads 1" Editing the .xml also solved the problem within the galaxy surface. Now my question is, is there any other workaround without editing the" ncbi_blastn_wrapper.xml"? Searching in the documentation and the mailing list, I did not find anything about this. We believe this could be of common interest. It would be great if anybody could comment on this. Thanks a lot, Thomas

4 3

database upgrade crashes after upgrade to the June 3, 2013 Galaxy release
by Christophe Antoniewski 12 Jun '13

12 Jun '13

Hi, Sorry to bother again, but we have now a problem with the last release of galaxy-dist The lack of a python module asked by a python eggs seems to interrupt our postgresql database upgrade from version '114' to '115' sh manage_db.sh upgrade Traceback (most recent call last): File "./scripts/manage_db.py", line 13, in <module> from migrate.versioning.shell import main File "/home/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/shell.py", line 12, in <module> from migrate.versioning import api File "/home/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/api.py", line 33, in <module> from migrate.versioning import (repository, schema, version, File "/home/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/repository.py", line 13, in <module> from migrate.versioning import version, pathed, cfgparse File "/home/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/version.py", line 10, in <module> from migrate.versioning import pathed, script File "/home/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/pathed.py", line 11, in <module> from migrate.versioning.util import KeyedInstance File "/home/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/util/__init__.py", line 7, in <module> from decorator import decorator ImportError: No module named decorator Any idea ? Thanks Chris -- Christophe Antoniewski Drosophila Genetics and Epigenetics Laboratoire de Biolologie du Développement 9, Quai St Bernard, Boîte courrier 24 75252 Paris Cedex 05 Tel +33 1 44 27 34 39 Fax +33 1 44 27 34 45 Mobile +33 6 68 60 51 50 http://drosophile.org

3 3

Galaxy Security Notification
by Nate Coraor 12 Jun '13

12 Jun '13

A security vulnerability was recently discovered by Björn Grüning with Galaxy's "user impersonation" feature that can expose an administrator's active history to users whom they impersonate. Only Galaxy instances with `allow_user_impersonation = True` set in their configurations are affected, and only if an administrator makes use of the impersonation feature. By default, user impersonation is disabled. A fix (id: 9d42f1e32efb) has been provided in the stable branch of Galaxy. To apply the fix, ensure you are on the stable branch and upgrade to the latest changeset: % hg branch stable % hg pull -u For Galaxy installations on relatively old versions that administrators are not yet ready to upgrade, there are three workarounds. First, the patch can be downloaded and applied manually: % wget -o security.patch https://bitbucket.org/galaxy/galaxy-central/commits/9d42f1e32efb654fda1e011… and then: % hg patch security.patch or: % patch -p1 < security.patch Second, the impersonation feature can be disabled by setting the following option in Galaxy's configuration file: allow_user_impersonation = False In all of the above cases, the Galaxy server process(es) must be restarted for the change to take effect. Third, the feature can be left enabled and unpatched, and the vulnerability can be worked around by educating administrators who use the feature. As long as a new history is created by the administrator prior to switching to the impersonated user, no data will be exposed to the impersonated user. --nate Galaxy Team

1 0

Re: [galaxy-dev] Toolshed bug related to readme files
by Lance Parsons 12 Jun '13

12 Jun '13

I'm running the latest stable. The 500 error seems to be coming from the toolshed. Do you also get a 500 error at the toolshed URL in my previous message? There is nothing other than standard info line from the request in my pasted log. I'm not able to get to the log right now, but I'll send it once I get a chance tomorrow. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University -------- Original message -------- From: Greg Von Kuster <greg(a)bx.psu.edu> Date: 06/11/2013 10:00 PM (GMT-05:00) To: Lance Parsons <lparsons(a)princeton.edu> Cc: galaxy-dev(a)lists.bx.psu.edu Subject: Re: [galaxy-dev] Toolshed bug related to readme files Hi Lance, What version of Galaxy are you running? Is the 500 error coming from your Galaxy server or the tool shed server? Can you send the relevant information from your Galaxy paster log when attempting to reinstall? Thanks, Greg Von Kuster On Jun 11, 2013, at 3:52 PM, Lance Parsons <lparsons(a)princeton.edu> wrote: > I have run into an issue when trying to reinstall a tool that was originally installed via migration scripts (toolshed version of picard). > > When I attempt to reinstall (after successfully uninstalling) I get a 500 error. After a bit of poking around, it seems that this URL generates the 500 error: > > http://toolshed.g2.bx.psu.edu/repository/get_readme_files?name=picard&owner… > > Looks like this changeset is relevant: https://bitbucket.org/galaxy/galaxy-central/commits/f72fd96d3fac5e17ec5c9aa… > > > -- > Lance Parsons - Scientific Programmer > 134 Carl C. Icahn Laboratory > Lewis-Sigler Institute for Integrative Genomics > Princeton University > > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/

2 2