- galaxy-dev - lists.galaxyproject.org

December 2014 Galaxy Newsletter
by Dave Clements 01 Dec '14

01 Dec '14

Hello all, As always there's a lot going on in the Galaxy this month. "Like what?" you say. Well, read the dang December Galaxy Newsletter <https://wiki.galaxyproject.org/GalaxyUpdates/2014_12> we say! Highlights include: - Galaxy Day! In Paris! This Wednesday! <https://wiki.galaxyproject.org/GalaxyUpdates/2014_12#Galaxy_Day:_3_December…> - Near Richmond, Virginia? There's a Galaxy Workshop at Virginia State U on December 12 <https://wiki.galaxyproject.org/GalaxyUpdates/2014_12#Intro_to_Galaxy_Worksh…> . - GCC2015 needs sponsors <https://wiki.galaxyproject.org/GalaxyUpdates/2014_12#GCC2015:_6-8_July.2C_N…> ! - Other upcoming events <https://wiki.galaxyproject.org/GalaxyUpdates/2014_12#Other_Events> on two continents - *96 new papers <https://wiki.galaxyproject.org/GalaxyUpdates/2014_12#New_Papers>*, including 6 highlighted papers, referencing, using, extending, and implementing Galaxy. - Job openings at 7+ organizations <https://wiki.galaxyproject.org/GalaxyUpdates/2014_12#Who.27s_Hiring> - A new mailing list: Galaxy-Training <https://wiki.galaxyproject.org/GalaxyUpdates/2014_12#New_Galaxy-Training_Ma…> - 15 new ToolShed repositories from 10 contributors <https://wiki.galaxyproject.org/GalaxyUpdates/2014_12#ToolShed_Contributions> - And, 10 other juicy <https://wiki.galaxyproject.org/GalaxyUpdates/2014_12#Other_News> (well maybe not *juicy*, but certainly not *crunchy*) bits of news <https://wiki.galaxyproject.org/GalaxyUpdates/2014_12#Other_News> Happy December! Dave Clements and the *crisp* Galaxy Team <https://wiki.galaxyproject.org/GalaxyTeam> -- http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ https://wiki.galaxyproject.org/

1 0

Creation of a backup/migrate script for data/users
by Rémy Dernat 28 Nov '14

28 Nov '14

Hi, I would like to create a code to backup and migrate(*) galaxy datas, based on the api and/or with the script I did here: https://github.com/remyd1/galaxy_debug_params/blob/master/source/Templates/… To do that, I would use the galaxy model, to retrieve everything in the old one and recreate everything in the new one. However, I would like to be able to do that without my web server running. That is to say, a "cold" migration to avoid any current task to write additionnal things into the database. Obviously, that is a problem for the api because this one use the web server. Could you give some advice to do that ? Should I recreate a code from 0 ? Could I use the code in the api ? Is there a way to create a code that could use the api differently (recreate a copy of the api without the web server) ? Kind regards, Remy (*) this is important; that means you could import data from an old galaxy to a recent one (or the opposite); a simple backup of files and a dump of the db is not enough.

2 2

datatype dependencies
by Eric Rasche 28 Nov '14

28 Nov '14

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Let's pretend for a second that I'm rather lazy (oh...wait), and I have ZERO interest in writing datatype parsers to sniff and validate whether or not a specific file is a specific datatype. I'm a sysadmin and bioinformatician, and I've worked with dozens of libraries that exist to parse file formats, and they all die in flames when I feed them bad data. Would it be possible to somehow define requirements for datatypes? I don't want to take on the burden of code I write saying "yes, I've sniffed+validated this and it is absolutely a genbank file". That's a lot of responsibility, especially if people have malformed genbank files and their tools fail as a result. I would like to do this with BioPython and turf the validation to another library that exists to parse genbank files, that will raise and exception if they're invalid. > def sniff(self, filename): > from Bio import SeqIO > try: > self.records = list(SeqIO.parse( filename, "genbank" )) > return True > except: > self.records = None > return False > > def validate(self, dataset): > from Bio import SeqIO > errors = list() > try: > self.records = list(SeqIO.parse( dataset.file_name, "genbank" )) > except Exception, e: > errors.append(e) > return errors > > def set_meta(self, dataset, **kwd): > if self.records is not None: > dataset.metadata.number_of_sequences = len(self.records) so much easier! And I can shift the burden of validation and sniffing to upstream, rather than any failures being my fault and requiring maintenance of a complex sniffer. Cheers, Eric - -- Eric Rasche Programmer II Center for Phage Technology Texas A&M University College Station, TX 77843 404-692-2048 esr(a)tamu.edu rasche.eric(a)yandex.ru -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQIcBAEBAgAGBQJTyBmyAAoJEMqDXdrsMcpVQa0P/jj0edAKM6QsodhRWHglR92W tej1tJjtPgtJ15wsFzq6wVfhbL5J39ytsWjjtk//jhVNXh4FEE/OFZe6Nx9uTFKP ybazyTrLSCrxsST+w+Rx8Q9vfzShr87vjP+fC1k5i2EZOgogPOcQml1ouOHHjC6z pArrwPOvL3ZxWJG7oEcZjUjrPD8+ffhfQ/x096YYIMw7Hg74d50ARwtawJRoslZD JnYWa+aUOcsvC3QMrLKkDm4qBaTHa5x7x7P07Lcx7X65iMPDcuMZNtImiLztNscF QwbbdJdcs8oeSRRnmKgAllRAKf4dMeiyaSI+muVzNlpvLlSMZBNawD0bO1OXmIQH vAaV0eU+rYmDJSGo330o+RydvlDJENTXOkDt0TxmvfYAPtg2TlJCiWUdL7V1LqqF n8J5Z7Cu/sqRGSr5ww6KY27QHq6TU1WZDsVZiyEWJeKg3HGzp0MUmzMdr7iSZawK gnZxv6qg3+FlSqA30niyAuxEq588vS8uEFjjOfhnNLsUM7FAuFANF5z9bPOhG2qM Xjc3/NY7NsERd9nsIwfRuz0DWni8upvZ39vfeRZ3OAW9NwjRzqXrQiQp08XHa934 z4EBnpcWc9rNSV/3APF/imecBTOoiKtZfzIfILLtOPGE407Bmd8cE8hWyW7ipvrT QU6DIimj3eoMn+elXDfX =M+s5 -----END PGP SIGNATURE-----

4 7

galaxy fails to start when setuptools>=6.1
by Evan Bollig PhD 26 Nov '14

26 Nov '14

I tested all versions of setuptools available in PyPI from 0.6c11 up to 7.0. I found that galaxy fails to fetch eggs with the following error when setuptools > 6.0.2 (i.e., 6.1 and 7.0). ##### * Creating directory in /mnt for galaxy data * Starting Galaxy Galaxy Some eggs are out of date, attempting to fetch... Warning: MarkupSafe (a dependent egg of Mako) cannot be fetched Traceback (most recent call last): File "./scripts/fetch_eggs.py", line 46, in <module> c.resolve() # Only fetch eggs required by the config File "/opt/galaxy/web/lib/galaxy/eggs/__init__.py", line 347, in resolve egg.resolve() File "/opt/galaxy/web/lib/galaxy/eggs/__init__.py", line 192, in resolve if e.args[1].key != e.args[0].key: IndexError: tuple index out of range Fetch failed. ##### Cheers, -Evan Bollig Research Associate | Application Developer | User Support Consultant Minnesota Supercomputing Institute 599 Walter Library 612 624 1447 evan(a)msi.umn.edu boll0107(a)umn.edu

2 1

galaxy with torque
by Fernandez Edgar 26 Nov '14

26 Nov '14

Hello all, My name is Edgar Fernandez. I'm a sys. admin. at University of Montreal. I've contacted you a while back about installing galaxy and I've successfully done it on a redhat 6 server. I see myself in a situation where I need to utilise all my redhat servers (who are identical to the server running the galaxy website). I've also successfully installed a server torque with compute notes and clients nodes. What are the last step to make the link between galaxy and torque? Also, once that connection is made, how will galaxy keep track of the jobs sent? I mean who will it know this job that just finished is for this user and not another ? Also, my torque installation is so that my server running galaxy is a submit node and a client node. I hope this is not a problem. Please help! Cordialement / Regards, Edgar Fernandez System Administrator (Linux) Direction Générale des Technologies de l'Information et de la Communication * Bur. : 1-514-343-6111 poste 16568 Université de Montréal PAVILLON ROGER-GAUDRY, bureau X-218

2 2

Re: [galaxy-dev] samtools: command not found in bowtie2
by Eric Kuyt 26 Nov '14

26 Nov '14

Could you try going to Admin -> Manage Installed tool shed repositories uninstall -> reinstall package_samtools_0_1_18 On 26 November 2014 at 16:55, Anna Terry <iseeglass(a)googlemail.com> wrote: > yway, I am also getting the problem with samtools filter sam or bam -- Central Veterinary Institute of Wageningen UR (CVI) Department of Infection Biology PO box 65, 8200 AB Lelystad, NL Visiting address: ASG, Edelhertweg 15, 8219 PH Lelystad Tel: +31-(0)320-293391 Fax: +31-(0)320-238153 E-mail: eric.kuijt(a)wur.nl Web: http://www.cvi.wur.nl

3 5

Re: [galaxy-dev] samtools: command not found in bowtie2
by Anna Terry 26 Nov '14

26 Nov '14

So I actually still have a problem, (I guess the setting was still there somewhere from the bash_profile) Anyway, I am also getting the problem with samtools filter sam or bam Traceback (most recent call last): File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/runners/local.py", line 129, in queue_job job_wrapper.finish( stdout, stderr, exit_code ) File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 998, in finish dataset.datatype.set_meta( dataset, overwrite=False ) # call datatype.set_meta directly for the initial set_meta call during dataset creation File "/home/galaxy/galaxy-dist/lib/galaxy/datatypes/binary.py", line 251, in set_meta raise Exception, "Error Setting BAM Metadata: %s" % stderr Exception: Error Setting BAM Metadata: /bin/sh: samtools: command not found I have tried also what you said, linking the samtools binary to default, but still the same problem. Any other ideas? On 26 November 2014 at 14:01, Anna Terry <iseeglass(a)googlemail.com> wrote: > Great, I've done it and it works. > > galaxy-dist/tools/samtools/default > cd galaxy-dist/tools/samtools/default/ > cp -rs ~/galaxy-dist/tools/samtools/0.1.19/* . > > Thanks Bjoern and Eric > > > On 26 November 2014 at 13:54, Eric Kuyt <eric.kuijt(a)wur.nl> wrote: > >> Bjoerns comment was the right one, samtools isn't automatically installed >> as a dependency somehow. >> >> so the best option is to install samtools >> https://toolshed.g2.bx.psu.edu/view/devteam/package_samtools_0_1_19 >> >> and link the installed binary to ~/tool_deps/samtools/0.19/iuc/2828727744/ >> to ~/tool_deps/samtools/default >> >> Eric >> >> >> On 26 November 2014 at 14:44, Anna Terry <iseeglass(a)googlemail.com> >> wrote: >> >>> Here is the contents of the log file: >>> >>> Successfully downloaded from url: >>> http://depot.galaxyproject.org/package/linux/x86_64/bowtie2/bowtie2-2.1.0-l… >>> >>> >>> On 26 November 2014 at 13:26, Eric Kuyt <eric.kuijt(a)wur.nl<mailto: >>> eric.kuijt(a)wur.nl>> wrote: >>> Hi Anna, >>> >>> Setting your own path variables shouldn't be necessary for toolshed >>> tools, so probably something is wrong with the installation. could you post >>> your >>> >>> >>> tooldependencies/bowtie2/2.1.0/devteam/package_bowtie2_2_1_0/017a00c265f1/INSTALLATION.log >>> >>> file? >>> >>> On 26 November 2014 at 13:49, Anna Terry <iseeglass(a)googlemail.com >>> <mailto:iseeglass@googlemail.com>> wrote: >>> Hi, >>> I installed bowtie2 from the toolshed, but get the following error when >>> I run it: >>> >>> >>> Traceback (most recent call last): >>> File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/runners/local.py", line >>> 129, in queue_job >>> job_wrapper.finish( stdout, stderr, exit_code ) >>> File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 998, >>> in finish >>> dataset.datatype.set_meta( dataset, overwrite=False ) # call >>> datatype.set_meta directly for the initial set_meta call during dataset >>> creation >>> File "/home/galaxy/galaxy-dist/lib/galaxy/datatypes/binary.py", line >>> 251, in set_meta >>> raise Exception, "Error Setting BAM Metadata: %s" % stderr >>> Exception: Error Setting BAM Metadata: /bin/sh: samtools: command not >>> found >>> >>> I'm not sure why this won't work? >>> samtools is a dependency for the bowtie2 tool and is installed, but >>> cannot seem to find it. >>> There must me something slightly wrong on my server as I can't see loads >>> of errors like this on this mailing list. >>> >>> I have added the path from the samtools env file to .bash_profile for >>> galaxy and this allows this tool to run, but is not the correct solution, >>> as doesn't work when versions change. >>> >>> Could you let me know where I'm going wrong >>> Thanks >>> Anna >>> >>> >>> >>> -- >>> Central Veterinary Institute of Wageningen UR (CVI) >>> Department of Infection Biology >>> PO box 65, 8200 AB Lelystad, NL >>> Visiting address: ASG, Edelhertweg 15, 8219 PH Lelystad >>> >>> Tel: +31-(0)320-293391 >>> Fax: +31-(0)320-238153 >>> E-mail: eric.kuijt(a)wur.nl<mailto:eric.kuijt@wur.nl> >>> Web: http://www.cvi.wur.nl >>> >>> >> >> >> -- >> Central Veterinary Institute of Wageningen UR (CVI) >> Department of Infection Biology >> PO box 65, 8200 AB Lelystad, NL >> Visiting address: ASG, Edelhertweg 15, 8219 PH Lelystad >> >> Tel: +31-(0)320-293391 >> Fax: +31-(0)320-238153 >> E-mail: eric.kuijt(a)wur.nl >> Web: http://www.cvi.wur.nl >> > >

1 1

Egg build failed for bx_python 0.7.2
by Rafael Guerrero-Preston 26 Nov '14

26 Nov '14

I have sensitive clinical data and would like to run Galaxy on a local production server. While creating/acquiring the required Python eggs to start the server for the first time using the sh run.sh command I got the following error message. How should I proceed? gcc-4.0 -fno-strict-aliasing -fno-common -dynamic -arch ppc -arch i386 -g -O2 -DNDEBUG -g -O3 -Isrc/kent -Isrc -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c lib/bx/bitset.c -o build/temp.macosx-10.3-fat-2.7/lib/bx/bitset.o unable to execute gcc-4.0: No such file or directory error: command 'gcc-4.0' failed with exit status 1 Traceback (most recent call last): File "scripts/scramble.py", line 50, in <module> egg.scramble() File "/Users/rafaelguerrero-preston/galaxy-dist/lib/galaxy/eggs/scramble.py", line 57, in scramble self.run_scramble_script() File "/Users/rafaelguerrero-preston/galaxy-dist/lib/galaxy/eggs/scramble.py", line 206, in run_scramble_script raise ScrambleFailure( self, "%s(): Egg build failed for %s %s" % ( sys._getframe().f_code.co_name, self.name, self.version ) ) galaxy.eggs.scramble.ScrambleFailure: run_scramble_script(): Egg build failed for bx_python 0.7.2

2 1

importing a large number of samples via a sample sheet
by Ryan G 26 Nov '14

26 Nov '14

Bumping this question....no response in over a week... Hi all - I've recently come back to Galaxy to see how its progressed over the last few years. The tools shed is completely new to me and will take some exploring on my part. My immediate question is if there is a way to import a large number of samples associated with a project via a sample sheet (txt) file? I have a text file with 3 columns: sample name, location of fastq files. I haven't seen a way to import this and haven't seen anything in the docs either. I suspect the answer is no, but I'm hoping the answer is actually yes....?

2 4

samtools: command not found in bowtie2
by Anna Terry 26 Nov '14

26 Nov '14

Hi, I installed bowtie2 from the toolshed, but get the following error when I run it: Traceback (most recent call last): File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/runners/local.py", line 129, in queue_job job_wrapper.finish( stdout, stderr, exit_code ) File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 998, in finish dataset.datatype.set_meta( dataset, overwrite=False ) # call datatype.set_meta directly for the initial set_meta call during dataset creation File "/home/galaxy/galaxy-dist/lib/galaxy/datatypes/binary.py", line 251, in set_meta raise Exception, "Error Setting BAM Metadata: %s" % stderr Exception: Error Setting BAM Metadata: /bin/sh: samtools: command not found I'm not sure why this won't work? samtools is a dependency for the bowtie2 tool and is installed, but cannot seem to find it. There must me something slightly wrong on my server as I can't see loads of errors like this on this mailing list. I have added the path from the samtools env file to .bash_profile for galaxy and this allows this tool to run, but is not the correct solution, as doesn't work when versions change. Could you let me know where I'm going wrong Thanks Anna

2 1