- galaxy-dev - lists.galaxyproject.org

Custom Tool Dev Questions
by Ryan G 19 Jun '15

19 Jun '15

Hi all - I have a tool which is essentially an entire NGS pipeline bundled into a perl script and associated modules. I'd like to offer this tool to users via Galaxy. I'm trying to pass a bunch of fastq files to the tool, but noticed that operating on multiple datasets creates a separate job for each dataset. I want to pass all the datasets to the tool so only one galaxy job is actually created. Is this possible, and if so, how do I go about doing this? Ryan

2 2

BAM datatype - bam & bai
by Alexander Vowinkel 19 Jun '15

19 Jun '15

Hi, when I 'produce' a bam file in my output, I can download the bam file and a corresponding bai file. I am now wondering where does the bai file come from? When is it being generated, what code is used for that? When I do multiple actions with my bam file - is the index generated every time? (overhead) Where is the bai saved on disk (if it is)? I found that in the IUC gatk, a new bai is always generated when a gatk tool is started. This creates quite an overhead. Isn't there a better solution? @ref https://github.com/galaxyproject/tools-iuc/issues/194 Best, Alexander

2 3

macro tag naming conventions
by Alexander Vowinkel 19 Jun '15

19 Jun '15

Hi, in https://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax#Direct_XML_Macr…, macros are defined with <macro name="dbKeyActions"> [...] </macro> But when I look in IUC macros, I find the following: <xml name="..."> [...] </xml> Which one is the current standard? Best, Alexander

2 1

package_bowtie_2_2_4 Executable Permissions - can't run bowtie as non-galaxy user
by David Trudgian 19 Jun '15

19 Jun '15

Installed bowtie2 from the toolshed, which brings in package_bowtie_2_2_4, which downloads bowtie binaries from depot.galaxyproject.org http://depot.galaxyproject.org/package/linux/x86_64/bowtie2/bowtie2-2.2.4-l… Jobs submitted on our setup, running as a 'real user' (not galaxy user) via pulsar + SLURM fail with a permissions error. This is due to the permissions on some of the bowtie executables in the .tar.gz from depot being owner/group executable only. Other users cannot run bowtie2: -rwxrwx---. 1 galaxy galaxy 18K Oct 22 2014 bowtie2 If I download the tar.gz from depot manually and extract then I get: -rwxrwx--- 1 dtrudgian biohpc_admin 18K Oct 22 2014 bowtie2 Could the .tar.gz be fixed, so that the executables will work in a run as real user configuration? Is there any sensible way Galaxy could enforce permissions to guard against this issue? Cheers, -- David Trudgian Ph.D. Computational Scientist, BioHPC UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833 Please contact biohpc-help@utsouthwestern with general BioHPC inquries. ________________________________ UT Southwestern Medical Center The future of medicine, today.

2 2

Bug in commit 38679a0
by Ryan G 18 Jun '15

18 Jun '15

In scripts/functional_tests.py, line 71, the change is causing a directory called "test-data,https://github.com/galaxyproject/galaxy- test-data.git" to get created. I'm not sure that's the intention of this fix. default_galaxy_test_file_dir is assigned to test_dir, then never used again. test_dir is used to create the testing directory.

2 1

table_recalibration cannot be installed through tool_shed
by Jingchao Zhang 18 Jun '15

18 Jun '15

Hi all, The table_recalibration tool cannot be installed through tool_shed. There seems to be a broken link for the repository. (https://toolshed.g2.bx.psu.edu/repos/devteam/table_recalibration/gatk_table…) Here is the most recent log from the "Automated tool test results" 2015-05-07 10:37:18 Automated test environment Time tested: 2015-05-07 10:37:18 System: Linux 3.13.0-36-generic Architecture: x86_64 Python version: 2.7.6 Galaxy revision: v15.03.2-904-g3e80076 Galaxy database version: 128 Tool shed revision: 16935:64266df71fa7 Tool shed database version: 25 Tool shed mercurial version: 3.2.4 Installation errors This repository Error Error getting revision 30e1dd77e99c of repository table_recalibration owned by devteam: An entry for the repository was not found in the database. Can someone please help to fix this? Thanks, Jingchao

1 0

Fwd: Adding packages for bedtools and ucsc-utils for Trackster?
by Eric Rasche 18 Jun '15

18 Jun '15

(We discussed this over IRC, but responding to the list posting for posterity/searchability) The JSON.pm file isn't found in your PERL5LIB path because the original version of my JBrowse tool failed to specify the full dependency list and assumed you had some perl modules manually installed. The latest version of my JBrowse tool now includes the full set of dependencies. Since chatting on IRC, this version has been uploaded to the MTS and is available here: https://toolshed.g2.bx.psu.edu/view/iuc/jbrowse/497c6bb3b717 (For the rest of the mailing list reading this, an unofficial release announcement) JBrowse 1.11.6 is available as a tool which can be run and can display: - bed/gff/gff3 - BAM - Blast XML data (automatically converted to gapped GFF3 and displayed with evalue-based intensities) - BigWig data - VCF data putting it closer to being on-par with trackster. It's an excellent end-of-workflow step to summarize all of your generated genomic analyses. If any of the community have feature requests, send them my way! NB: BAM/BigWigs do NOT play nicely with galaxy if you aren't using X-SendFile. 2015-06-18 5:42 GMT-05:00 Peter van Heusden <pvh(a)sanbi.ac.za>: > Thanks Bjoern > > My initial install didn't pick up the Perl dependencies correctly, but I > re-installed and now the install seems to have the required modules but is > missing something in the environment variables. Specifically I get an error > import JSON.pm, see below. Note that the > jbrowse/1.11.6/iuc/package_jbrowse_1_11_6/d5ab749c9e97/lib/perl5 directory > isn't included in @INC and in fact the paths that *are* included seem to > imply that the wrong perl interpreter is being picked up. > > > Fatal error: Exit code 1 () > Can't locate JSON.pm in @INC (you may need to install the JSON module) (@INC contains: /net/datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/database/job_working_directory/000/29/dataset_29_files/JBrowse-1.11.6/bin/../src/perl5 /etc/perl /usr/local/lib/perl/5.18.2 /usr/local/share/perl/5.18.2 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.18 /usr/share/perl/5.18 /usr/local/lib/site_perl .) at /net/datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/database/job_working_directory/000/29/dataset_29_files/JBrowse-1.11.6/bin/../src/perl5/Bio/JBrowse/JSON.pm line 10. > BEGIN failed--compilation aborted at /net/datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/database/job_working_directory/000/29/dataset_29_files/JBrowse-1.11.6/bin/../src/perl5/Bio/JBrowse/JSON.pm line 10. > Compilation failed in require at /net/datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/database/job_working_directory/000/29/dataset_29_files/JBrowse-1.11.6/bin/../src/perl5/Bio/JBrowse/Cmd/FormatSequences.pm line 21. > BEGIN failed--compilation aborted at /net/datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/database/job_working_directory/000/29/dataset_29_files/JBrowse-1.11.6/bin/../src/perl5/Bio/JBrowse/Cmd/FormatSequences.pm line 21. > Compilation failed in require at bin/prepare-refseqs.pl line 7. > BEGIN failed--compilation aborted at bin/prepare-refseqs.pl line 7. > Traceback (most recent call last): > File "/net/datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/shed_tools/toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/2c9e5136b416/jbrowse/jbrowse.py", line 44, in <module> > process_genome(jbrowse_dir, os.path.realpath(args.genome.name)) > File "/net/datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/shed_tools/toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/2c9e5136b416/jbrowse/jbrowse.py", line 17, in process_genome > subprocess.check_output(['perl', 'bin/prepare-refseqs.pl', '--fasta', genome], cwd=jbrowse_dir) > File "/usr/lib/python2.7/subprocess.py", line 573, in check_output > raise CalledProcessError(retcode, cmd, output=output) > subprocess.CalledProcessError: Command '['perl', 'bin/prepare-refseqs.pl', '--fasta', '/net/datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/database/files/000/dataset_3.dat']' returned non-zero exit status 2 > > > On 18 June 2015 at 10:01, Bjoern Gruening <bjoern.gruening(a)gmail.com> > wrote: > >> Peter, >> >> I'm in a hurry ... have a look at >> https://github.com/galaxyproject/tools-iuc/tree/master/tools/jbrowse >> Developed by IUC member Eric Rasche. >> >> Cheers, >> Bjoern >> >> >> On 18.06.2015 10:00, Peter van Heusden wrote: >> >> I've tried that - I've added the Environment Modules for the relevant >> tools to the local_env.sh file, so they're there in the environment when >> Galaxy runs. These tools are scheduled via our cluster, however, so I don't >> think the environment passes across to them. They depend on these Galaxy >> packages, however, and those (bedtools and ucsc-utils) are now installed, >> so I'm not sure why it can't pick them up from the shed_tools. >> >> On 18 June 2015 at 09:10, Hans-Rudolf Hotz <hrh(a)fmi.ch> wrote: >> >>> >>> >>> On 06/18/2015 09:06 AM, Peter van Heusden wrote: >>> >>>> Hi there >>>> >>>> I'm trying to visualise some of our genome data using Trackster. Initial >>>> input is 790,000 lines of GFF3. When I try and create the visualisation, >>>> I get errors from two tools: >>>> >>>> Convert GFF to Interval Index on data 3 >>>> -- an out of memory error - I'll try and fix this by upping the memory >>>> limit for our Galaxy jobs >>>> >>>> Convert BED, GFF, or VCF to BigWig on data 3 >>>> -- bedtools and bedGraphToBigWig not found errors. I tried to resolve >>>> these by installing the ucsc-tools and bedtools packages - and they did >>>> install correctly - but I still get these file not found errors. >>>> >>>> >>> Hi Peter >>> >>> make sure 'galaxy' (ie the user galaxy is running as) has the newly >>> installed tools in its path. You might need to add them to "run.sh" and >>> restart the server. >>> >>> >>> Hans-Rudolf >>> >>> >>> Any idea how to fix this second problem? >>>> >>>> Thanks! >>>> Peter >>>> P.S. as an alternative to Trackster, does anyone have Galaxy interfaced >>>> to JBrowse? >>>> >>>> >>>> ___________________________________________________________ >>>> Please keep all replies on the list by using "reply all" >>>> in your mail client. To manage your subscriptions to this >>>> and other Galaxy lists, please use the interface at: >>>> https://lists.galaxyproject.org/ >>>> >>>> To search Galaxy mailing lists use the unified search at: >>>> http://galaxyproject.org/search/mailinglists/ >>>> >>>> >> >> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> https://lists.galaxyproject.org/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ >> >> >> > > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ > -- Eric Rasche Programmer II Center for Phage Technology Rm 312A, BioBio Texas A&M University College Station, TX 77843 404-692-2048 esr(a)tamu.edu rasche.eric(a)yandex.ru

1 0

GalaxyAdmins Online Meetup this Thursday: Peter Briggs on Galactic Engineering at Manchester
by Dave Clements 18 Jun '15

18 Jun '15

Hello all, The June 2015 online GalaxyAdmins meetup <https://wiki.galaxyproject.org/Community/GalaxyAdmins/Meetups/2015_06_18> will be this Thursday, 18 June, at 5pm Central European Time (see your local time <http://bit.ly/1F7zN8l>). Peter Briggs <http://galacticengineer.blogspot.co.uk/> of the University of Manchester <http://www.manchester.ac.uk/> and author of the *Galactic Engineer <http://galacticengineer.blogspot.co.uk/>* blog will talk about his Galaxy work deploying Galaxy and developing tools within the Bioinformatics Core Facility <http://www.ls.manchester.ac.uk/research/facilities/#bioinf> at the University of Manchester. John Chilton <https://wiki.galaxyproject.org/JohnChilton> will also give an update on recent and upcoming work on Planemo <https://github.com/galaxyproject/planemo>, a set of command-line utilities to assist in building and publishing Galaxy tools. We'll also discuss the upcoming GalaxyAdmins meetup, and the hackathons at GCC2015 <http://gcc2015.tsl.ac.uk/>. We'll post a link on the meetup page <https://wiki.galaxyproject.org/Community/GalaxyAdmins/Meetups/2015_06_18> on Thursday. Hope to see/hear you on Thursday, Dave C -- http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ https://wiki.galaxyproject.org/

4 4

Adding packages for bedtools and ucsc-utils for Trackster?
by Peter van Heusden 18 Jun '15

18 Jun '15

Hi there I'm trying to visualise some of our genome data using Trackster. Initial input is 790,000 lines of GFF3. When I try and create the visualisation, I get errors from two tools: Convert GFF to Interval Index on data 3 -- an out of memory error - I'll try and fix this by upping the memory limit for our Galaxy jobs Convert BED, GFF, or VCF to BigWig on data 3 -- bedtools and bedGraphToBigWig not found errors. I tried to resolve these by installing the ucsc-tools and bedtools packages - and they did install correctly - but I still get these file not found errors. Any idea how to fix this second problem? Thanks! Peter P.S. as an alternative to Trackster, does anyone have Galaxy interfaced to JBrowse?

3 4

functional tests / database lock
by Ryan G 17 Jun '15

17 Jun '15

Hi all - I'm running functional tests on a set of tools using 'run_tests.sh -sid ...'. A few of the tests fail. When I go back and run the tool tests individually, they succeed. So I looked into why they fail when running them in batch, and it turns out to be a database lock. I'm using the default sqllite database that run_tests.sh creates. Does this need to be tuned somehow? galaxy.jobs.handler INFO 2015-06-12 10:28:35,854 (14) Job dispatched Error - <class 'sqlalchemy.exc.OperationalError'>: (OperationalError) database is locked u'SELECT api_keys.id AS api_keys_id, api_keys.create_time AS api_keys_create_time, api_keys.user_id AS api_keys_user_id, api_keys."key" AS api_keys_key \nFROM api_keys \nWHERE api_keys."key" = ?' (u'e4fe9bf787edb9ef746714beb6212bc8',) URL: http://localhost:8363/api/histories/cb227cec8ca83994?key=e4fe9bf787edb9ef74… File '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ app_iter = self.application(environ, sr_checker) File '/apps/sys/galaxy/galaxy-v15.05-staging/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/apps/sys/galaxy/galaxy-v15.05-staging/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/base.py', line 132 in __call__ return self.handle_request( environ, start_response ) File '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/base.py', line 159 in handle_request trans = self.transaction_factory( environ ) File '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py', line 71 in <lambda> self.set_transaction_factory( lambda e: self.transaction_chooser( e, galaxy_app, session_cookie ) ) File '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py', line 102 in transaction_chooser return GalaxyWebTransaction( environ, galaxy_app, self, session_cookie ) File '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py', line 194 in __init__ self.error_message = self._authenticate_api( session_cookie ) File '/apps/sys/galaxy/galaxy-v15.05-staging/lib/galaxy/web/framework/webapp.py', line 283 in _authenticate_api provided_key = self.sa_session.query( self.app.model.APIKeys ).filter( self.app.model.APIKeys.table.c.key == api_key ).one() File 'build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py', line 2184 in one File 'build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py', line 2227 in __iter__ File 'build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py', line 2242 in _execute_and_instances File 'build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py', line 1449 in execute File 'build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py', line 1584 in _execute_clauseelement File 'build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py', line 1698 in _execute_context File 'build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py', line 1691 in _execute_context File 'build/bdist.linux-x86_64/egg/sqlalchemy/engine/default.py', line 331 in do_execute OperationalError: (OperationalError) database is locked u'SELECT api_keys.id AS api_keys_id, api_keys.create_time AS api_keys_create_time, api_keys.user_id AS api_keys_user_id, api_keys."key" AS api_keys_key \nFROM api_keys \nWHERE api_keys."key" = ?' (u'e4fe9bf787edb9ef746714beb6212bc8',)

4 9