- galaxy-dev - lists.galaxyproject.org

Inconsistent menus in Galaxy Tool Shed
by Peter Cock 02 May '13

02 May '13

Hi all, This is just a usability bug, but I think it should be a simple fix that would make working with the Tool Shed a bit easier as the current menus seem randomised. --------------- If I click on "Repositories I own", then each tool has a drop down with only one action: * Mark as deprecated [image: Inline image 1] (This puzzles me - why just that rarely used option? Why not say upload, or view tests?) --------------- If I click on a tool, I go to the management page. Here the top right menu offers: * View tool functional test results * Upload files to repository * View change log * Browse or delete repository tip files * Reset all repository metadata * Mark repository as deprecated * Download as a .tar.gz file * Download as a .tar.bz2 file * Download as a .zip file [image: Inline image 2] --------------- If I go to the functional test results, the menu changes: * Manage repository * Upload files to repository * View change log * Browse or delete repository tip files * Download as a .tar.gz file * Download as a .tar.bz2 file * Download as a .zip file [image: Inline image 3] i.e. Aside the obvious change given which page we are on, why are "Reset all repository metadata" and "Mark repository as deprecated" missing? --------------- If I goto browse/delete tip files: * Manage repository * Upload files to repository * Browse or delete repository tip files * Download as a .tar.gz file * Download as a .tar.bz2 file * Download as a .zip file [image: Inline image 5] Notice "View change log" is now missing (as well as functional tests etc). --------------- If I go to the change log page, the menu changes again: * Upload files to repository * Manage repository * Browse or delete repository tip files * Download as a .tar.gz file * Download as a .tar.bz2 file * Download as a .zip file [image: Inline image 4] Again, several things are missing - and in particular the functional tests entry. To me that is highly relevant when looking at the change log. Also note the ordering of "Upload files to repository" and "Manage repository" has flipped which is confusing. --------------- My suggested fix is quite simple - use the same drop down menu for the repository actions on every page, either with the current page still on the list but disabled (and visually distinct as marker) is possible, or simply removed. Thanks, Peter

2 3

ASN.1 (text and binary) formats in Galaxy & Tool Shed
by Peter Cock 02 May '13

02 May '13

Hello all, Although they are these days also offering XML for many tools, the NCBI still make heavy use of the older ASN.1 file format (both as plain text and binary). This crops up in BLAST (e.g. as the BLAST archive format, or as dustmasker output), in the Entrez Utilities (e.g. for sequence data as an alternative to GenBank for FASTA format etc, or pubmed, etc) and also for 3D structures. I think it could make sense to define generic 'asn1' and 'asn1-binary' formats in the Galaxy core (name suggestions welcome), and even perhaps 'ncbi-asn1' and 'ncbi-asn1-binary' too. Then ToolShed entries can define domain specific subclasses. For instance, the BLAST+ wrapper could include definitions for the dustmasker output, and perhaps the BLAST archive format too. Separately anyone working with 3D structures as ASN.1 could define another sub-format, etc. I see this as a clear analogy to the assorted XML file formats in existence - defined in Galaxy as subclasses of the core XML format included with the Galaxy core. Would a pull request implementing this be acceptable? Peter P.S. Does anyone know an authoritative source for the MIME types used by the NCBI? Using the BLAST website they offer plain text ASN.1 just as text/plain, likewise efetch also seems to use text/plain for ASN.1 downloads. However I've seen references to chemical/ncbi-asn1-ascii and chemical/ncbi-asn1-binary mime-types mentioned, e.g. http://www.ncbi.nlm.nih.gov/data_specs/asn/NCBI_all.asn i.e. It appears that 3D structure NCBI ASN.1 files use a well defined MIME type, while most NCBI ASN.1 text files default to text/plain - which we can handle nicely in Galaxy as subclasses.

4 10

Output external file stored outside galaxy-dist dir
by Venice Margarette Juanillas 02 May '13

02 May '13

Hi all, I'm Venice and I'm new in Galaxy. I installed a local Galaxy and I just want to know how I can output to history an external file stored outside my galaxy-dist folder. I created an xml explicitly stating the absolute path, but when I run the tool, it says "Either this dataset does not exist or you do not have permission to access it." I don't know what to do so I do hope you can help me with this. Thanks and Best Regards, Venice Juanillas

1 0

Galaxy Keeps Freezing
by Kim Spradling 02 May '13

02 May '13

I have been trying to join the intervals of two datasets side-by-side, but the job never finishes. I've gotten a message stating that Galaxy is unavailable a couple of times, but I have tried running this job multiple times with no success. Any help will be greatly appreciated. Thanks, Kim

2 1

Installing FreeBayes via Toolshed: Tool dependency installation error
by Iry Witham 01 May '13

01 May '13

Hi All, I am trying to get Freebayes installed on a cloud man instance from the galaxy toolshed. I have successfully installed the Samtools 0.1.18 package, but the freebayes package errors. The Error is: Tool shed repository: freebayes Tool shed repository changeset revision: 28e422c589ac Tool dependency status: Error Tool dependency installation error: /bin/sh: cmake: not found make[1]: *** [../bamtools/lib/libbamtools.a] Error 127 make: *** [all] Error 2 Tool dependency installation directory: /mnt/galaxyTools/tools/freebayes/0.9.6_9608597d12e127c847ae03aa03440ab63992fedf/devteam/freebayes/28e422c589ac I get the following message after the error occurs: This tool dependency is not installed correctly (see the Tool dependency installation error below). Choose Uninstall this tool dependency from the Repository Actions menu, correct problems if necessary, and try installing the dependency again. I have followed this procedure and still have the same error. I have looked at the Installation.log file and this is what I found: ############################################# git clone --recursive git://github.com/ekg/freebayes.git STDOUT Initialized empty Git repository in /mnt/galaxyData/tmp/tmpCUrxI7/freebayes/.git/ Submodule 'bamtools' (git://github.com/pezmaster31/bamtools.git) registered for path 'bamtools' Submodule 'intervaltree' (git://github.com/ekg/intervaltree.git) registered for path 'intervaltree' Submodule 'vcflib' (git://github.com/ekg/vcflib.git) registered for path 'vcflib' Initialized empty Git repository in /mnt/galaxyData/tmp/tmpCUrxI7/freebayes/bamtools/.git/ Submodule path 'bamtools': checked out '3fe66b92028b60629826d827ee9058405fa7e4be' Initialized empty Git repository in /mnt/galaxyData/tmp/tmpCUrxI7/freebayes/intervaltree/.git/ Submodule path 'intervaltree': checked out 'd151b487804861dc9f932e9f1fe4f8c499673cec' Initialized empty Git repository in /mnt/galaxyData/tmp/tmpCUrxI7/freebayes/vcflib/.git/ Submodule path 'vcflib': checked out '480d96e16a552d36369dc9cb2b39ceedb260944a' Submodule 'fastahack' (git://github.com/ekg/fastahack.git) registered for path 'fastahack' Submodule 'fsom' (git://github.com/ekg/fsom.git) registered for path 'fsom' Submodule 'intervaltree' (git://github.com/ekg/intervaltree.git) registered for path 'intervaltree' Submodule 'multichoose' (git://github.com/ekg/multichoose.git) registered for path 'multichoose' Submodule 'smithwaterman' (git://github.com/ekg/smithwaterman.git) registered for path 'smithwaterman' Submodule 'tabixpp' (git://github.com/ekg/tabixpp.git) registered for path 'tabixpp' Initialized empty Git repository in /mnt/galaxyData/tmp/tmpCUrxI7/freebayes/vcflib/fastahack/.git/ Submodule path 'fastahack': checked out '6acaf9875f2dbcdf86c9c690a8d188000e90d388' Initialized empty Git repository in /mnt/galaxyData/tmp/tmpCUrxI7/freebayes/vcflib/fsom/.git/ Submodule path 'fsom': checked out '1b888f32c4bd571d1a0bea88d39cd7d7f23ea4b8' Initialized empty Git repository in /mnt/galaxyData/tmp/tmpCUrxI7/freebayes/vcflib/intervaltree/.git/ Submodule path 'intervaltree': checked out '1290744283cef8076bb8a2968d4899b7228435f4' Initialized empty Git repository in /mnt/galaxyData/tmp/tmpCUrxI7/freebayes/vcflib/multichoose/.git/ Submodule path 'multichoose': checked out '73d35daa18bf35729b9ba758041a9247a72484a5' Initialized empty Git repository in /mnt/galaxyData/tmp/tmpCUrxI7/freebayes/vcflib/smithwaterman/.git/ Submodule path 'smithwaterman': checked out 'd9724f156c07cf16d00d251bebc20c6383eb6bde' Initialized empty Git repository in /mnt/galaxyData/tmp/tmpCUrxI7/freebayes/vcflib/tabixpp/.git/ Submodule path 'tabixpp': checked out 'fbdde774adfc79ca75a65b4321b89b67cd81b921' ############################################# ############################################# git clone --recursive git://github.com/ekg/freebayes.git STDERR ############################################# ############################################# git checkout 9608597d12e127c847ae03aa03440ab63992fedf STDOUT M bamtools M vcflib ############################################# ############################################# git checkout 9608597d12e127c847ae03aa03440ab63992fedf STDERR Note: checking out '9608597d12e127c847ae03aa03440ab63992fedf'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example: git checkout -b new_branch_name HEAD is now at 9608597... example pipeline script ############################################# ############################################# git submodule update --recursive STDOUT Submodule path 'bamtools': checked out '8a5d6508d8fc52b25c9129037e44cc448bafebc7' Submodule path 'vcflib': checked out '3f0d5e19596252b858a7f6eff1578e4295f549c8' Submodule path 'fastahack': checked out 'd1bcd7a2d52ee0e2823884c8ede22ce228d3c060' Submodule path 'smithwaterman': checked out '2882633a3b89f820d692a9b3316f4891dc3a1ff8' ############################################# ############################################# git submodule update --recursive STDERR warning: unable to rmdir fsom: Directory not empty ############################################# ############################################# make STDOUT cd src && make make[1]: Entering directory `/mnt/galaxyData/tmp/tmpCUrxI7/freebayes/src' cd ../bamtools && mkdir -p build && cd build && cmake .. && make make[1]: Leaving directory `/mnt/galaxyData/tmp/tmpCUrxI7/freebayes/src' ############################################# ############################################# make STDERR /bin/sh: cmake: not found make[1]: *** [../bamtools/lib/libbamtools.a] Error 127 make: *** [all] Error 2 ############################################# Any ideas on how to resolve this? Thanks, Iry Witham The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.

3 5

Re: [galaxy-dev] toolshed gives internal server error
by Greg Von Kuster 01 May '13

01 May '13

Hello Mattias, I've created the following Trello card for this and we'll get it fixed asap. Thanks for letting us know. https://trello.com/card/toolshed-unable-to-download-repositories-as-zipped-… Greg Von Kuster On Apr 29, 2013, at 3:35 AM, Hollander, Mattias de wrote: > Hi Greg, > > I am trying to download the wrappers for repeatmasker from the general > toolshed (http://toolshed.g2.bx.psu.edu/) but I retrieve an 'Internal > Server Error' when I want to download the tar.gz (or any other zipped > file) via the Repository Actions menu. > > Can you have a look at it? > > Thanks! > > Mattias

3 2

SnpEff Ver 3.1 produces 'no peek' results
by Iry Witham 01 May '13

01 May '13

Hi All, I have installed snpEff 3.1 from the galaxy toolshed into my cloud man instance. The tool runs to completion, but I get an empty file. It says as 'no peek' in the history. When I view the details and check the stderr and get the following: WARNING: Chromosome '@PG' not found. File '/mnt/galaxyData/files/000/dataset_58.dat', line 85 java.lang.ArrayIndexOutOfBoundsException: 5 at ca.mcgill.mcb.pcingola.fileIterator.SeqChangeFilePileUp.readNext(SeqChangeFilePileUp.java:69) at ca.mcgill.mcb.pcingola.fileIterator.SeqChangeFilePileUp.readNext(SeqChangeFilePileUp.java:1) at ca.mcgill.mcb.pcingola.fileIterator.FileIterator.hasNext(FileIterator.java:69) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEffCmdEff.iterateSeqChange(SnpEffCmdEff.java:139) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEffCmdEff.runAnalysis(SnpEffCmdEff.java:750) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEffCmdEff.run(SnpEffCmdEff.java:689) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEffCmdEff.run(SnpEffCmdEff.java:564) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEff.run(SnpEff.java:276) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEff.main(SnpEff.java:75) I reinstalled the tool and reran the job. Now I get this from the stderr: java.lang.ArrayIndexOutOfBoundsException: 7 at ca.mcgill.mcb.pcingola.fileIterator.SeqChangeFilePileUp.readNext(SeqChangeFilePileUp.java:70) at ca.mcgill.mcb.pcingola.fileIterator.SeqChangeFilePileUp.readNext(SeqChangeFilePileUp.java:1) at ca.mcgill.mcb.pcingola.fileIterator.FileIterator.hasNext(FileIterator.java:69) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEffCmdEff.iterateSeqChange(SnpEffCmdEff.java:139) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEffCmdEff.runAnalysis(SnpEffCmdEff.java:750) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEffCmdEff.run(SnpEffCmdEff.java:689) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEffCmdEff.run(SnpEffCmdEff.java:564) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEff.run(SnpEff.java:276) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEff.main(SnpEff.java:75) Any ideas? Thanks, Iry Witham The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.

1 0

Re: [galaxy-dev] BLAST Database error: No alias or index file found
by Mike Dyall-Smith 01 May '13

01 May '13

Thanks Peter. My answers are below: >What query sequences are you using? I have just been using one fasta protein sequence. >Meanwhile monitor the system with top Top says only a max of 27% cpu usage, but the linux screen eventually freezes, and I have to restart. I am not sure how to read out RAM and disk IO from top. >grep blastp paster.log Tried that, and it says there is no such file or directory >Could you try running BLAST from the host Mac OX Yes. And it works fine! I get a good match in a relatively short time. I then made a very small protein database, checked it by commandline blastp in both host OS X and in guest linux, and it worked fine. Added it to the blastdb_p.loc file, restarted and saw it listed in galaxy. Tried to use it for a blastp, and got the same error as before with the huge NCBI nr database. So, it is not a matter of size.... Thanks for your comments about RAM and blast searches. It gives me hope that I can get galaxy running usefully. I only chose biolinux because of the suite of programs and the apparent ease of use. The other reason was that I could not install galaxy on OS X (10.6). I get errors that others have noted on the discussion lists but no-one seems to have a solution for. regards, Mike DS -------------------------------------------------------------- On Tue, Apr 30, 2013 at 10:49 PM Peter Cock wrote: > Hi Mike, > > On Tue, Apr 30, 2013 at 12:54 PM, Mike Dyall-Smith > <mike.dyallsmith(a)gmail.com> wrote: > > Dear Peter, thanks for the advice. I think I can now run blastp from the > > commandline, both in the host and in the linux virtual machine. I say > > 'think' because it runs in both cases, but then either never completes (I > > cancelled after 30 min on OS X) or freezes (VBox/ubuntu). However, the > > blastp search in galaxy gives the same error as before. This, I don't > > understand. > > What query sequences are you using? I'd try just one or two protein > sequences in a small FASTA file, and rerun blastp against nr. > Meanwhile monitor the system with top or similar to see what > the CPU usage, RAM usage, and disk IO is like. > > That should help determine if this is a simple as not enough RAM > leading to lots of paging to disk, and therefore a very slow search. > > > I simplified the directory structure (so the path) to the database, and > > altered the appropriate configuration files (blastdb_p.loc, > > blast_environment.sh). With 'env', I see: > BLASTDB=/media/sf_mikeds_bioinf/db > > I also checked that "ls /media/sf_mikeds_bioinf/db/nr*" listed all the > > database files. > > > > The relevant lines are now: > > ----------------from blastdb_p.loc file------------------------- > > #Your blastdb_p.loc file should include an entry per line for each "base > > name" > > #you have stored. For example: > > # > > #nr_05Jun2010 NCBI NR (non redundant) 05 Jun 2010 > > /data/blastdb/05Jun2010/nr > > #nr_15Aug2010 NCBI NR (non redundant) 15 Aug 2010 > > /data/blastdb/15Aug2010/nr > > nr_08_Apr2013 NCBI_nrprot_08Apr2013 /media/sf_mikeds_bioinf/db/nr > > ... > > ---------------------------------------------------------------------- > > That looks OK. > > > The blast+ blastp error from galaxy is: > > ----------------------------------------------------------------------- > > An error occurred running this job: blastp: 2.2.26+ > > Package: blast 2.2.26, build Aug 15 2012 17:48:54 > > BLAST Database error: No alias or index file found for protein database > > [/media/sf_mikeds_bioinf/db/nr] in search path > > [/var/lib/galaxy-server/database/job_working_directory/000/18::] > > ----------------------------------------------------------------------- > > > > Very strange, clearly something is not right with the Galaxy config. > on the bright side, Galaxy is finding the binaries . Could you try this > on the Galaxy log file, > > $ grep blastp paster.log > > > However, while this might point to a real issue for the use of galaxy > within > > VirtualBox/ubuntu (and with the database on an external USB3 drive), I > > suspect my idea of running a local instance of galaxy on my macbook is > not > > possible. I have 8 Gb of RAM, and set 4Gb for use in the linux guest > system. > > The Genbank nr protein db directory has files totaling 24 Gb. > > The oldest cluster nodes we're still using have 8GB of RAM, and are > fine to run BLASTP against NR with. The previous nodes only have > 2GB and could not cope - so the threshold is somewhere in between. > I suspect your guest VM with only 4GB of RAM is struggling. > > Could you try running BLAST from the host Mac OX X instead, with > access to the full 8GB of RAM? > > > Running blastp > > from the commandline does not give a result in any reasonable length of > > time, even when a single query sequence is used (both on the host and > guest > > systems). Even if I were able to handle raw sequence datasets of moderate > > size (less than 1 Gb? as yet untested) in Galaxy, it would be of little > use > > to me if I can't blastp or blastx search the resulting contigs. Do I set > up > > galaxy to send blast search queries to NCBI (i.e. many thousands of > > queries??) or is there some more elegant solution? > > ... > Regards, > > Peter >

2 5

Adding data sources in the main galaxy instance
by Fingerman, Ian (NIH/NLM/NCBI) [E] 01 May '13

01 May '13

I have a question regarding data sources currently available on the main instance of Galaxy (https://main.g2.bx.psu.edu) How can one get added as a new data source? I work at the NCBI on the Epigenomics database (http://www.ncbi.nlm.nih.gov/epigenomics) . We currently hold a large volume of NGS data in the form of wig files, and some users have expressed interest in using Galaxy for data analysis. We provide an easy to use user interface for examining these data as seen here (http://www.ncbi.nlm.nih.gov/epigenomics/browse/) and currently host over 4200 data tracks. I think it would useful if we could integrate Galaxy functionality into our resource. I also thing providing our resource as a "Data Source" would also be convenient for users less familiar with our database but are current Galaxy users. I have been looking through the Galaxy wiki and I am struggling to find documentation that details step-by-step, what exactly needs to be done. One thing to note, I am not a developer, I'm the scientific lead for the project and my programming/developing skills are lacking. I was hoping someone could point me to thorough documentation mainly to pass on to developers on my team. I guess also understanding my options with regards to integrating or interfacing with Galaxy would be very valuable to me too. Thank you for any help/suggestions you may have. Ian Ian Fingerman, Ph.D. Staff Scientist NIH/NLM/NCBI Building 45, Room 4AN28D-29 45 Center Drive MSC-6510 Bethesda, MD 20894 Phone: (301) 496-6806 fingerma(a)ncbi.nlm.nih.gov

4 4

Creating multiple datasets in a libset
by Rob Leclerc 30 Apr '13

30 Apr '13

I'm looking at example_watch_folder.py and it's not clear from the example how you submit multiple datasets to a library. In the example, the first submit returns a libset [] with only a single entry and then proceeds to iterate through each dataset in the libset in the following section: data = {} data['folder_id'] = library_folder_id data['file_type'] = 'auto' data['dbkey'] = '' data['upload_option'] = 'upload_paths' *data['filesystem_paths'] = fullpath* data['create_type'] = 'file' libset = submit(api_key, api_url + "libraries/%s/contents" % library_id, data, return_formatted = False) time.sleep(5) for ds in libset: if 'id' in ds: wf_data = {} wf_data['workflow_id'] = workflow['id'] wf_data['history'] = "%s - %s" % (fname, workflow[ 'name']) wf_data['ds_map'] = {} for step_id, ds_in in workflow['inputs' ].iteritems(): wf_data['ds_map'][step_id] = {'src':'ld', 'id' :ds['id']} res = submit( api_key, api_url + 'workflows', wf_data, return_formatted=False) Rob Leclerc, PhD <http://www.linkedin.com/in/robleclerc> <https://twitter.com/#!/robleclerc> P: (US) +1-(917)-873-3037 P: (Shanghai) +86-1-(861)-612-5469 Personal Email: rob.leclerc(a)aya.yale.edu

2 6