- galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] collections with more than 25,000 items
by Mohammad Heydarian 30 Aug '18

30 Aug '18

We initially had issues with Collections containing thousands of datasets that was related to the limit of jobs in the Slurm queue - excessively increasing this limit fixed our issue. Cheers, Mo Heydarian On Thu, Aug 30, 2018 at 11:18 AM Peter Cock <p.j.a.cock(a)googlemail.com> wrote: > There is a sweet spot for splitting your BLAST query fasta file > by sequence - one big file with 25000 sequences is not great, > but one sequence per file is the worst possible option. > > This is due to all the extra overheads, you would have 25000 > jobs submitted to the cluster, each of which would load the > BLAST binary and database off disk etc. And there are also > going to be Galaxy overheads with a large collection as well. > > I would suggest somewhere around 500 to 1000 gene sequences > per FASTQ query file is likely a safe choice. If you have very > long sequences (e.g. chromosomes or contigs), then use less. > > As to the number of threads for each BLAST job, more is better, > but what to pick will depend on your cluster and how often there > are threads free on nodes. I would suggest trying 4, 8 or 16 threads. > > I hope that helps. > > Peter > > > On Thu, Aug 30, 2018 at 3:50 PM Jochen Bick <jochen.bick(a)usys.ethz.ch> > wrote: > > > > Thanks Peter, > > > > so my idea was to split my problem into single blast jobs and run them > > only on one core... > > So my file has 25000 sequences and I'm blasting them against all NCBI > > proteins (nr). This just take to long time. I guess because the database > > is also very big? I tested this on the first 10 sequences and it took > > about 10mins. But maybe this is still not faster than running all at > once? > > How many cores would you give such a job? > > > > Cheers Jochen > > > > On 30.08.2018 16:44, Peter Cock wrote: > > > If there are any limits, it would be down to the Galaxy Admin's job > > > settings - something generic with collections. > > > > > > Personally I've not done this - I tend to concatenate FASTA files > > > to make large files with multiple sequences instead. > > > > > > (And then we have the optional task splitting enabled so that Galaxy > > > breaks up the multiple-sequence FASTA file into chunks which > > > get shared out on our cluster for better throughput before > > > concatenating the output back into a single file.) > > > > > > Peter > > > On Thu, Aug 30, 2018 at 3:37 PM Jochen Bick <jochen.bick(a)usys.ethz.ch> > wrote: > > >> > > >> Hi, > > >> > > >> is there any limit to run BLAST jobs from a collection of single FASTA > > >> files? I started a job but is does not get executed... its just > sending > > >> for about an hour. > > >> > > >> Cheers Jochen > > >> > > >> -- > > >> ETH Zurich > > >> *Jochen Bick* > > >> Animal Physiology > > >> Institute of Agricultural Sciences > > >> Postal address: Universitätstrasse 2 / LFW B 58.1 > > >> Office: Tannenstrasse 1 / TAN D 6.2 > > >> 8092 Zurich, Switzerland > > >> > > >> Phone +41 44 632 28 25 > > >> jochen.bick(a)usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> > > >> www.ap.ethz.ch > > >> ___________________________________________________________ > > >> Please keep all replies on the list by using "reply all" > > >> in your mail client. To manage your subscriptions to this > > >> and other Galaxy lists, please use the interface at: > > >> https://lists.galaxyproject.org/ > > >> > > >> To search Galaxy mailing lists use the unified search at: > > >> http://galaxyproject.org/search/ > > > > -- > > ETH Zurich > > *Jochen Bick* > > Animal Physiology > > Institute of Agricultural Sciences > > Postal address: Universitätstrasse 2 / LFW B 58.1 > > Office: Tannenstrasse 1 / TAN D 6.2 > > 8092 Zurich, Switzerland > > > > Phone +41 44 632 28 25 > > jochen.bick(a)usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> > > www.ap.ethz.ch > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/

1 0

Re: [galaxy-dev] collections with more than 25,000 items
by Peter Cock 30 Aug '18

30 Aug '18

There is a sweet spot for splitting your BLAST query fasta file by sequence - one big file with 25000 sequences is not great, but one sequence per file is the worst possible option. This is due to all the extra overheads, you would have 25000 jobs submitted to the cluster, each of which would load the BLAST binary and database off disk etc. And there are also going to be Galaxy overheads with a large collection as well. I would suggest somewhere around 500 to 1000 gene sequences per FASTQ query file is likely a safe choice. If you have very long sequences (e.g. chromosomes or contigs), then use less. As to the number of threads for each BLAST job, more is better, but what to pick will depend on your cluster and how often there are threads free on nodes. I would suggest trying 4, 8 or 16 threads. I hope that helps. Peter On Thu, Aug 30, 2018 at 3:50 PM Jochen Bick <jochen.bick(a)usys.ethz.ch> wrote: > > Thanks Peter, > > so my idea was to split my problem into single blast jobs and run them > only on one core... > So my file has 25000 sequences and I'm blasting them against all NCBI > proteins (nr). This just take to long time. I guess because the database > is also very big? I tested this on the first 10 sequences and it took > about 10mins. But maybe this is still not faster than running all at once? > How many cores would you give such a job? > > Cheers Jochen > > On 30.08.2018 16:44, Peter Cock wrote: > > If there are any limits, it would be down to the Galaxy Admin's job > > settings - something generic with collections. > > > > Personally I've not done this - I tend to concatenate FASTA files > > to make large files with multiple sequences instead. > > > > (And then we have the optional task splitting enabled so that Galaxy > > breaks up the multiple-sequence FASTA file into chunks which > > get shared out on our cluster for better throughput before > > concatenating the output back into a single file.) > > > > Peter > > On Thu, Aug 30, 2018 at 3:37 PM Jochen Bick <jochen.bick(a)usys.ethz.ch> wrote: > >> > >> Hi, > >> > >> is there any limit to run BLAST jobs from a collection of single FASTA > >> files? I started a job but is does not get executed... its just sending > >> for about an hour. > >> > >> Cheers Jochen > >> > >> -- > >> ETH Zurich > >> *Jochen Bick* > >> Animal Physiology > >> Institute of Agricultural Sciences > >> Postal address: Universitätstrasse 2 / LFW B 58.1 > >> Office: Tannenstrasse 1 / TAN D 6.2 > >> 8092 Zurich, Switzerland > >> > >> Phone +41 44 632 28 25 > >> jochen.bick(a)usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> > >> www.ap.ethz.ch > >> ___________________________________________________________ > >> Please keep all replies on the list by using "reply all" > >> in your mail client. To manage your subscriptions to this > >> and other Galaxy lists, please use the interface at: > >> https://lists.galaxyproject.org/ > >> > >> To search Galaxy mailing lists use the unified search at: > >> http://galaxyproject.org/search/ > > -- > ETH Zurich > *Jochen Bick* > Animal Physiology > Institute of Agricultural Sciences > Postal address: Universitätstrasse 2 / LFW B 58.1 > Office: Tannenstrasse 1 / TAN D 6.2 > 8092 Zurich, Switzerland > > Phone +41 44 632 28 25 > jochen.bick(a)usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> > www.ap.ethz.ch

2 1

collections with more than 25,000 items
by Jochen Bick 30 Aug '18

30 Aug '18

Hi, is there any limit to run BLAST jobs from a collection of single FASTA files? I started a job but is does not get executed... its just sending for about an hour. Cheers Jochen -- ETH Zurich *Jochen Bick* Animal Physiology Institute of Agricultural Sciences Postal address: Universitätstrasse 2 / LFW B 58.1 Office: Tannenstrasse 1 / TAN D 6.2 8092 Zurich, Switzerland Phone +41 44 632 28 25 jochen.bick(a)usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> www.ap.ethz.ch

2 1

java options
by Matthias Bernt 28 Aug '18

28 Aug '18

Dear list, I'm struggling to set java options for galaxy tools. Currently I use `<env file="job_setup_script.sh"/> in my job_conf.xml and in the script I explored two ways to set java options: `_JAVA_OPTIONS="-Xmx5G -Xms1G -Djava.io.tmpdir=/work/songalax/tmp"` But then java prints "Picked up _JAVA_OPTIONS: ..." to stderr which is be interpreted as error by some tools. If I'm correct this is currently the default .. I have seen this happening, but can't remember the name of the tool. So I switched to this 'trick': `alias java='java -Xmx5G -Xms1G -Djava.io.tmpdir=/work/songalax/tmp'` but the problem is that the tool script is called and not sourced and therefore aliases are not used (could this be changed?). Furthermore, for tools which explicitly set the java options my settings would be ignored anyway (an example is the MSGFPlusAdapter which calles `java -Xmx3500m` .. so my -Xmx is overwritten .. I passing GALAXY_MEMORY_MB to the corresponding parameter might be a solution for this tool). Any thought or suggestions on how to set the parameters in a production environment are very welcome. Cheers, Matthias -- ------------------------------------------- Matthias Bernt Bioinformatics Service Molekulare Systembiologie (MOLSYB) Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 482296, m.bernt(a)ufz.de, www.ufz.de Sitz der Gesellschaft/Registered Office: Leipzig Registergericht/Registration Office: Amtsgericht Leipzig Handelsregister Nr./Trade Register Nr.: B 4703 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig Wilfried Kraus Wissenschaftlicher Geschäftsführer/Scientific Managing Director: Prof. Dr. Dr. h.c. Georg Teutsch Administrative Geschäftsführerin/ Administrative Managing Director: Prof. Dr. Heike Graßmann -------------------------------------------

3 4

European Galaxy Days, 19/20 November 2018 in Freiburg, Germany.
by Hans-Rudolf Hotz 27 Aug '18

27 Aug '18

Dear all We are happy to announce the European Galaxy Days which will be held 19 and 20 November 2018 in Freiburg, Germany. https://galaxyproject.org/events/2018-europe-dev/ Similar to the events we organized in 2016, 2014 and 2012 the aim is to discuss the status of the Galaxy project, new developments, interfaces to other systems, extensions and best practice in reproducible research. The program is planed as follows: Monday, November 19th We intend to have a full day of talks from you. We especially encourage Galaxy User to present their work with Galaxy. Please indicate your interest in presenting when you register (we will then get in contact with you). Or contact us directly. Tuesday, November 20th This day is more on the technical side, with presentation, tutorials and hands on exercises. Currently we plan to discuss/present: - Galaxy and Machine learning - Single Cell RNA Seq analysis with Galaxy - Combining Galaxy with Shiny Registration is now open (space is limited). There will be no conference fee. Though, you need to cover for your food and accommodation (we are currently looking for sponsors to cover the lunches on the two days). We recommend you register soon to secure your spot: https://tinyurl.com/EGD2018 Looking forward to see many of you in Freiburg Regards, Jean-François, Bjoern and Hans-Rudolf

1 0

Permanently Deleted Data
by Emm L 19 Aug '18

19 Aug '18

Hello, I am emailing in desperate hope for some help here. I trimmed my files and waited for days for them to finish. When they were finished, I meant to purge my deleted data to make space in my account...I accidentally purged my entire history. The galaxy account I am using is under the email emm.elle(a)outlook.com I was wondering if someone in IT could get me back my history. The history name is WES for ML3. Thank you very much. Michelle Lim

2 1

tool datatypes
by Matthias Bernt 17 Aug '18

17 Aug '18

Dear list, just a request for links to documentation: How can I realize tool specific data types. I'm just developing a set of tools that need their own data types, but I don't want to add them to Galaxy's core data types (yet). I've seen examples of tools that had a datatypes_conf.xml. So I created one, but it seems that it is ignored. Cheers, Matthias -- ------------------------------------------- Matthias Bernt Bioinformatics Service Molekulare Systembiologie (MOLSYB) Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 482296, m.bernt(a)ufz.de, www.ufz.de Sitz der Gesellschaft/Registered Office: Leipzig Registergericht/Registration Office: Amtsgericht Leipzig Handelsregister Nr./Trade Register Nr.: B 4703 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig Wilfried Kraus Wissenschaftlicher Geschäftsführer/Scientific Managing Director: Prof. Dr. Dr. h.c. Georg Teutsch Administrative Geschäftsführerin/ Administrative Managing Director: Prof. Dr. Heike Graßmann -------------------------------------------

2 7

Unable to install tools from toolshed/mercurial issue in release_18.05 instance
by Peter Briggs 16 Aug '18

16 Aug '18

Dear devs I've encountered what appears to be a subtle bug with release 18.05, which breaks the installation of tools from the toolshed, and appears to be a result of not having mercurial (hg) available in /usr/bin on the system that Galaxy is installed on (in this case Scientific Linux 6.5). When attempting to install a tool (e.g. devteam/fastqc) from the main toolshed via the admin interface, after clicking "install" the tool installation status goes immediately to "Error". The tool repository isn't cloned to "shed_tools" and no dependencies are installed. I've been unable to find any error messages in the logs. However, attempting to install via the API does return the message: Error cloning repository: [Errno 2] No such file or directory which comes from the "clone_repository" function in lib/tool_shed/util/hg_util.py (when something goes wrong with the "hg clone ..." command). Installing Mercurial 1.3 via yum on the server and attempting tool installation again gives a slightly different error via the API: Error cloning repository: Command '['hg', 'clone', '-r', u'17', u'https://toolshed.g2.bx.psu.edu/repos/devteam/fastqc', u'/XXXXXXXXXXXXXX/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/fastqc/c15237684a01/fastqc']' returned non-zero exit status 255 Output was: abort: No such file or directory: /XXXXXXXXXXXXXXXXXX/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/fastqc/c15237684a01/fastqc Uninstalling the system Mercurial and instead installing version 3.7.3 and making a link from /usr/bin/hg seems to fix the problem, and tools can be installed without problems. I couldn't find any evidence of this being reported before, and I don't know if I've missed some configuration detail which means that Galaxy isn't picking up hg from its virtualenv instead of /usr/bin. Has anyone else encountered this problem? and is there a fix/workaround for it (other than horrible links to /usr/bin/hg)? Any advice gratefully received! Best wishes Peter -- Peter Briggs peter.briggs(a)manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482

3 3

one dataset won't import
by Langhorst, Brad 10 Aug '18

10 Aug '18

Hi Galaxy folks: We have one data set in a shared library that will not import to a history. I am able to import it using the page for that dataset, but not from the list of all datasets in the folder. I’m not sure the source of the error - this particular row does not look odd to me.= Any Ideas? here’s a row that will import: <tr class="dataset_row light library-row" data-id="e369c8b86fc4c943" style="display: table-row;"><td><span title="Dataset" class="fa fa-file-o"></span></td><td style="text-align: center; "><input style="margin: 0;" type="checkbox"></td><td><a href="#folders/F09026885d1dcbcc0/datasets/e369c8b86fc4c943" class="library-dataset">MM809_9_x4_B5.1.fastq</a><a></a></td><td></td><td>fastqsanger</td><td>31.5 MB</td><td>2018-08-10 02:38 PM</td><td></td><td><span data-toggle="tooltip" data-placement="top" title="" style="color:grey;" class="fa fa-globe fa-lg" data-original-title="Unrestricted dataset"></span><a href="#folders/F09026885d1dcbcc0/datasets/e369c8b86fc4c943/permissions"><button data-toggle="tooltip" data-placement="top" class="primary-button btn-xs permissions-dataset-btn" title="" data-original-title="Permissions of 'MM809_9_x4_B5.1.fastq'"><span class="fa fa-group"></span> Manage</button></a></td></tr> here’s one that will not: <td><span title="Dataset" class="fa fa-file"></span></td><td style="text-align: center; "><input style="margin: 0;" type="checkbox"></td><td><a href="#folders/F09026885d1dcbcc0/datasets/7582870467626242" class="library-dataset dark">MM809_9_x4_B5.2.fastq</a><a class="dark"></a></td><td></td><td>fastqsanger</td><td>31.5 MB</td><td>2018-08-10 02:38 PM</td><td></td><td><span data-toggle="tooltip" data-placement="top" title="" style="color:grey;" class="fa fa-globe fa-lg" data-original-title="Unrestricted dataset"></span><a href="#folders/F09026885d1dcbcc0/datasets/7582870467626242/permissions" class="dark"><button data-toggle="tooltip" data-placement="top" class="primary-button btn-xs permissions-dataset-btn" title="" data-original-title="Permissions of 'MM809_9_x4_B5.2.fastq'"><span class="fa fa-group"></span> Manage</button></a></td> here’s the error in the js console: TypeError: n.substring is not a function[Learn More] library-foldertoolbar-view.js:1205:16<webpack:///galaxy/scripts/mvc/library/library-foldertoolbar-view.js> findCheckedItems/< library-foldertoolbar-view.js:1205:16<webpack:///galaxy/scripts/mvc/library/library-foldertoolbar-view.js> each jquery.js:365<webpack:///galaxy/scripts/libs/jquery/jquery.js> each jquery.js:137<webpack:///galaxy/scripts/libs/jquery/jquery.js> findCheckedItems library-foldertoolbar-view.js:1201:8<webpack:///galaxy/scripts/mvc/library/library-foldertoolbar-view.js> processImportToHistory library-foldertoolbar-view.js:283:28<webpack:///galaxy/scripts/mvc/library/library-foldertoolbar-view.js> importAllIntoHistory library-foldertoolbar-view.js:272:12<webpack:///galaxy/scripts/mvc/library/library-foldertoolbar-view.js> Import library-foldertoolbar-view.js:231:32<webpack:///galaxy/scripts/mvc/library/library-foldertoolbar-view.js> dispatch jquery.js:4736<webpack:///galaxy/scripts/libs/jquery/jquery.js> e.dispatch jquery.event.drag.js:377<webpack:///galaxy/scripts/libs/jquery/jquery.event.drag.js> add/m.handle jquery.js:4549<webpack:///galaxy/scripts/libs/jquery/jquery.js>

1 0

links to UCSC Genome Browser dev server
by Dave Clements 10 Aug '18

10 Aug '18

Hi All, If you run a Galaxy server and that server links to the UCSC Genome Browser *development* server, then you will need to update the URL you link to. See below for details from Ann Zweig at UCSC. I am writing because a few of your servers provide links to the UCSC Genome > Browser development server at genome-test.*cse*.ucsc.edu or genome-test. > *soe*.ucsc.edu (see list below). This weekend we are retiring that server > and bringing it back to life next week with a new name: genome-test.*gi*. > ucsc.edu. Cheers, Dave C -- https://galaxyproject.org/ https://usegalaxy.org/

1 0