- galaxy-dev - lists.galaxyproject.org

new datasets appear in database/files/000/
by Jochen Bick 22 May '19

22 May '19

Hi, today we found out that some of our files in your galaxy instance are new in "database/files/000/" which is the first folder of your database files. Could some please explain me how galaxy generates new file names? and how this is possible? Thanks in advance Jochen -- ETH Zurich *Jochen Bick* Animal Physiology Institute of Agricultural Sciences Postal address: Universitätstrasse 2 / LFW B 58.1 8092 Zurich, Switzerland Office: Eschikon 27 8315 Lindau, Switzerland Phone +41 52 354 92 06 jochen.bick(a)usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> www.ap.ethz.ch

2 2

Re: galaxy-dev Digest, Vol 155, Issue 3
by Matthias Bernt 22 May '19

22 May '19

Dear Jochen, I don't know if its excpected. But at least I can tell that the same is happening here. And I can add if the 000 folder contains 000-999 the pattern repeats in 001. Cheers, Matthias On 21.05.19 17:38, galaxy-dev-request(a)lists.galaxyproject.org wrote: > Coming back to my question: > > this is how an ls -ls looks in my database folder... > > 2. Apr 14:40 000/ > 9. Nov 2015 005/ > 19. Nov 2018 007/ > 19. Nov 2018 008/ > 3. Dez 2015 010/ > 10. Jan 2017 012/ > 10. Jan 2017 013/ > 29. Jun 2017 014/ > 4. Jul 2017 015/ > 9. Jan 2018 018/ > 29. Jun 2017 020/ > 21. Sep 2016 021/ > 20. Mär 2018 032/ > 29. Jan 09:38 033/ > 29. Jan 10:32 034/ > 16. Jul 2018 035/ > 30. Okt 2018 036/ > 30. Okt 2018 037/ > 30. Okt 2018 038/ > 30. Okt 2018 039/ > 29. Aug 2018 070/ > 29. Aug 2018 071/ > 29. Aug 2018 072/ > 29. Aug 2018 073/ > 29. Aug 2018 074/ > 29. Aug 2018 075/ > 29. Aug 2018 076/ > 29. Aug 2018 077/ > 29. Aug 2018 078/ > 29. Aug 2018 079/ > 29. Aug 2018 080/ > 29. Aug 2018 081/ > 29. Aug 2018 082/ > 29. Aug 2018 083/ > 29. Aug 2018 084/ > 29. Aug 2018 085/ > 29. Aug 2018 086/ > 29. Aug 2018 087/ > 29. Aug 2018 088/ > 29. Aug 2018 089/ > 29. Aug 2018 090/ > 29. Aug 2018 091/ > 29. Aug 2018 092/ > 29. Aug 2018 093/ > 29. Aug 2018 094/ > 29. Aug 2018 095/ > 31. Aug 2018 096/ > 30. Aug 2018 097/ > 1. Sep 2018 098/ > 24. Okt 2018 099/ > > > if you look into the 000 folder it restarts: > > 10. Dez 19:36 100/ > 30. Nov 10:45 101/ > 10. Dez 19:39 102/ > 10. Dez 19:37 103/ > 11. Jan 15:13 104/ > 11. Jan 11:31 105/ > 22. Jan 14:33 106/ > 8. Feb 10:46 107/ > 14. Feb 15:47 108/ > 21. Mär 11:00 109/ > 2. Apr 15:22 110/ > 21. Mai 17:17 111/ > > is this the expected behavior? > > The only thing we did was we archived some of the very early generated > datasets. Is this maybe the problem? > > > Cheers Jochen > > > On 14.05.19 15:32, Jochen Bick wrote: >> Hi, >> >> today we found out that some of our files in your galaxy instance are >> new in "database/files/000/" which is the first folder of your database >> files. Could some please explain me how galaxy generates new file names? >> and how this is possible? >> >> Thanks in advance Jochen >> > -- ETH Zurich *Jochen Bick* Animal Physiology Institute of Agricultural > Sciences Postal address: Universitätstrasse 2 / LFW B 58.1 8092 Zurich, > Switzerland Office: Eschikon 27 8315 Lindau, Switzerland Phone +41 52 > 354 92 06 jochen.bick(a)usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> > www.ap.ethz.ch -- ------------------------------------------- Matthias Bernt Bioinformatics Service Molekulare Systembiologie (MOLSYB) Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 482296, m.bernt(a)ufz.de, www.ufz.de Helmholtz-Zentrum für Umweltforschung GmbH - UFZ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 1269, Fax +49 341 235 1468 (optional) max.mustermann(a)ufz.de, www.ufz.de Sitz der Gesellschaft/Registered Office: Leipzig Registergericht/Registration Office: Amtsgericht Leipzig Handelsregister Nr./Trade Register Nr.: B 4703 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig Wilfried Kraus Wissenschaftlicher Geschäftsführer/Scientific Managing Director: Prof. Dr. Georg Teutsch Administrative Geschäftsführerin/Administrative Managing Director: Dr. Sabine König

2 1

URL prefix doesn't work properly in Galaxy v18.09
by Ping Luo 20 May '19

20 May '19

I configured URL prefix in galaxy 18.09. It has problems of resolving some of the links, including: workflow visualize -> create visualization shared data -> workflows shared data -> visualizations I will use workflow as an example to illustrate what happens. When the mouse cursor is put on workflow, the URL shown in at the left bottom of the browser shows https://xxx.xxx.tamu.edu/galaxy/workflows/list But after I clicked 'workflow', the real URL shown in the address bar is actually https://xxx.xxxx.tamu.edu/galaxy/galaxy//workflows/list?__identifer=sk0xsyw… Of course, there is no such a page and the browser shows a blank page. If I manually remove one galaxy in the URL, the page is correctly shown. Beside the four mentioned above, other links work fine. If galaxy is configured without prefix, then there is no problem. I also tested v18.05. With all the configuration being the same, this version doesn't have the problem with prefix like v18.09 does. So I think this is a bug in v18.09 with URL prefix. Ping

2 3

Tagging collection datasets from file with hashtags
by Charles Girardot 20 May '19

20 May '19

Dear all, I am trying to tag datasets with hashtags (so they propagate) with tags like #key:value . The idea is to transport information through the workflow that is used at the end of the WF by a custom tool to rename and organise datasets depending on their tags I can do this in the GUI: I tag datasets manually with "#key:value” tags, create a collection with all of them and run a WF => the tags properly propagate. However, when I try to do the same using the collection tool “Tag elements from file”, it does not work properly : the tags are visible in the collection produced by the “Tag elements from file” but are not propagated. I assume this means that the “Tag elements from file” escapes the “#” somehow and my tags are simple tags, not hashtags. Note that my input file is like : element_identifier1 #key1:value1 #key2:value1 element_identifier2 #key1:value2 #key2:value2 Now the questions : - Is this a bug or intended ? - where can I find the file “tag_collection_from_file.xml”, I can t locate it :-/ - any suggestion for a fix ? Thx a bunch Charles

1 0

Share Docker Volumes when integrating Containers in Galaxy
by Schäfer, Richard 20 May '19

20 May '19

Hello, I want to integrate a Docker container as a tool in Galaxy. This works great. However, I need the forward a Volume to the container (at least that how it works outside of Galaxy). In particular, this would be the BLAST database. This means, I would want to share the location specified in blastdb.loc and forward it to the container. Is there a way to do this? I see that there is an option to expose volumes to containers by specifying it in the job.conf.xml <param id="docker_volumes">$defaults,/mnt/galaxyData/libraries:ro,/mnt/galaxyData/indices:ro</param> However, I guess this will then be done for all containers? And how do I integrate blastdb.loc in that command? For that, my tool config looks like this: <param name="database" type="select" display="radio" label="NCBI-nt database"> <options from_file="blastdb.loc"> <column name="name" index="1"/> <column name="value" index="2"/> </options> </param> I simply want to access the path inside of the container and therefore shared it with the container. Thanks for any help Cheers Rich

2 1

GCC2019 Early Reg ends this Friday
by Dave Clements 15 May '19

15 May '19

Hi All, The latest GCC2019 news: - A (final) reminder that early registration for the 2019 Galaxy Community Conference ENDS THIS FRIDAY NIGHT, 17 May, at midnight Freiburg time (CEST / GMT +2). After that registration goes up by ~ 60%. So, register this week! https://galaxyproject.org/events/gcc2019/registration/ - All accepted speakers have been notified and talks are being added to the schedule as presenters confirm. There were far more talk submissions this year than in any previous year and that is reflected in the breadth and quality of the accepted talks. https://gcc2019.sched.com/ - Poster and demo submission is still open! However, space *will* fill up and when it's gone, it's gone. Got something to present? Submit it *soon*. https://galaxyproject.org/events/gcc2019/abstracts/ We hope to see you in Freiburg! Dave C on behalf of the GCC2019 Organizers ------------------ 2019 Galaxy Community Conference (GCC2019) 1-8 July, Freiburg, Germany https://galaxyproject.org/gcc2019 ------------------ The 2019 Galaxy Community Conference (GCC2019) will be held 1-8 July in Freiburg, Germany. GCC brings together hundreds of faculty, clinicians, researchers, and students, all working in and supporting data intensive science that is accessible, sharable and reproducible. GCC2019 features oral presentations, lightning talks, posters, demos, birds-of-a-feather gatherings (BoFs), training, a CollaborationFest, and plenty of opportunities for networking. Presentations will cover the full spectrum of Galaxy applications, enhancements and deployments. If you are working in data intensive science then GCC2019 is an ideal conference for sharing your work, learning from others, and finding new collaborators. Registration Early registration starts at €49/day for students and postdocs, and €79 / day for other academics and non-profit researchers. Childcare is available and travel fellowships are available as well. Early registration ends 17 May, when rates go up by 60%. So, register early. About Galaxy Galaxy (https://galaxyproject.org/) is a platform for data integration and analysis in the life sciences. It enables researchers to build, run, share, and repeat their own complex computational analyses using only a web browser and without having to first learn system administration and command line interfaces. The Galaxy Project is driven by a vibrant community who publish workflows and analyses, wrap new tools, maintain and enhance the source code, provide support, and write documentation and training materials. Galaxy is open-source and freely available, and is deployed in hundreds of organizations, running on everything from laptops through supercomputers to public and private clouds. Over 150 of these platforms are publicly available and can be used with little or no setup. Thousands of tools have been ported to Galaxy ("wrapped") and are deployable from the Galaxy Tool Shed. Galaxy was developed to support life science research, but the software is domain agnostic and is now used in domains as diverse as natural language processing, constructive solid geometry, and social science. -- https://galaxyproject.org/gcc2019/ https://galaxyproject.org/ https://usegalaxy.org/

1 0

Galaxy jobs on Apache Mesos with chronos.py runner
by Pietro Mandreoli 14 May '19

14 May '19

Dear experts, I’m trying to run Galaxy (version 18.05) jobs with Apache MESOS (version 1.5.0). I configured a NFS between galaxy and mesos cluster nodes, thus sharing the directories /home/galaxy and /path/to/galaxy/database with the directories (citations, compiled_templates, files, ftp, home, job_working_directory, object_store_cache, tmp). Then I have changed the job_conf.xml accordingly: ( https://gist.github.com/pmandreoli/6ffba03193717393a2322586686f9aed ). It works fine with my very simple test wrapper here: https://gist.github.com/pmandreoli/ce120612afd0ac9ee80ce70c90e7d324 Then I enabled mulled containers in the galaxy.yml file in order to test the configuration using fastQC (version 0.72, owner devteam). In this case the job was correctly executed on the mesos node (using the container quay.io/biocontainers/fastqc:0.11.8--1) but the results were not linked in the history (please see the attached fig1). [fig1. history panel screenshot for fastQC job on chronos destination] You can have a look to the output here: https://gist.github.com/pmandreoli/bbbeb2eab5c1d1772872220c01678e15 I checked the docker run command on chronos and the working directory is located on /root/working. In order to check if the problems is related to my job_conf.xml configuration and not to the mulled containers i changed the job_conf.xml to run jobs locally ( https://gist.github.com/pmandreoli/484566b2c548d39d8bddb5aa54461ecc ) and i have run the same tool (fastQC 0.72). In this case everything was fine. Is my job_conf.xml configuration correct? I would also like to ask if it is possible to add more than one volume to the docker container run on MESOS in order to add the location for the reference data, located on my mesos slave node in /cvmfs. I tried to modify the job_conf.xml block in this way <param id="volumes">/export/galaxy/database/,/cvmfs/</param> but the job failed. The docker run command sent to chronos was indeed wrong: “-v /export/galaxy/database/,/cvmfs/:/export/galaxy/database/,/cvmfs/:rw “. This is probably due to the definition of the “volumes” field in the chronos.py runner: https://github.com/galaxyproject/galaxy/blob/3b3b52f013ac8c6b5bf8a4765f9fe9… which, if I understand well, is allowing to use only one path. Did I miss something? Any suggestion and correction is, of course, more than welcome. Best regards, Pietro Mandreoli <signaturebeforequotedtext></signaturebeforequotedtext><signatureafterquotedtext></signatureafterquotedtext>

2 1

Random failed job after update 18.05 -> 19.01
by edouard.hirchaud＠anses.fr 10 May '19

10 May '19

Hello all, After updating our galaxy instance (18.05 ->19.01) we have a strange problem. When we run or re-run a job from a history the job failed randomly. We have no errors in log file (level_log DEBUG) but in the dataset we see "cannot import name find_instance_nested" Cheers Edouard

1 0

galaxy not running after galaxy.yml changes
by Jochen Bick 10 May '19

10 May '19

Hi all, we are testing a newly cloned galaxy. Which was running as expected a few hours ago. After some modification to galaxy.yml file we got this error message but after using the original galaxy.yml version it still does not run. Regenerating static plugin directories. Activating virtualenv at .venv Traceback (most recent call last): File "./scripts/manage_tool_dependencies.py", line 60, in <module> main() File "/opt/galaxy/galaxy/lib/galaxy/util/script.py", line 45, in main action_func(args, kwargs) File "./scripts/manage_tool_dependencies.py", line 20, in _init_if_needed _build_dependency_manager_no_config(kwargs) File "./scripts/manage_tool_dependencies.py", line 48, in _build_dependency_manager_no_config dependency_manager = DependencyManager(**dependency_manager_kwds) File "/opt/galaxy/galaxy/lib/galaxy/tools/deps/__init__.py ", line 77, in __init__ self.dependency_resolvers = self.__build_dependency_resolvers(conf_file) File "/opt/galaxy/galaxy/lib/galaxy/tools/deps/__init__.py ", line 241, in __build_dependency_resolvers plugin_source = plugin_config.plugin_source_from_path(conf_file) File "/opt/galaxy/galaxy/lib/galaxy/util/plugin_config.py ", line 78, in plugin_source_from_path return ('xml', ElementTree.parse(path).getroot()) File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1182, in parse tree.parse(source, parser) File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 656, in parse parser.feed(data) File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1642, in feed self._raiseerror(v) File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1506, in _raiseerror raise err xml.etree.ElementTree.ParseError: junk after document element: line 54, column 0 Cheers Jochen

3 3

May Galaxy News: GCC2019 early reg, GTN CoFest, new pubs, platforms, jobs,...
by Dave Clements 06 May '19

06 May '19

Hello all, The May 2019 Galactic News is here! See https://galaxyproject.org/news/2019-05-galaxy-update/ for details - Events - GCC2019: - **Early registration ends 17 May** - Poster and Demo Abstract Submission is open - Travel Fellowships: 2nd round ends this Friday - GigaScience returns as a GCC sponsor! - GTN CoFest and Community Call: 16 May - Plus 17 other upcoming events in the next 90 days - 265 new publications, great resources lead to great insight. - Some most excellent Galaxy Platform News, including a single cell 'omics workbench, network analysis, subcellular spatial organization modeling, and more! - A new entry to The Galactic Blog, Galaxy hearts GCP. - At least 11 Open positions in six countries on two continents. - Updates to training materials. - New releases. See you in two months in Freiburg, Dave C, Mo, and the Galaxy Community -- https://galaxyproject.org/gcc2019/ https://galaxyproject.org/ https://usegalaxy.org/

1 0