- galaxy-dev - lists.galaxyproject.org

Deploy on cloud
by Stavros 0 Sachtouris 03 Mar '17

03 Mar '17

Hi all, we are going to deploy galaxy on ~okeanos cloud ( okeanos.grnet.gr ). We have already developed a working storage backend for our object store system ( https://github.com/galaxyproject/galaxy/pull/3611 ). Now we focus on how to make Galaxy run instances/tasks on our VMs and I would like some advice on how to approach this issue. What we need is a (potentially large) Galaxy deployment with the ability to run workflows of dockerized "tasks". Each task should run on a provisioned VM on our cloud. There are many tools out there to let us automate VM provisioning, managing docker clusters, scaling, as well as monitoring and accounting, and we want to provide all of these features to our partners in European Academia. The question is, how does Galaxy communicate with a cluster manager like Apache Mesos or Docker swarm? Any other tools I should look into, which are known to work with Galaxy? Will we have to drill into Galaxy code base to provide these features? Thank you in advance for you suggestions, Stavros Sachtouris developer @ www.grnet.gr

2 1

Problem with installing tools using Conda
by Ashok Varadharajan 03 Mar '17

03 Mar '17

Dear Team, I have problem with the installation of tools using conda. I am getting following error message when i try to install tools from galaxy. I tried to install these tools manually with the same conda used by the galaxy and it worked. Can anyone help us to solve this issue ? Traceback (most recent call last): File "/xxxx/xxxxx/xxxx-xxxx/conda//bin/conda", line 4, in <module> from conda.cli.main import main ImportError: No module named conda.cli.main galaxy.tools.deps.resolvers.conda DEBUG 2017-03-02 17:35:49,759 Removing failed conda install of samtools, version '1.3.1' Traceback (most recent call last): File "/xxxx/xxxxx/xxxx-xxxx/conda//bin/conda", line 4, in <module> from conda.cli.main import main ImportError: No module named conda.cli.main galaxy.tools.deps.resolvers.conda DEBUG 2017-03-02 17:35:49,941 Removing failed conda install of samtools, version 'None' Traceback (most recent call last): File "/xxxx/xxxxx/xxxx-xxxx/conda//bin/conda", line 4, in <module> from conda.cli.main import main ImportError: No module named conda.cli.main galaxy.tools.deps.resolvers.conda DEBUG 2017-03-02 17:35:50,142 Removing failed conda install of bwa, version '0.7.15' Traceback (most recent call last): File "/xxxx/xxxxx/xxxx-xxxx/conda//bin/conda", line 4, in <module> from conda.cli.main import main ImportError: No module named conda.cli.main We are running galaxy-16.07 behind the proxy using nginx, uwsgi and supervisor in scientific linux 7. Below are the details of galaxy.ini and supervisor configuration. *### galaxy.ini * conda_prefix = /xxxx/xxxxx/xxxx-xxxx/conda/ #conda_exec = conda_debug = False conda_ensure_channels = r,bioconda,iuc #conda_auto_install = False #conda_verbose_install_check=False conda_auto_init = True *### supervisor* [supervisord] [program:galaxy_web] command = $GALAXY_PATH/.venv/bin/uwsgi --virtualenv $GALAXY_PATH/.venv --ini-paste $GALAXY_PATH/config/galaxy.ini --logdate --master --processes 3 --threads 6 --logto $GALAXY_PATH/logs/galaxy.uwsgi.log --socket 127.0.0.1:4001 --pythonpath lib --stats 127.0.0.1:9191 directory = $GALAXY_PATH umask = 022 autostart = true autorestart = true startsecs = 20 user = galaxy environment = PATH=$GALAXY_PATH/.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin,PYTHONHOME=$GALAXY_PATH/.venv,DRMAA_LIBRARY_PATH=/usr/local/lib/libdrmaa.so numprocs = 1 stoplight = INT startretries = 15 process_name = web%(process_num)s [program:handler] command = $GALAXY_PATH/.venv/bin/python ./lib/galaxy/main.py -c $GALAXY_PATH/config/galaxy.ini --server-name=handler%(process_num)s --log-file=$GALAXY_PATH/logs/ >galaxy_handler%(process_num)s.log directory = $GALAXY_PATH process_name = handler%(process_num)s numprocs = 3 umask = 022 autostart = true autorestart = true startsecs = 20 user = galaxy environment = PATH=$GALAXY_PATH/.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin,PYTHONHOME=$GALAXY_PATH/.venv,DRMAA_LIBRARY_PATH=/usr/local/lib/libdrmaa.so startretries = 15 -- *Thanks and Regards,* *Ashok Vardharajan*

4 6

2017 Galaxy Community Conference: Registration & Abstract Submission are open
by Dave Clements 03 Mar '17

03 Mar '17

We are pleased to announce that early registration <https://gcc2017.sciencesconf.org/page/registration> and abstract submission <https://gcc2017.sciencesconf.org/page/submit> for the 2017 Galaxy Community Conference (GCC2017) <https://gcc2017.sciencesconf.org/> are now open. GCC2017 will be in Montpellier, France, 26-30 June and will feature two days of presentations, discussions, poster sessions, lightning talks, computer demos, keynotes, and birds-of-a-feather meetups, all about data-intensive biology and the tools that support it. GCC2017 also includes data and coding hackathons <https://gcc2017.sciencesconf.org/page/hackathon>, and two days of training <https://gcc2017.sched.com/overview/type/Training> covering 16 different topics. GCC2017 will be held at Le Corum Conference Centre <http://www.montpellier-events.com/en/The-Corum/Presentation> in the heart of Montpellier <https://gcc2017.sciencesconf.org/page/location>, just 10km from the Mediterranean. This event will gather several hundred researchers addressing diverse questions and facing common challenges in data intensive life science research. GCC participants work across the tree of life, come from around the world, and work at universities, research organizations, industry, medical schools and research hospitals. If you work in or support data intensive life science research then GCC2017 is an ideal opportunity to present your work. Abstract submission <https://gcc2017.sciencesconf.org/page/submit> is now open. The deadline for oral presentations is Apr 15 (23:59 Paris local time), and posters and computer demonstration submission closes on May 27 (23:59 Paris local time). The Lightning talk submission deadline is June 23 (23:59 Paris time). Early registration <https://gcc2017.sciencesconf.org/page/registration> is also open and starts at less than 55€ / day for post-docs and students. You can also book low cost conference housing <https://gcc2017.sciencesconf.org/page/lodging> when you register. (Travel fellowships will be offered by the Galaxy Community Fund and will be announced shortly.) About Galaxy Galaxy <https://galaxyproject.org/> is an open, web-based platform for data-intensive biomedical analysis used by tens of thousands of researchers around the world. It supports ad hoc exploration and analysis through scalable and repeatable data analysis pipelines for large research studies. Galaxy is available in over 90 free and publicly accessible web servers, on commercial and national cloud infrastructures, and is locally installed at hundreds, if not thousands, of research organisations around the world. We hope to see you this summer in Montpellier! Au revoir, The GCC2017 Organising Committee PS: Please redistribute this email to any interested communities. -- http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ https://wiki.galaxyproject.org/

1 0

Re: [galaxy-dev] Shared Data Library, large number of files
by Hans-Rudolf Hotz 02 Mar '17

02 Mar '17

This is bizarre...our production server is on 16.04 and the old UI is no longer installed. On 03/01/2017 06:03 PM, D K wrote: > Hi Hans, > > I'm running 16.10. If I'm not mistaken, the old Data Library interface > is still accessible in the Admin menu isn't it? I have the same issue > when navigating to a Data Library either from the Admin or Shared Data > links. I also tried this in 16.07 with the same result. If it makes a > difference, I'm using PostgreSQL and Apache. > > Dwight > > > > On Tue, Feb 28, 2017 at 11:47 PM, Hans-Rudolf Hotz <hrh(a)fmi.ch > <mailto:hrh@fmi.ch>> wrote: > > Hi Daniel > > Just double checking which version of galaxy are you running? In > other words: are you still using the old Data Library interface? > (if I remember correctly, the old UI was deprecated in 15.10) > We had similar issues with the old UI. > > The new one works fine and is very fast. E.g. we have a 'MISEQ' > folder with more than 100 sub folders and a HISEQ folder with more > than 600 sub folders. Each containing up to 100plus fastq and text > files. > > Regards, Hans-Rudolf > > > > On 03/01/2017 12:13 AM, D K wrote: > > Hi, > > I've noticed that the more files / folders (think hundreds of > folders > each with 20+ fastq files) I add to a Shared Data Library, the > longer > and longer it takes to initially navigate the folder structure > (after > clicking on the specific Shared Data Library). Just opening up the > Shared Data Library can take several minutes and often the > browser times > out. Is there a way to speed this up? > > If it's relevant, when I add shared data, it's linked rather > than copied > into Galaxy. Any suggestions would be greatly appreciated > > Thanks! > > > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ <https://lists.galaxyproject.org/> > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ > <http://galaxyproject.org/search/mailinglists/> > >

1 0

ansible-galaxy-tools and venv
by Matthias Bernt 01 Mar '17

01 Mar '17

Dear galaxy developers, Is it expected that the virtualenv in .venv/ does include the python binary but does not include the python shared libraries? Some more details: I have installed galaxy on CentOS 6.8 which includes the required python 2.7 only via software collections. For the installation of galaxy I have enabled the software collection by calling "source /opt/rh/python27/enable" (which sets PATH, LD_LIBRARY_PATH, MANPATH, XDG_DATA_DIRS, and PKG_CONFIG_PATH). Currently I was trying to run the ansible role ansible-galaxy-tools from https://github.com/galaxyproject/ansible-galaxy-tools which runs "/home/galaxy/galaxy/.venv/bin/python manage_bootstrap_user.py ..." and gives me /home/galaxy/galaxy/.venv/bin/python: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory. This python call would work if I again "source /opt/rh/python27/enable". But then the venv python binary is used with the library of the system. Intuitively I would expect that the library and binary belong together. Cheers, Matthias -- ------------------------------------------- Matthias Bernt Bioinformatics Service Molekulare Systembiologie (MOLSYB) Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 482296, m.bernt(a)ufz.de, www.ufz.de Sitz der Gesellschaft/Registered Office: Leipzig Registergericht/Registration Office: Amtsgericht Leipzig Handelsregister Nr./Trade Register Nr.: B 4703 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig Wilfried Kraus Wissenschaftlicher Geschäftsführer/Scientific Managing Director: Prof. Dr. Dr. h.c. Georg Teutsch Administrative Geschäftsführerin/ Administrative Managing Director: Prof. Dr. Heike Graßmann -------------------------------------------

2 2

Fatal Error while uploading a File with special Characters
by SAPET, Frederic 01 Mar '17

01 Mar '17

Hello Like Matthias (http://gmod.827538.n3.nabble.com/Fatal-Error-while-uploading-a-File-with-sp…) I can't upload a file with special characters. My test file is : Frédéric.txt The database itself seems to be ok, I can see that the insertion is ok in "history_dataset_association" : id history_id dataset_id create_time update_time copied_from_history_dataset_association_id hid name info 328 1 330 2017-03-01 08:13:56 2017-03-01 08:14:05 NULL 293 Frédéric.txt uploaded txt file 1 line But until I change Frédéric.txt to Frederic.txt within the name column, my history is broken. Errors are the same as described by Matthias. galaxy.web.framework.decorators ERROR 2017-03-01 09:14:18,321 Uncaught exception in exposed API method: Traceback (most recent call last): File "/softs/bioinfo/galaxy-prod/lib/galaxy/web/framework/decorators.py", line 284, in decorator rval = _format_return_as_json( rval, jsonp_callback, pretty=trans.debug ) File "/softs/bioinfo/galaxy-prod/lib/galaxy/web/framework/decorators.py", line 317, in _format_return_as_json json = safe_dumps( rval, **dumps_kwargs ) File "/softs/bioinfo/galaxy-prod/lib/galaxy/util/json.py", line 67, in safe_dumps dumped = json.dumps( obj, allow_nan=False, **kwargs ) File "/usr/local/lib/python2.7/json/__init__.py", line 251, in dumps sort_keys=sort_keys, **kw).encode(obj) File "/usr/local/lib/python2.7/json/encoder.py", line 209, in encode chunks = list(chunks) File "/usr/local/lib/python2.7/json/encoder.py", line 431, in _iterencode for chunk in _iterencode_list(o, _current_indent_level): File "/usr/local/lib/python2.7/json/encoder.py", line 332, in _iterencode_list for chunk in chunks: File "/usr/local/lib/python2.7/json/encoder.py", line 390, in _iterencode_dict yield _encoder(value) UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 2: invalid continuation byte I have read a lot of threads about this kind of error (mostly from stackoverflow) It is related to Galaxy or it's related to my python installation ? For a production server, it means that I will have to update manually the database each time a user try to upload such a file (find his id, find the broken history and the file name to correct !) I'm running Galaxy 17.01 with a mysql (5.7.11-4, UTF-8) database. I've modified lib/galaxy/datatypes/binary.py (for XLS datatypes) and lib/galaxy/jobs/runners/__init__.py (truncate the job name for PBS-PRO) # On branch release_17.01 # Your branch is ahead of 'origin/release_17.01' by 3 commits. # # Changed but not updated: # (use "git add <file>..." to update what will be committed) # (use "git checkout -- <file>..." to discard changes in working directory) # # modified: lib/galaxy/datatypes/binary.py # modified: lib/galaxy/jobs/runners/__init__.py Thanks in advance! Fred --- Frederic Sapet Bioinformatics Research Engineer BIOGEMMA - Upstream Genomics Group Centre de Recherche de Chappes CS 90126 63720 CHAPPES FRANCE Tel : +33 (0)4 73 67 88 54 Fax : +33 (0)4 73 67 88 99 E-mail : frederic.sapet(a)biogemma.com<mailto:frederic.sapet@biogemma.com>

1 0

Shared Data Library, large number of files
by D K 01 Mar '17

01 Mar '17

Hi, I've noticed that the more files / folders (think hundreds of folders each with 20+ fastq files) I add to a Shared Data Library, the longer and longer it takes to initially navigate the folder structure (after clicking on the specific Shared Data Library). Just opening up the Shared Data Library can take several minutes and often the browser times out. Is there a way to speed this up? If it's relevant, when I add shared data, it's linked rather than copied into Galaxy. Any suggestions would be greatly appreciated Thanks!

2 1

Galaxy and Torque
by Yip, Miu ki 28 Feb '17

28 Feb '17

Hi all, Is Torque still supported as a resource manager? This page ( https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster ) mentioned the usage of Torque and how to set it up. However, the setup instructions refer to a /script/scramble.py file which I assume interfaces with eggs. If I recall correctly, eggs were upgraded to wheels. Does this mean there are upgraded scripts in place of scramble.py and the eggs.ini? I saw this page ( https://biostar.usegalaxy.org/p/20470/ ) and it looks like nobody has a solution to this. Thanks!

2 1

Re: [galaxy-dev] Execute command once job has completed
by Peter Cock 28 Feb '17

28 Feb '17

I do not believe that is possible (nor desirable in general, not everyone will need or want the converted file and the overhead that comes with creating and storing it). What do you want to do with the bigwig file, and is the automatic conversion working for you? i.e. If you have tool X in Galaxy which wants an bigwig input file, does Galaxy offer an automated conversion of your bedgraph file as an input in the drop down file list? Peter On Tue, Feb 28, 2017 at 5:27 PM, evan clark <eclark28(a)fau.edu> wrote: > By automatic I mean that once any bedgraph file is finished writing it will > automatically run the convert to bigwig command > > > Peter Cock > February 28, 2017 12:21 PM > It depends on what you mean by automatic. You could > create a Galaxy Workflow where a tool is run which > generates the bedgraph file, and then as the next > step it is converted to bigwig. > > For this specific task, it might be possible to exploit > the Galaxy datatype definition for bedgraph to allow > automatic conversion to bigwig - is this working for you? > > https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/datatypes/conve… > > Peter > evan clark > February 28, 2017 11:47 AM > Is there a method to setup a command to run depending on the filetype of a > job. Essentially I want to setup a task where everytime a bedgraph file is > generated it will either automatically start a new job and convert it to > bigwig or, within the same job file it will convert the bedgraph to bigwig. > >

1 0

Execute command once job has completed
by evan clark 28 Feb '17

28 Feb '17

Is there a method to setup a command to run depending on the filetype of a job. Essentially I want to setup a task where everytime a bedgraph file is generated it will either automatically start a new job and convert it to bigwig or, within the same job file it will convert the bedgraph to bigwig.

2 1