April 2015 - galaxy-dev - lists.galaxyproject.org

Announcement: Galaxy Docker Image 15.03
by Björn Grüning 25 Apr '15

25 Apr '15

Hi all! We are proud to present you the next release of Galaxy Docker. https://github.com/bgruening/docker-galaxy-stable https://registry.hub.docker.com/u/bgruening/galaxy-stable/ A few highlights: * based on Galaxy 15.03, with all it's new awesome features * Docker image is now based on Ubuntu 14.04 * move all required Galaxy dependencies to toolshed/requirements * use Ansible roles to build large parts of the image * export the supervisord web-interface (port 9002) * enable Galaxy reports webapp on GALAXY_URL/reports * enable/disable services like FTP or Galaxy reports during start-up The IPython integration was also updated and has a few new libraries to make it more convenient for R users. A big thanks to John Chilton for his enormous support and constant input. The following new releases are available: * docker pull bgruening/galaxy-stable:latest_2015.01.13 * docker pull bgruening/galaxy-stable:15.03 * docker pull bgruening/galaxy-stable (-> currently 15.03) As usual feedback is more than welcome! Cheers, Bjoern

2 4

InvalidRequestError: kombu.transport.sqlalchemy.Queue
by Asma Riyaz 24 Apr '15

24 Apr '15

Hi, I am receiving this traceback when I do restart Apache and not sure what to make of it: Exception in thread WorkflowRequestMonitor.monitor_thread: Traceback (most recent call last): File "/broad/software/free/Linux/redhat_6_x86_64/pkgs/python_2.7.1-sqlite3-rtrees/lib/python2.7/threading.py", line 530, in __bootstrap_inner self.run() File "/broad/software/free/Linux/redhat_6_x86_64/pkgs/python_2.7.1-sqlite3-rtrees/lib/python2.7/threading.py", line 483, in run self.__target(*self.__args, **self.__kwargs) File "/seq/regev_genome_portal/SOFTWARE/galaxy/lib/galaxy/workflow/scheduling_manager.py", line 158, in __monitor self.__schedule( workflow_scheduler_id, workflow_scheduler ) File "/seq/regev_genome_portal/SOFTWARE/galaxy/lib/galaxy/workflow/scheduling_manager.py", line 162, in __schedule invocation_ids = self.__active_invocation_ids( workflow_scheduler_id ) File "/seq/regev_genome_portal/SOFTWARE/galaxy/lib/galaxy/workflow/scheduling_manager.py", line 191, in __active_invocation_ids handler=handler, File "/seq/regev_genome_portal/SOFTWARE/galaxy/lib/galaxy/model/__init__.py", line 3254, in poll_active_workflow_ids WorkflowInvocation File "/seq/regev_genome_portal/SOFTWARE/galaxy/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/scoping.py", line 150, in do return getattr(self.registry(), name)(*args, **kwargs) File "/seq/regev_genome_portal/SOFTWARE/galaxy/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/session.py", line 1165, in query return self._query_cls(entities, self, **kwargs) File "/seq/regev_genome_portal/SOFTWARE/galaxy/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/query.py", line 108, in __init__ self._set_entities(entities) File "/seq/regev_genome_portal/SOFTWARE/galaxy/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/query.py", line 118, in _set_entities self._set_entity_selectables(self._entities) File "/seq/regev_genome_portal/SOFTWARE/galaxy/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/query.py", line 151, in _set_entity_selectables ent.setup_entity(*d[entity]) File "/seq/regev_genome_portal/SOFTWARE/galaxy/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/query.py", line 3036, in setup_entity self._with_polymorphic = ext_info.with_polymorphic_mappers File "/seq/regev_genome_portal/SOFTWARE/galaxy/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/util/langhelpers.py", line 725, in __get__ obj.__dict__[self.__name__] = result = self.fget(obj) File "/seq/regev_genome_portal/SOFTWARE/galaxy/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/mapper.py", line 1877, in _with_polymorphic_mappers configure_mappers() File "/seq/regev_genome_portal/SOFTWARE/galaxy/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/mapper.py", line 2589, in configure_mappers mapper._post_configure_properties() File "/seq/regev_genome_portal/SOFTWARE/galaxy/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/mapper.py", line 1694, in _post_configure_properties prop.init() File "/seq/regev_genome_portal/SOFTWARE/galaxy/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/interfaces.py", line 144, in init self.do_init() File "/seq/regev_genome_portal/SOFTWARE/galaxy/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/relationships.py", line 1549, in do_init self._process_dependent_arguments() File "/seq/regev_genome_portal/SOFTWARE/galaxy/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/relationships.py", line 1605, in _process_dependent_arguments self.target = self.mapper.mapped_table File "/seq/regev_genome_portal/SOFTWARE/galaxy/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/util/langhelpers.py", line 725, in __get__ obj.__dict__[self.__name__] = result = self.fget(obj) File "/seq/regev_genome_portal/SOFTWARE/galaxy/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/relationships.py", line 1522, in mapper argument = self.argument() File "/seq/regev_genome_portal/SOFTWARE/galaxy/eggs/SQLAlchemy-0.9.8-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/ext/declarative/clsregistry.py", line 283, in __call__ (self.prop.parent, self.arg, n.args[0], self.cls) InvalidRequestError: When initializing mapper Mapper|Queue|kombu_queue, expression 'Message' failed to locate a name ("name 'Message' is not defined"). If this is a class name, consider adding this relationship() to the <class 'kombu.transport.sqlalchemy.Queue'> class after both dependent classes have been defined. This happens sometimes but when I shut down galaxy and re-start I don't get it anymore. Can someone guide me to a more permanent solution to this? Asma

2 1

README.rst missing on Tool Shed display?
by Peter Cock 24 Apr '15

24 Apr '15

Hi all, I don't see tool README files on the main view of a repository any more. Is this a recent regression? e.g. https://toolshed.g2.bx.psu.edu/view/bgruening/deeptools and https://testtoolshed.g2.bx.psu.edu/view/bgruening/deeptools should show the readme.rst file e.g. https://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus and https://testtoolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus should show the README.rst file This might be specific to restructured-text markup - I'd have to trawl the ToolShed looking for a plain text README file to check that. Regards, Peter

1 0

History panel error
by Jose Juan Almagro Armenteros 24 Apr '15

24 Apr '15

Hello all, I have updated my Galaxy distribution today to the latest version 15.03 and the history panel is not being shown. The history seems to be there but Galaxy can't display the datasets. Even if I create a new history and I run a tool the dataset is not shown. It is just displayed an error which says: "An error occurred while getting updates from the server..." and the details shows this: { "agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36", "url": "http://galaxy.bric.ku.dk/history/view_multipl/api/histories/782c6cc7ec4cfd9…", "data": "", "options": { "data": {}, "silent": true, "parse": true, "emulateHTTP": false, "emulateJSON": false }, "xhr": { "readyState": 4, "responseText": "{\"err_msg\": \"Uncaught exception in exposed API method:\", \"err_code\": 0}", "responseJSON": { "err_msg": "Uncaught exception in exposed API method:", "err_code": 0 }, "status": 500, "statusText": "Internal Server Error", "responseHeaders": { "Date": "Fri, 24 Apr 2015 12:09:40 GMT\r", "cache-control": "max-age=0,no-cache,no-store\r", "Transfer-Encoding": "chunked\r", "Server": "PasteWSGIServer/0.5 Python/2.7.3\r", "Connection": "close\r", "x-frame-options": "SAMEORIGIN\r", "content-type": "application/json\r" } }, "source": [], "user": { "username": "jose", "quota_percent": 0, "total_disk_usage": 101352147, "nice_total_disk_usage": "96.7 MB", "email": "jjalmagro(a)gmail.com", "is_admin": true, "tags_used": [], "model_class": "User", "id": "72ad249754f05d26" } } Do you have any idea what could go wrong? Thank you and best regards, Jose

2 2

question about splitting bams
by Roberto Alonso CIPF 24 Apr '15

24 Apr '15

Hello, I ma trying ti write some code in order to give the possibility of parallelize some tasks. Now, I was with the problem of splitting a bam in some parts, for this I create this simple tool <parallelism method="multi" split_size="3" split_mode="number_of_parts" merge_outputs="output" split_inputs="input" ></parallelism> <command> java -jar /home/ralonso/software/GenomeAnalysisTK-3.3-0/GenomeAnalysisTK.jar -T UnifiedGenotyper -R /home/ralonso/BiB/Galaxy/data/chr_19_hg19_ucsc.fa -I $input -o $output 2> /dev/null; </command> <inputs> <param format="bam" name="input" type="data" label="bam"/> </inputs> <outputs> <data format="vcf" name="output" /> </outputs> But I have one problem, when I execute the tool it goes through this part of code (I am working in dev branch): *$galaxy/lib/galaxy/jobs/splitters/multi.py, line 75:* for input in parent_job.input_datasets: if input.name in split_inputs: this_input_files = job_wrapper.get_input_dataset_fnames(input.dataset) if len(this_input_files) > 1: log_error = "The input '%s' is composed of multiple files - splitting is not allowed" % str(input.name) log.error(log_error) raise Exception(log_error) input_datasets.append(input.dataset) So, it is raising the exception because this_input_files=2, concretely: ['/home/ralonso/galaxy/database/files/000/dataset_171.dat', '/home/ralonso/galaxy/database/files/_metadata_files/000/metadata_13.dat'], I guess that: *dataset_171.dat*: It is the bam file. *metadata_13.dat*: It is the bai file. So, Galaxy can't move on and I don't know which would be the best solution. Maybe change the *if* to check only non-metadata files? I think I should use both files in order to create the bam sub-files, but this would be inside the Bam class, under *binary.py* file. Could you please guide me before I mess things up? Thanks so much -- Roberto Alonso Functional Genomics Unit Bioinformatics and Genomics Department Prince Felipe Research Center (CIPF) C./Eduardo Primo Yúfera (Científic), nº 3 (junto Oceanografico) 46012 Valencia, Spain Tel: +34 963289680 Ext. 1021 Fax: +34 963289574 E-Mail: ralonso(a)cipf.es

2 3

Shared Data Library issues with permissions
by Ganote, Carrie L 23 Apr '15

23 Apr '15

Hi Devs, Maybe I'm missing something, but I'm having trouble getting permissions right on the Data Libraries. I can reproduce this: As an admin, I go to Manage data libraries. On top right, I click Create new data library. Let's call it test. I Add a folder in test, called "more testing". I check the permissions of the library in Library Actions->Edit permissions and make sure that all roles associated are blank. I check permissions in more testing to make sure they are all blank. I click on Shared Data -> Data Libraries to see if I can view the files with the user perspective. I click on the Test library I just created: Data Library “Test” The data library 'Test' does not contain any datasets that you can access. I have a fairly recent (April 2015) update to Galaxy. Any advice? I first started having trouble getting access for a user to their own library; when I associated their role with their own folder, they still couldn't see anything they could access. I should at least get an empty folder. Sincerely, Carrie Ganote

2 1

Serving a galaxy instance over Apache
by Asma Riyaz 23 Apr '15

23 Apr '15

Hi, I have been through previous issues regarding this issue. So far, I have included the rewrite Engine logic written in /etc/apache2/sites-available/defaults - looks like so: <VirtualHost *:80> ..... ..... RewriteEngine on RewriteRule ^/galaxy$ /galaxy/ [R] RewriteRule ^/galaxy/static/style/(.*) /seq/SOFTWARE/galaxy/static/june_2007_style/blue/$1 [L] RewriteRule ^/galaxy/static/scripts/(.*) /seq/SOFTWARE/galaxy/static/scripts/packed/$1 [L] RewriteRule ^/galaxy/static/(.*) /seq/SOFTWARE/galaxy/static/$1 [L] RewriteRule ^/galaxy/favicon.ico /seq/SOFTWARE/galaxy/static/favicon.ico [L] RewriteRule ^/galaxy/robots.txt /seq/SOFTWARE/galaxy/static/robots.txt [L] RewriteRule ^/galaxy(.*) http://[my IP address]/$1 [P] I restarted apache after this and accessed : http://[my IP address]/galaxy but no webpage is found. Any advice on how I can proceed to configure the apache correctly? Thank you, Asma

2 2

Re: [galaxy-dev] Galaxy on HPC and Bright Cluster Manager?
by David Trudgian 23 Apr '15

23 Apr '15

Hi Carlos, sorry for the slow reply. With a small 10 user setup (I assume just one group) your installation is probably going to be a lot less complex than ours. We are running services for users from multiple labs and departments, so a chief concern was making sure private datasets always stay private - which is relatively involved when using Galaxy to run jobs on a general-purpose shared cluster. I think if I had to recommend picking a job scheduler I would suggest SLURM since it's probably the most 'fashionable' choice at present. You also have the advantage that SLURM is used with Galaxy in the Galaxy docker image etc. which ensures people notice if the Galaxy->DRMAA->SLURM setup isn't working. I've also used Galaxy with GridEngine in the past, and that was fine - but is becoming a less common choice as a scheduler. Having said that, I don't think that the job scheduler needs to be your biggest concern. I would focus most on the file system and user account setup you are going to need. * How are you going to migrate from standalone Galaxy to a situation where your new cluster can see the Galaxy data files, tools etc. Are you purchasing storage with the cluster? If so do you move Galaxy onto that storage, or can you mount existing Galaxy data onto the cluster nodes? If you can do that, is your networking such that performance is sufficient for the type of analysis you are going to run? * Do you need to, or will you need to, keep track of per-user usage of the cluster for things that Galaxy will be running? If not then you can just have a galaxy user on your cluster and things are pretty easy for file permissions etc. If you need to track jobs per-user then it becomes more complex, and the solution depends on how much privacy you need for datasets, how your cluster will authenticate users etc. The filesystem and user accounts issues are, in my mind, the ones to focus on. You can always modify Galaxy's config to switch to a different job scheduler fairly easily. You cannot as easily move around large amounts of data, and reconcile local vs cluster user accounts, should that be necessary. Cheers, Dave Trudgian -----Original Message----- From: Carlos Lijeron [mailto:clijeron@hunter.cuny.edu] Sent: Wednesday, April 22, 2015 10:54 AM To: David Trudgian; John Chilton Cc: RODRIGO GONZALEZ SERRANO Subject: Re: [galaxy-dev] Galaxy on HPC and Bright Cluster Manager? Hello David, Thank you for the great feedback. We are at Hunter College in NYC, part of the City University of New York. We recently ordered the cluster which comes with Bright Cluster Management, and our PI wants to implement Galaxy for all the users (about 10) on the cluster and manage all job submissions through a job scheduler. So, to answer your question, we are not really using any scheduler at this point, but only a stand alone server with a local installation of Galaxy. Our Cluster should be assembled and installed by the end of May, so I¹m trying to gather as much information as possible in preparation for the deployment. Based on your experience, what do you think I should focus on to ensure we maximize outcome and reduce the possibility of mistakes? In other words, any lessons learned that you would like to share will be greatly appreciated. Thanks again, Carlos Lijeron. On 4/22/15, 10:34 AM, "David Trudgian" <David.Trudgian(a)UTSouthwestern.edu> wrote: >Carlo, > >We have Bright Cluster Manager in use on our cluster for node >provisioning etc. but the actual job scheduler in use in our case is >SLURM, which we use directly. > >Are you using one of the integrated workload managers such as SLURM / >SGE / TORQUE directly, or indirectly via cmsub? > >I guess the easiest way to come up with some kind of advice is if you >can provide an example of generic job script you are using on your system. >If you're using cmsub is it specifying a --wlmanager etc. > >DT > >-----Original Message----- >From: galaxy-dev [mailto:galaxy-dev-bounces@lists.galaxyproject.org] On >Behalf Of John Chilton >Sent: Wednesday, April 22, 2015 8:26 AM >To: Carlos Lijeron >Cc: galaxy-dev(a)lists.galaxyproject.org >Subject: Re: [galaxy-dev] Galaxy on HPC and Bright Cluster Manager? > >Hello Carlos, > > I have never heard of anyone running Galaxy with Bright Cluster >Manager (though hopefully someone will chime in if they have). If you >are interested in adding support it should be possible. One >complication is that Bright Cluster Manager doesn't appear to have a >DRMAA interface >(http://www.drmaa.org/) which is the most direct way to utilize new DRMs. >Without that my approach would be to build a new CLI runner: > >There are a few examples here that one can use as template: > >https://github.com/galaxyproject/galaxy/tree/dev/lib/galaxy/jobs/runner >s/u >til/cli/job > >I guess you would have to write a new one targeting cmsub I guess - you >also need to be able to parse a job status somehow - I haven't figured >out how to do that from the documentation - but I assume there is a way. > >I looks like Bright supports running SGE, SLURM, and Torque on the >cluster - doing this and interfacing with one of those more common >options directly might be a better approach for Galaxy (and other users >if your cluster has them). > >-John > > > > > > > >On Wed, Apr 22, 2015 at 8:56 AM, Carlos Lijeron ><clijeron(a)hunter.cuny.edu> wrote: >> Good day everyone, >> >> >> >> Has anyone of you been able to implement Galaxy on a HPC using Bright >> Cluster Manager as the main DRM? I noticed that only a few have been >> known to work with Galaxy, but the list does not include Bright. Any >> advice/ideas will be greatly appreciated. >> >> >> >> TORQUE Resource Manager >> >> PBS Professional >> >> Open Grid Engine >> >> Univa Grid Engine (previously known as Sun Grid Engine and Oracle >> Grid >> Engine) >> >> Platform LSF >> >> HTCondor >> >> Slurm >> >> Galaxy Pulsar (formerly LWR) >> >> >> >> >> >> Thanks. >> >> >> >> >> >> Carlo >> >> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this and other >> Galaxy lists, please use the interface at: >> https://lists.galaxyproject.org/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ >___________________________________________________________ >Please keep all replies on the list by using "reply all" >in your mail client. To manage your subscriptions to this and other >Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > >To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ > >________________________________ > >UT Southwestern > > >Medical Center > > > >The future of medicine, today. >

1 0

Writing Auth Layer as WSGI Middleware
by Stephen Rosen 23 Apr '15

23 Apr '15

Hi All, I have a need for authentication as a layer in front of Galaxy which is more specialized than the available options -- specifically 3-legged OAuth against site of my choice. After looking into writing this in PHP and having the webserver (nginx) set remote_user, I decided to nix that approach for a couple of reasons -- one of which is that I don't have PHP experience. After a few discussions with other devs, I've decided that there are two easy options available to me: - Write a WSGI app which does authentication, and *proxies* authenticated requests to Galaxy with remote_user set. Since that's a WSGI app doing proxying, obvious code smell there - Write a WSGI middleware that wraps the existing Galaxy WSGI app, and passes authenticated requests directly to the Galaxy app That second solution seems much better, but I'm now faced with the question of "How do I do it?" Looking over the sample config, I see these lines: # The factory for the WSGI application. This should not be changed. paste.app_factory = galaxy.web.buildapp:app_factory I'm thinking that I could change that to my middleware, which will turn to `galaxy.web.buildapp` when the time comes. One problem I'm seeing is that my middleware and galaxy both have to run in the same virtualenv, so there's potential for dependency conflicts. The lib I want to use for this does rely on PyYAML and a few other things which Galaxy also needs, so that possibility is very real. Other than that hurdle, are there any gotchas I should be aware of with this approach? Are there similarly simple alternatives to this which I am not seeing? Ultimately, if I have to write an app that does proxying, I'd prefer that to the wide variety of highly effortful solutions I have envisioned. Those include, but are not limited to, a PAM which does the OAuth and doing Basic Authentication against that, just to give a flavor. Thanks very much for your help, -Stephen

1 0

Galaxy Tool Shed security vulnerability
by Nate Coraor 23 Apr '15

23 Apr '15

*Please note: This notice affects Galaxy Tool Shed servers only. Galaxy servers are unaffected.* A security vulnerability was recently discovered by Daniel Blankenberg of the Galaxy Team that would allow a malicious person to execute arbitrary code on a Galaxy Tool Shed server. The vulnerability is due to reuse of tool loading code from Galaxy, which executes "code files" defined by Galaxy tool config files. Because the Tool Shed allows any user to create and "load" tools, any user could cause arbitrary code to be executed by the Tool Shed server. In Galaxy, administrators control which tools are loaded, which is why this vulnerability does not affect Galaxy itself. Although we recommend upgrading to the latest stable version (15.03.2), a fix for this issue has been committed to Galaxy versions from 14.08 and newer. If you are using Mercurial, you can update with (where YY.MM corresponds to the Galaxy release you are currently running): % hg pull % hg update release_YY.MM If you are using git, you can update with (assuming your remote upstream is set to https://github.com/galaxyproject/galaxy/) If you have not yet set up a remote tracking branch for the release you are using: % git fetch upstream % git checkout -b release_YY.MM upstream/release_YY.MM Otherwise: % git pull upstream release_YY.MM For the changes to take effect, *you must restart all Tool Shed server processes*. Credit for the arbitrary code execution fix also goes to my fellow Galaxy Team member Daniel Blankenberg. On behalf of the Galaxy Team, --nate

1 0