March 2013 - galaxy-dev - lists.galaxyproject.org

GalaxyAdmins meetup on March 20th at 10 AM Central
by IIHG Galaxy Administrator 04 Mar '13

04 Mar '13

Hello, The next GalaxyAdmins Meetup will be held on Wednesday, March 20, 2013 from 10 AM to 11 AM US Central time at https://globalcampus.uiowa.edu:443/join_meeting.html?meetingId=1262344508128. Hailiang (Leon) Mei and David van Enckevort will present on "NBIC Galaxy at SURFsara's HPC cloud". There will also be a Galaxy Project Update by a Galaxy Team member. We encourage and look forward to your participation in the meetup. Downloading the required Blackboard Collaborate web conferencing system applet in advance, and using a headphone with a microphone to prevent audio feedback during the call is recommended. Thank you. -- Srinivas Maddhi Senior Systems Administrator Iowa Institute for Human Genetics ITS Research Services, The University of Iowa

1 0

deleted datasets inconsistent
by Sarah Diehl 04 Mar '13

04 Mar '13

Hi all, it seems my clean-up enthusiasm caused some inconsistencies with datasets' deleted status. I encounter the following phenomenon: datasets appear totally fine in the data libraries, but once they are loaded into a history, they appear as deleted there. I dug through the database and found out that the following query returns all datasets in the libraries that were uploaded before my clean-up: SELECT * FROM library_dataset_dataset_association INNER JOIN library_dataset ON library_dataset_dataset_association.library_dataset_id = library_dataset.id WHERE library_dataset_dataset_association.deleted = TRUE AND library_dataset.deleted = FALSE; When I set both deleted to false, only datasets that were added after the clean-up appear. Do I break something, if I reset library_dataset_dataset_association.deleted to FALSE for the inconsistent datasets? I tried it for one dataset and that seems to have fixed the problem. Is this some bug in the clean-up scripts? I guess that actually they should have been removed completely. Best regards, Sarah

1 0

tool shed can't install via admin user
by shenwiyn 04 Mar '13

04 Mar '13

Hi, I want to install our local Galaxy for more than one users. When I want to install some tools from tool shed via admin user ,and I followed the instructions in the follow link: http://wiki.galaxyproject.org/InstallingRepositoriesToGalaxy#Installing_Gal… but these are some errors: Server Error URL: http://183.62.232.129:8081/galaxy/admin_toolshed/prepare_for_install?tool_s… Module paste.exceptions.errormiddleware:143 in __call__ >> app_iter = self.application(environ, start_response) Module paste.debug.prints:98 in __call__ >> environ, self.app) Module paste.wsgilib:539 in intercept_output >> app_iter = application(environ, replacement_start_response) Module paste.recursive:80 in __call__ >> return self.application(environ, start_response) Module paste.httpexceptions:632 in __call__ >> return self.application(environ, start_response) Module galaxy.web.framework.base:160 in __call__ >> body = method( trans, **kwargs ) Module galaxy.web.framework:208 in decorator >> return func( self, trans, *args, **kwargs ) Module galaxy.webapps.galaxy.controllers.admin_toolshed:1177 in prepare_for_install >> response = urllib2.urlopen( url ) Module urllib2:126 in urlopen >> return _opener.open(url, data, timeout) Module urllib2:391 in open >> response = self._open(req, data) Module urllib2:409 in _open >> '_open', req) Module urllib2:369 in _call_chain >> result = func(*args) Module urllib2:1172 in http_open >> return self.do_open(httplib.HTTPConnection, req) Module urllib2:1147 in do_open >> raise URLError(err) Could anyone tell me why ,and how to solve? Thanks shenwiyn

2 1

Re: [galaxy-dev] Implementing dataset collections
by Peter Cock 04 Mar '13

04 Mar '13

Hi all, I've retitled this from "Composite datatype output for Cuffdiff" to "Implementing dataset collections" to try and link into Jeremy's thread. References: http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-February/013574.html http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-March/013634.html http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-March/013636.html On Mon, Mar 4, 2013 at 4:08 AM, <Alex.Khassapov(a)csiro.au> wrote: >> Hi John, >> >> Are you saying that "composite multiple file dataset" isn't required >> and won't be implemented? >> >> We are using your implementation of multifiles dataset ("m:xxx" type) >> and hope that eventually it will be pushed into main Galaxy >> implementation. >> >> As we are using Galaxy for CT reconstruction tools, where input >> and output can consist of a couple thousand files, other options >> are not feasible, i.e. grouping datasets. >> >> -Alex On Mon, Mar 4, 2013 at 5:42 AM, John Chilton <chilton(a)msi.umn.edu> wrote: > Hi Alex, > > Thanks for the comments. The galaxy team has made it clear here and > to me privately that this will NOT be included in the Galaxy main code > base. I hope and am I confident that they will make grouping datasets > work, hopefully even to thousands of files. > > I do not believe the two ideas are mutually exclusive and I will be > maintaining a fork of galaxy-central with these additions, I will set > this up this week hopefully. I will do my best to respond to support > requests and make multiple file datasets and composite types in > general as robust as possible, keep up with Galaxy updates, etc.... > Obviously, it is risky to let a code base drift so far from galaxy > main's however and you, me, and others who might want to use them will > have to carefully weigh the risks when determining if multiple file > datasets are worth the headache. > > Thanks for all your help and inputs. I am sorry this did not turn > out differently, I feel I have really failed here. > > -John Hi John, Does your multiple file system work on composite datatypes? For example, given a 'blastdbn' file (several files making up a BLAST nucleotide database) could we use 'm:blastdbn' to refer to several BLAST databases? Is this at least possible in principle based on your work so far? How different is your m:xxx grouping of multiple files of a given datatype at odds with the dataset collections idea as outlined by Jeremy Goecks here? http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-February/013574.html https://trello.com/c/325AXIEr Jeremy's idea mentions the special case of a unique dataset collection object for paired-end reads - well if that was defined using the existing core Galaxy functionality as a 'pairedfastq' composite datatype (made of two FASTQ files), then could we use 'm:pairedfastq' (with your enhancement) for a whole bunch of paired FASTQ files. Nice? :) Regards, Peter

2 2

Resizing EBS on Amazon
by Scooter Willis 04 Mar '13

04 Mar '13

Used cloud man to create a new cluster on Feb 22 and picked 500GB as the initial size of the data drive. Working with TCGA exome DNA seq data didn't take long to fill that up. Used the cloud man admin interface to resize from 500GB to 1TB and the resize operation took 15 hours. Not sure if that is expected so wanted to give some heads up in case that is an area for optimization. Since I now have a local storage problem as I need to work with more than 1TB of data I tried to go the route of setting a S3 bucket using Fuse. Ran into a problem where the first s3fs software I tried to install had a version issue with Ubuntu 10. I remember something in a support email that better support for Amazon S3 was in the works. Can you provide any guidance or thoughts on how to work with more than 1TB of data using cost effective S3 versus expensive EBS? The same applies for storing results at S3. With s3fs the file system can hide many of the complexities of moving files back and forth with caching where working with 30GB+ files isn't going to be fun. Thanks Scooter

2 1

Error when upgrading the database to version 110
by Derrick Lin 04 Mar '13

04 Mar '13

Hi guys, I was given the following error when upgrading the database from 109 to 110: 0109_add_repository_dependency_tables DEBUG 2013-03-01 15:14:15,554 Creating repository_repository_dependency_association table failed: (OperationalError) (1059, "Identifier name 'ix_repository_repository_dependency_association_tool_shed_repository_id' is too long") u'CREATE INDEX ix_repository_repository_dependency_association_tool_shed_repository_id ON repository_repository_dependency_association (tool_shed_repository_id)' () 0109_add_repository_dependency_tables DEBUG 2013-03-01 15:14:15,554 Creating repository_repository_dependency_association table failed: (OperationalError) (1059, "Identifier name 'ix_repository_repository_dependency_association_tool_shed_repository_id' is too long") u'CREATE INDEX ix_repository_repository_dependency_association_tool_shed_repository_id ON repository_repository_dependency_association (tool_shed_repository_id)' () So I guess the indexes weren't created. I am using MySQL, is it a big deal for Galaxy? Cheers, Derrick

3 6

Bug in handling user job limits
by Lance Parsons 04 Mar '13

04 Mar '13

I submitted a trello "ticket" [(https://trello.com/c/6vxkqdjT) regarding this, but wanted to make sure I brought it to someone's attention (it's causing me some queue issues with my instance of galaxy). When registered_user_job_limit and anonymous_user_job_limit are set in universe.wsgi jobs cannot be run, instead the following error occurs: galaxy.jobs.handler ERROR 2012-12-04 12:44:51,869 failure running job 22396 Traceback (most recent call last): File "/data/local/galaxy/galaxy-prod/lib/galaxy/jobs/handler.py", line 183, in __monitor_step job_state = self.__check_if_ready_to_run( job ) File "/data/local/galaxy/galaxy-prod/lib/galaxy/jobs/handler.py", line 253, in __check_if_ready_to_run state = self.__check_user_jobs( job ) File "/data/local/galaxy/galaxy-prod/lib/galaxy/jobs/handler.py", line 274, in __check_user_jobs if not self.user_job_count: AttributeError: 'JobHandlerQueue' object has no attribute 'user_job_count' Commenting out the lines for registered_user_job_limit and anonymous_user_job_limit in universe.wsgi allows job to be queue one again. It looks like this is due to the fact that `self.__clear_user_job_count()` on line 159 of `handler.py` is only called when jobs are tracked in the database. If jobs are not tracked in the database (as in my case), the error occurs. Perhaps the fix would be to simply move the call outside the `if` block. It appears this was broken in the 2012-11-13 revision (73e05bc). -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University

3 5

Reloading *.loc files
by Peter Cock 04 Mar '13

04 Mar '13

Hello all, Re: BitBucket Issue 538 - Admin action to reload ".loc" files without restarting Galaxy https://trello.com/card/538-admin-action-to-reload-loc-files-without-restar… Currently Galaxy administrators can reload a tool's XML configuration file, which is very helpful (especially as a developer working on changes to a single tool). Similarly it would be useful to be able to reload one (or all) of the *.loc files (e.g. if a new BLAST database is added), without having to restart Galaxy. Further to this, it would be useful to trigger this from a script via the Galaxy API or otherwise. Should this be filed as a new issue, or a comment on the existing issue? Another idea for enhancement would be to monitor the loc files for *automatic* reloading without user or script intervention (but that might be too much magic), or simply (re)loading loc files on demand rather than at startup. Do the Galaxy team have any plans in this area? Thanks, Peter

1 1

blend4j
by Marc Logghe 03 Mar '13

03 Mar '13

Hi all, Is this the correct forum to post blend4j issues ? Apologies if this is not the case. I was trying to run a custom workflow via blend4j, according to https://github.com/jmchilton/blend4j In my hands, the call to workflowDetails.getInputs() returns an empty Map. I believe the corresponding REST request looks like: /galaxy/api/workflows/<workflow id>?key=<API key> In the JSON response, the 'inputs' attribute is empty indeed ({"url": "/galaxy/api/workflows/f09437b8822035f7", "inputs": {}, ...). I do not understand why this is the case because 2 steps require inputs to be set at runtime. Via the Galaxy web interface those required parameters can be set and the workflow is running smoothly. Or has the term 'inputs' nothing to do with 'parameters' ? Regards, Marc ________________________________________ THIS E-MAIL MESSAGE IS INTENDED ONLY FOR THE USE OF THE INDIVIDUAL OR ENTITY TO WHICH IT IS ADDRESSED AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, CONFIDENTIAL AND EXEMPT FROM DISCLOSURE. If the reader of this E-mail message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately at ablynx(a)ablynx.com. Thank you for your co-operation. "NANOBODY" and "NANOCLONE" are registered trademarks of Ablynx N.V. ________________________________________

2 4

Using dependencies from tool dir at execution time
by Kempenaar, M 03 Mar '13

03 Mar '13

Dear list, I have a few tools that use Python which subsequently tries to load files from a subfolder in the tool dir (i.e. galaxy-dist/tools/myTool/templates/template.sh). However at runtime it seems the Python file(s) are copied to Galaxy's working directory making all other files located in 'myTool' unavailable. Importing a Python file does work so these get copied / moved to the working directory as well but any other file is unavailable. What is the solution of using a different file (for simplicities sake, let's consider any kind of file) that is located in the above mentioned path -without- using any hardcoded paths in my code? Thanks! Best regards, Marcel ________________________________ De inhoud van dit bericht is vertrouwelijk en alleen bestemd voor de geadresseerde(n). Anderen dan de geadresseerde(n) mogen geen gebruik maken van dit bericht, het niet openbaar maken of op enige wijze verspreiden of vermenigvuldigen. Het UMCG kan niet aansprakelijk gesteld worden voor een incomplete aankomst of vertraging van dit verzonden bericht. The contents of this message are confidential and only intended for the eyes of the addressee(s). Others than the addressee(s) are not allowed to use this message, to make it public or to distribute or multiply this message in any way. The UMCG cannot be held responsible for incomplete reception or delay of this transferred message.

2 1