February 2012 - galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] Status on importing BAM file into Library does not update
by Ryan Golhar 13 Feb '12

13 Feb '12

On Tue, Jan 10, 2012 at 11:43 AM, Ryan Golhar <ngsbioinformatics(a)gmail.com>wrote: > >> Hi Ryan, >> >> You could check it in lib/galaxy/config.py, after it's read. By any >> chance, are you using galaxy-central vs. galaxy-dist? It's possible that >> due to a bug I recently fixed and a certain combination of options, >> metadata for BAMs would always fail externally and be retried internally, >> although you should still see log messages indicating that this has >> happened. >> >> --nate >> >> set_metadata_externally is definitely set to True. I added 1 line to check this: self.set_metadata_externally = string_as_bool( kwargs.get( "set_metadata_externally", "False" ) ) self.retry_metadata_internally = string_as_bool( kwargs.get( "retry_metadata_internally", "True" ) ) + log.warning( "Ryan Golhar - self.set_metadata_externally = %s" % self.set_metadata_externally ) and the log files show: [galaxy@bic galaxy-dist]$ grep Ryan *.log runner0.log:WARNING:galaxy.config:Ryan Golhar - self.set_metadata_externally = True web0.log:WARNING:galaxy.config:Ryan Golhar - self.set_metadata_externally = True web1.log:WARNING:galaxy.config:Ryan Golhar - self.set_metadata_externally = True web2.log:WARNING:galaxy.config:Ryan Golhar - self.set_metadata_externally = True web3.log:WARNING:galaxy.config:Ryan Golhar - self.set_metadata_externally = True web4.log:WARNING:galaxy.config:Ryan Golhar - self.set_metadata_externally = True so I know its being read in correctly. I then proceeded to add the same check in /home/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py: #but somewhat trickier (need to recurse up the copied_from tree), for now we'll call set_meta() + log.warning( "Ryan Golhar - self.set_metadata_externally = %s" % self.app.config.set_metadata_externally ) if not self.app.config.set_metadata_externally or \ and it is also set to True: [galaxy@bic galaxy-dist]$ grep Ryan *.log runner0.log:galaxy.jobs WARNING 2012-01-10 22:17:26,381 Ryan Golhar - self.set_metadata_externally = True Clearly something else is going on here. On my last import of BAM file, even after samtools finished indexing the BAM files, Galaxy never registered those jobs as completed. So this problem is still there as well.

5 21

Re: [galaxy-dev] cufflinks could not move file as directed by from_work_dir
by Nate Coraor 13 Feb '12

13 Feb '12

On Feb 13, 2012, at 11:07 AM, P.Baskaran(a)leeds.ac.uk wrote: > Hi Nate, > > I got the following error in the log file after running cufflinks on the local instance of galaxy: > Hi Praveen, Please use the galaxy-dev mailing list (CC'd) for these types of questions so that you can receive a more timely and accurate response. > galaxy.jobs.runners.drmaa DEBUG 2012-02-13 16:03:26,703 (340/2364848) state change: job is running > 10.12.152.44 - - [13/Feb/2012:16:03:29 +0100] "POST /root/history_item_updates HTTP/1.1" 200 - "http://localhost:8181/history" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.46 Safari/535.11" > galaxy.jobs.runners.drmaa DEBUG 2012-02-13 16:03:33,081 (340/2364848) state change: job finished normally > galaxy.jobs DEBUG 2012-02-13 16:03:33,203 finish(): Could not move /nobackup/galaxy/database/job_working_directory/340/cufflinks_out/genes.fpkm_tracking to /nobackup/galaxy/database/files/000/dataset_397.dat as directed by from_work_dir > galaxy.jobs DEBUG 2012-02-13 16:03:33,236 finish(): Could not move /nobackup/galaxy/database/job_working_directory/340/cufflinks_out/isoforms.fpkm_tracking to /nobackup/galaxy/database/files/000/dataset_398.dat as directed by from_work_dir > 10.12.152.44 - - [13/Feb/2012:16:03:34 +0100] "POST /root/history_item_updates HTTP/1.1" 200 - "http://localhost:8181/history" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.46 Safari/535.11" > 10.12.152.44 - - [13/Feb/2012:16:03:34 +0100] "POST /root/history_get_disk_size HTTP/1.1" 200 - "http://localhost:8181/history" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.46 Safari/535.11" > galaxy.jobs DEBUG 2012-02-13 16:03:36,271 job 340 ended > galaxy.datatypes.metadata DEBUG 2012-02-13 16:03:36,272 Cleaning up external metadata files > galaxy.datatypes.metadata DEBUG 2012-02-13 16:03:36,489 Failed to cleanup MetadataTempFile temp files from ../../../../../../nobackup/galaxy/database/tmp/metadata_out_HistoryDatasetAssociation_529_nHaLSL: No JSON object could be decoded: line 1 column 0 (char 0) It looks like the job has failed and what you're seeing here is the failure of the job finish/cleanup process since the job failed to produce expected outputs. Try setting cleanup_job = never in your Galaxy config file and then have a look at the DRM's output and error files to see if this reveals anything about why the job failed. --nate > > > can you please help me to solve this. > > Thanks > Praveen

1 0

Problem with self.object_store.exists in set_dataset_sizes.py
by Peter Cock 13 Feb '12

13 Feb '12

Hi all, I've just updated our Galaxy to the current release 26920e20157f which now includes quota support. This means I need to run set_dataset_sizes.py in order to record the current usage, as explained here: http://wiki.g2.bx.psu.edu/Admin/Disk%20Quotas However, I have run into a problem: $ sudo -u galaxy python2.6 scripts/set_dataset_sizes.py Loading Galaxy model... Processing 3011 datasets... Completed 0% Traceback (most recent call last): File "scripts/set_dataset_sizes.py", line 45, in <module> dataset.set_total_size() File "lib/galaxy/model/__init__.py", line 702, in set_total_size if self.object_store.exists(self, extra_dir=self._extra_files_path or "dataset_%d_files" % self.id, dir_only=True): AttributeError: 'NoneType' object has no attribute 'exists' According to the comments in that file, self.object_store should be initialized in mapping.py (method init) by app.py - apparently that isn't happening. Has anyone else seen this? Thanks, Peter

2 1

generate dynamic select list based on other input dataset
by Holger Klein 13 Feb '12

13 Feb '12

Dear all, I have a working module which generates wig files for genomic annotation from a single column of a bigger input data matrix (Input A). In the current state, the user has to input the column name (Input B) from which to calculate the values in the wig file. Now I'd like to modify the xml in such a way, that depending on the input dataset (Input A) a dynamic list for Input B is generated. I found Hans-Rudolf Hotz' hints from some time ago on this list and thought that the following would be a good start: <param name = "InputB" label = "InputBName" format = "data" type = "select" help = "Use tickboxes to select model" display = "radio" dynamic_options = "getInputBOptions($InputA)" /> <code file="getInputBOptionsFromInputA.py"> getInputBOptionsFromInputA.py contains a single function def getInputBOptions($InputA): ## parse Input A ## create list InputBOptions return(InputBOptions) Using this approach I get an invalid syntax message when trying to even open the module - in any case I have the feeling that something is still missing here. Did anybody solve a similar problem already and could give me a hint on how to solve that? Cheers, Holger -- Dr. Holger Klein Core Facility Bioinformatics Institute of Molecular Biology gGmbH (IMB) http://www.imb-mainz.de/ Tel: +49(6131) 39 21511

2 3

Error with large BAM files?
by Brian Luerman 10 Feb '12

10 Feb '12

Hello, I'm very new to Galaxy development. What we've got is a tool we've created that will use our own API/web services in order to use BAM files in our database as the data source in Galaxy. I got it to work on a very tiny BAM file, but now that that is working, I tried it on a larger one and am seeing the following error: An error occurred running this job: Unable to fetch https://[address to method calling web service]: 'ascii' codec can't decode byte 0xc2 in position 25686: ordinal not in range(128) As far as I can tell from Google, it appears this error has to do with python expecting ascii, but I'm calling Galaxy with the following code: String MimeType = "application/octet-stream"; BufferedOutputStream out = new BufferedOutputStream(response.getOutputStream()); byte[] result = [call to my web service]; response.setContentType(MimeType); response.setHeader("Content-Disposition","attachment;filename=GalaxyResponse"); out.write(result, 0, result.length); Is there anything wrong with this? Or is there some other code I might need to modify, such as in data_source.py? Any help is appreciated, thanks! Brian Luerman Intrepid Bioinformatics (502) 212-2699 http://intrepidbio.com/

1 0

Problem with GMAP datatypes
by Anthonius deBoer 10 Feb '12

10 Feb '12

I tried to install the GMAP from the main toolshed, but I am getting this error when I actually try to run the GSNAP/GMAP tools It seems that there is a problem with the definition of the datatypes...When I check the paster.log it does complain a little, when loading GMAP stuff, but it seems just a problem with the markup of the help text Any ideas? Error in galaxy when running GSNAP WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'gmapdb': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'gmapsnpindex': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'iit': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'splicesites.iit': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'introns.iit': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'snps.iit': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'gmap_annotation': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'gmap_splicesites': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'gmap_introns': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'gmap_snps': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error appending sniffer for datatype 'galaxy.datatypes.gmap:IntervalAnnotation' to sniff_order: No module named gmap WARNING:galaxy.datatypes.registry:Error appending sniffer for datatype 'galaxy.datatypes.gmap:SpliceSiteAnnotation' to sniff_order: No module named gmap WARNING:galaxy.datatypes.registry:Error appending sniffer for datatype 'galaxy.datatypes.gmap:IntronAnnotation' to sniff_order: No module named gmap WARNING:galaxy.datatypes.registry:Error appending sniffer for datatype 'galaxy.datatypes.gmap:SNPAnnotation' to sniff_order: No module named gmap PASTER.LOG ========== galaxy.util.shed_util DEBUG 2012-02-09 12:51:43,862 Installing repository 'gmap' galaxy.util.shed_util DEBUG 2012-02-09 12:51:43,914 Cloning http://toolshed.g2.bx.psu.edu/repos/jjohnson/gmap galaxy.util.shed_util DEBUG 2012-02-09 12:51:46,834 Updating cloned repository to revision "93911bac43da" docutils WARNING 2012-02-09 12:51:47,128 <string>:10: (WARNING/2) Explicit markup ends without a blank line; unexpected unindent. galaxy.util.shed_util DEBUG 2012-02-09 12:51:47,190 Adding new row (or updating an existing row) for repository 'gmap' in the tool_shed_repository table. docutils WARNING 2012-02-09 12:51:47,238 <string>:10: (WARNING/2) Explicit markup ends without a blank line; unexpected unindent. docutils WARNING 2012-02-09 12:51:47,332 <string>:10: (WARNING/2) Explicit markup ends without a blank line; unexpected unindent. galaxy.tools DEBUG 2012-02-09 12:51:47,391 Reloading section: GMAP galaxy.tools DEBUG 2012-02-09 12:51:47,413 Loaded tool id: toolshed.g2.bx.psu.edu/repos/jjohnson/gmap/gmap_iit_store/2.0.0, version: 2.0.0. docutils WARNING 2012-02-09 12:51:47,445 <string>:10: (WARNING/2) Explicit markup ends without a blank line; unexpected unindent. galaxy.tools DEBUG 2012-02-09 12:51:47,462 Loaded tool id: toolshed.g2.bx.psu.edu/repos/jjohnson/gmap/gsnap/2.0.1, version: 2.0.1. galaxy.tools DEBUG 2012-02-09 12:51:47,482 Loaded tool id: toolshed.g2.bx.psu.edu/repos/jjohnson/gmap/gmap_snpindex/2.0.0, version: 2.0.0. galaxy.tools DEBUG 2012-02-09 12:51:47,501 Loaded tool id: toolshed.g2.bx.psu.edu/repos/jjohnson/gmap/gmap_build/2.0.0, version: 2.0.0. galaxy.tools DEBUG 2012-02-09 12:51:47,534 Loaded tool id: toolshed.g2.bx.psu.edu/repos/jjohnson/gmap/gmap/2.0.1, version: 2.0.1. galaxy.datatypes.registry DEBUG 2012-02-09 12:51:49,247 Loading datatypes from /mnt/ngs/analysis/tdeboer/galaxy-data/shed_tools/toolshed.g2.bx.psu.edu/repos/jjohnson/gmap/93911bac43da/gmap/tool-data/datatypes_conf.xml galaxy.datatypes.registry DEBUG 2012-02-09 12:51:49,248 Loaded sniffer for datatype 'galaxy.datatypes.gmap:IntervalAnnotation' galaxy.datatypes.registry DEBUG 2012-02-09 12:51:49,248 Loaded sniffer for datatype 'galaxy.datatypes.gmap:SpliceSiteAnnotation' galaxy.datatypes.registry DEBUG 2012-02-09 12:51:49,248 Loaded sniffer for datatype 'galaxy.datatypes.gmap:IntronAnnotation' galaxy.datatypes.registry DEBUG 2012-02-09 12:51:49,248 Loaded sniffer for datatype 'galaxy.datatypes.gmap:SNPAnnotation' 172.17.50.220 - - [09/Feb/2012:12:51:43 -0700] "POST /admin_toolshed/install_repository?tool_shed_url=http%3A%2F%2Ftoolshed.g2.bx.psu.edu%2F&repo_info_dict=5c18aa85fc57b4d2abf667fb4a49e9cb68d77ae9%3A7b22676d6170223a205b22474d41502047534e415020206d617070696e6720616e6420616c69676e6d656e7420222c2022687474703a2f2f746f6f6c736865642e67322e62782e7073752e6564752f7265706f732f6a6a6f686e736f6e2f676d6170222c2022393339313162616334336461225d7d&includes_tools=True HTTP/1.1" 302 - "http://srp106:8080/admin_toolshed/install_repository?tool_shed_url=http://t…" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:9.0.1) Gecko/20100101 Firefox/9.0.1" 172.17.50.220 - - [09/Feb/2012:12:51:49 -0700] "GET /admin_toolshed/browse_repositories?status=done&message=Installed+1+repository+and+all+tools+were+loaded+into+tool+panel+section+%3Cb%3EGMAP%3C%2Fb%3E%3A%3Cbr%2F%3EInstalled+repositories%3A+gmap.%3Cbr%2F%3E HTTP/1.1" 200 - "http://srp106:8080/admin_toolshed/install_repository?tool_shed_url=http://t…" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:9.0.1) Gecko/20100101 Firefox/9.0.1"

2 3

Re: [galaxy-dev] [galaxy-user] How to do BLAST on Galaxy sever?
by Peter Cock 10 Feb '12

10 Feb '12

On Thu, Feb 9, 2012 at 9:57 PM, Jennifer Jackson <jen(a)bx.psu.edu> wrote: > Hi Peter, > > It looks to me like there are versions both, with the Tool Shed version > having the more recent date stamp. But if those extra functions are not > needed, then yes, you are correct, the version included in the distribution > would be enough. > > Please send corrections if I have any of this wrong, so that David gets the > best information, > > Thanks Peter! > > Jen Hi Jen, I'm not sure which date stamps you are looking at, since I think it is the other way round. The NCBI BLAST+ wrappers shipped with Galaxy are currently at version 0.0.11, both in galaxy-central and galaxy-dist, dated 12 July 2011, e.g. here is the current XML file for the BLASTN wrapper: https://bitbucket.org/galaxy/galaxy-central/src/default/tools/ncbi_blast_pl… https://bitbucket.org/galaxy/galaxy-dist/src/default/tools/ncbi_blast_plus/… Over on the Tool Shed, Edward Kirton has a forked copy of a much older version of these wrappers at v0.0.1. The push date of 7 Jun 2011 is probably from the toolshed migration, I believe the true date is well before that. Edward added support for using local BLAST databases as a new Galaxy datatype, along with wrappers for makeblastdb and dustmasker. Assuming these have proved useful, it would make sense to merge these changes into the main BLAST+ wrappers. I have CC'd the galaxy-dev mailing list (and Edward) which is probably a better place to discuss this than the galaxy-user list. Peter

1 0

"Run as user" code
by Peter van Heusden 09 Feb '12

09 Feb '12

Hi there I recently wrote some code to run Galaxy jobs as a different user, and was just merging my codebase (https://bitbucket.org/pvanheus/galaxy) with galaxy-central when I noticed that code doing the same thing is now part of galaxy-central. The code in galaxy-central is more straightforward than mine, in that it uses external scripts and sudo, whereas mine spawns daemons (each on owned by the relevant system user) and provides a proxy object (implementing the bits of the Python DRMAA API that Galaxy uses) to direct work to the correct daemon. In practice the differences are, however, minor - the effect is the same: processes running as the system user of choice control jobs using DRMAA calls. One issue though is the question of Galaxy user to system user mapping. The current approach (in galaxy-central) is to split the Galaxy username at the @ sign and use everything before the @ as the system user (so pvh(a)sanbi.ac.za becomes pvh). If the system user does not in fact exist, an exception (in the DRMAA job runner) will result. I don't know if anyone will find it useful to have a different approach to Galaxy to system user mapping? For now the current approach is fine for the setup we have at SANBI. Is anyone using this "run as user" code in production? Thanks! Peter

1 0

Garli and Tandem Repeats Finder wrappers
by Oleksandr Moskalenko 08 Feb '12

08 Feb '12

I would like to avoid reinventing the wheel if possible, so I wonder if anyone has wrappers for Garli (https://www.nescent.org/wg_garli/Main_Page) and Tandem Repeats Finder (http://tandem.bu.edu/trf/trf.html) they could share. Thanks, Alex

2 2

set_metadata.py
by Bram Slabbinck 08 Feb '12

08 Feb '12

Hi I am running into a problem with the scripts/set_metadata.py file since the last update of galaxy-dist. In this script the name of the universe file is hard-coded as 'universe_wsgi.ini', see line 44: config_file_name = 'universe_wsgi.ini' # Safe assumption? I renamed the universe file in our setting, due to different Galaxy installs. Running the set_metadata.py script crashes however with this coding as it is constantly trying to make a tmpfile in './database/tmp' which is the default tmp path in universe_wsgi.ini. Replacing 'universe_wsgi.ini' by the name of our universe file solved the problem. So, how can I dynamically retrieve the name of the launched universe file? Thx for any help regards Bram -- ========================================================== Bram Slabbinck, PhD Bioinformatics& Systems Biology Division VIB Department of Plant Systems Biology, UGent Technologiepark 927, 9052 Gent, BELGIUM Email: Bram.Slabbinck(a)psb.ugent.be WWW: http://bioinformatics.psb.ugent.be ========================================================== Please consider the environment before printing this email

2 1