November 2013 - galaxy-dev - lists.galaxyproject.org

Several jobs stalled in the queue: Job runner assigned but no external ID recorded
by Derrick Lin 05 Feb '14

05 Feb '14

Hi guys, I have several jobs have been stuck in the queue for long time, I checked the handler log and found the following message: Job runner assigned but no external ID recorded, adding to the job handler queue Is there a way that I can clean these orphan jobs out of the queue? Cheers, Derrick

1 1

A new tool_data_table_conf.xml created under tool-data directory
by Derrick Lin 18 Jan '14

18 Jan '14

Hi guys, After installing the BWA from the Tools Shed, I find there is a new tool_data_table_conf.xml got created under tool-data directory. The file contains the entries of BWA's loc files. I believe the installed BWA still relies on the tool_data_table_conf.xml under galaxy root dir. Anyone can clarify what the one under tool-data dir is for? Regards, Derrick

3 7

trackster error for viewing rat data (rn5)
by Shu-Yi Su 14 Jan '14

14 Jan '14

Hi, I am trying to view rat data (rn5) using trackster with data format bam or bigwig on our galaxy installation, but it isn't working. It works with other organisms, such as dm3 or hg19. There is error message in the log: 10.11.70.120 - - [20/Nov/2013:16:14:21 +0200] "GET /galaxy-dev/api/genomes/rn5?low=0&num=100 HTTP/1.1" 200 - "http://gbcs-dev/galaxy-dev/visualization/trackster?dataset_id=83c20bbc71049…" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/536.30.1 (KHTML, like Gecko) Version/6.0.5 Safari/536.30.1" 10.11.70.120 - - [20/Nov/2013:16:14:21 +0200] "GET /galaxy-dev/api/datasets/83c20bbc7104922b?hda_ldda=hda&data_type=track_config HTTP/1.1" 200 - "http://gbcs-dev/galaxy-dev/visualization/trackster?dataset_id=83c20bbc71049…" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/536.30.1 (KHTML, like Gecko) Version/6.0.5 Safari/536.30.1" galaxy.webapps.galaxy.api.datasets ERROR 2013-11-20 16:14:21,795 Error in dataset API at listing contents: 'BinaryFileReader' object has no attribute 'read_bits64': 'BinaryFileReader' object has no attribute 'read_bits64' Traceback (most recent call last): File "/g/funcgen/galaxy-dev/lib/galaxy/webapps/galaxy/api/datasets.py", line 44, in show is_true( kwd.get( 'retry', False ) ) ) File "/g/funcgen/galaxy-dev/lib/galaxy/webapps/galaxy/api/datasets.py", line 106, in _converted_datasets_state if not data_provider.has_data( chrom ): File "/g/funcgen/galaxy-dev/lib/galaxy/visualization/data_providers/genome.py", line 1078, in has_data all_dat = bbi.query( chrom, 0, 2147483647, 1 ) or \ File "bbi_file.pyx", line 215, in bx.bbi.bbi_file.BBIFile.query (lib/bx/bbi/bbi_file.c:5596) File "bbi_file.pyx", line 222, in bx.bbi.bbi_file.BBIFile.query (lib/bx/bbi/bbi_file.c:5210) File "bbi_file.pyx", line 183, in bx.bbi.bbi_file.BBIFile.summarize (lib/bx/bbi/bbi_file.c:4475) File "bbi_file.pyx", line 248, in bx.bbi.bbi_file.BBIFile._get_chrom_id_and_size (lib/bx/bbi/bbi_file.c:5656) File "bpt_file.pyx", line 76, in bx.bbi.bpt_file.BPTFile.find (lib/bx/bbi/bpt_file.c:1388) File "bpt_file.pyx", line 55, in bx.bbi.bpt_file.BPTFile.r_find (lib/bx/bbi/bpt_file.c:1154) AttributeError: 'BinaryFileReader' object has no attribute 'read_bits64' I also attached the first few lines in our rn5.len: chr1 290094216 chr2 285068071 chr4 248343840 chr3 183740530 chr5 177180328 chr6 156897508 chrX 154597545 chr7 143501887 chr8 132457389 chr9 121549591 chr13 118718031 chr14 115151701 chr15 114627140 chr10 112200500 chr11 93518069 chr17 92503511 chr16 90051983 chr18 87229863 chr19 72914587 chr20 57791882 chr12 54450796 chr4_AABR06109730_random 227955 chrUn_JH620568 73090 chr2_JH620298_random 62026 chr5_JH620330_random 58807 chrUn_JH620520 57992 chrUn_JH620553 56877 chr1_AABR06109393_random 52834 chr18_JH620420_random 51685 chr9_JH620364_random 51490 chrX_JH620445_random 49212 chrUn_JH620623 47760 chr10_JH620371_random 44089 I have looked into this error for a while and still can't figure it out. Any idea or suggestions are very appreciated. Thank you very much. Best regards, Shu-Yi

2 4

Local installation problem
by Cesar Rodriguez 03 Jan '14

03 Jan '14

Hi there, I am trying to install Galaxy locally on my Mac OSX 10.9, python 2.7. Nonetheless, when I run it, it says that some eggs are out of date. I have fetched them, but the following message appears: Mac-Pro-Cesar-Rodriguez:scripts Rodriguez$ python fetch_eggs.py Warning: MarkupSafe (a dependent egg of Mako) cannot be fetched Warning: pycrypto (a dependent egg of Fabric) cannot be fetched Fetched http://eggs.galaxyproject.org/paramiko/paramiko-1.11.1-py2.7.egg One of Galaxy's managed eggs depends on something which is missing, this is almost certainly a bug in the egg distribution. Dependency "paramiko" requires "pycrypto>=2.1,!=2.4" Traceback (most recent call last): File "fetch_eggs.py", line 37, in <module> c.resolve() # Only fetch eggs required by the config File "/Users/Rodriguez/galaxy-dist/lib/galaxy/eggs/__init__.py", line 345, in resolve egg.resolve() File "/Users/Rodriguez/galaxy-dist/lib/galaxy/eggs/__init__.py", line 168, in resolve dists = pkg_resources.working_set.resolve( ( self.distribution.as_requirement(), ), env, self.fetch ) File "/Users/Rodriguez/galaxy-dist/lib/pkg_resources.py", line 569, in resolve raise VersionConflict(dist,req) # XXX put more info here pkg_resources.VersionConflict: (paramiko 1.11.1 (/Users/Rodriguez/galaxy-dist/eggs/paramiko-1.11.1-py2.7.egg), Requirement.parse('pycrypto>=2.1,!=2.4¹)) What should I do to fix this? Thank you so much for your help. Cesar

3 3

Including descriptions etc in extended BLAST+ tabular output
by Peter Cock 25 Dec '13

25 Dec '13

Hello all, FAO: Administrators of local Galaxy instances using the NCBI BLAST+ wrappers. Over on the galaxy_blast repository I have been updating the NCBI BLAST+ wrappers (including unit tests) to work with the current release, NCBI BLAST+ 2.2.28 (aka BLAST 2.2.28+): https://github.com/peterjc/galaxy_blast The initial set of changes is now on the Test Tool Shed, http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus This includes a workaround for a known regression in the makeblastdb tool dealing with duplicated identifiers: https://github.com/peterjc/galaxy_blast/commit/349e31c6cec4429c5523fde5975e… In terms of end-user features, the big improvement in the BLAST+ 2.2.28 release was the ability to get the BLAST match descriptions in the tabular output, and other fields: http://blastedbio.blogspot.co.uk/2012/05/blast-tabular-missing-descriptions… staxids means Subject Taxonomy ID(s), separated by a ';' sscinames means Subject Scientific Name(s), separated by a ';' scomnames means Subject Common Name(s), separated by a ';' sblastnames means Subject Blast Name(s), separated by a ';' (in alphabetical order) sskingdoms means Subject Super Kingdom(s), separated by a ';' (in alphabetical order) stitle means Subject Title salltitles means All Subject Title(s), separated by a '<>' sstrand means Subject Strand qcovs means Query Coverage Per Subject qcovhsp means Query Coverage Per HSP On this branch I am including the new salltitles field as the 25th column in the extended BLAST tabular output offered within the Galaxy interface: https://github.com/peterjc/galaxy_blast/tree/c25 However, I'm not so sure about the taxonomy fields. Since (thus far) they are not available via the XML, I am leaning to introducing a third tabular mode, e.g. * Standard 12 columns (can convert from XML) * Extended 25 columns (can convert from XML) * Extended also with taxonomy (cannot currently convert from XML) Instead, we could offer a pick-you-own columns route (in all the primary BLAST tools, handled via macros)?: * Standard 12 columns (can convert from XML) * Extended 25 columns (can convert from XML) * Pick your own columns from the full NCBI list (depending on columns, can convert from XML) This is inspired by JJ's changes to the BLAST XML to tabular conversion tool for Galaxy-P, https://github.com/jmchilton/galaxy_blast/commit/d79afc03522768323494818a40… I would be much keener on the pick-you-own columns option if it was possible for the tool to record arbitrary column names for a tabular file in Galaxy's metadata (I can't find a trello card, but I'm sure I've asked about this before). Any thoughts or comments? eg Hurry up and just release this branch adding the hit descriptions as column 25 - we want that now ;) [*] Regards, Peter [*] For our local instance, the taxonomy stuff will be useful, but right now I would prioritise the description, which we currently get via the BLAST XML using this tool: https://github.com/peterjc/galaxy_blast/tree/master/tools/blastxml_to_top_d…

4 13

Tool Access Control
by Eric Rasche 21 Dec '13

21 Dec '13

Howdy devs, I've implemented some rather basic tool access control and am looking for feedback on my implementation. # Why Our organisation wanted the ability to restrict tools to different users/roles. As such I've implemented as an "execute" tag which can be applied to either <section> or <tools> in the tool configuration file. # Example galaxy-admin changes For example: <section execute="a@b.co,b@b.co" id="EncodeTools" name="ENCODE Tools"> <tool file="encode/gencode_partition.xml" /> <tool execute="b(a)b.co" file="encode/random_intervals.xml" /> </section> which would allow A and B to access gencode_parition, but only B would be able to access random_intervals. To put it explicity - by default, everyone can access all tools - if section level permissions are set, then those are set as defaults for all tools in that section - if tool permissions are set, they will override the defaults. # Pros and Cons There are some good features - non-accessible tools won't show up in the left hand panel, based on user - non-accessible tools cannot be run or accessed. There are some caveats however. - existence of tools is not completely hidden. - Labels are not hidden at all. - workflows break completely if a tool is unavailable to a shared user and the user copies+edits. They can be copied, and viewed (says tool not found), but cannot be edited. Tool names/id/version info can be found in the javascript object due to the call to app.toolbox.tool_panel.items() in templates/webapps/galaxy/workflow/editor.mako, as that returns the raw tool list, rather than one that's filtered on whether or not the user has access. I'm yet to figure out a clean fix for this. Additionally, empty sections are still shown even if there aren't tools listed in them. For a brief overview of my changes, please see the attached diff. (It's missing one change because I wasn't being careful and started work on multiple different features) # Changeset overview In brief, most of the changes consist of - new method in model.User to check if an array of roles overlaps at all with a user's roles - modifications to appropriate files for reading in the new tool_config.xml's options - modification to get_tool to pass user information, as whether or not a tool exists is now dependent on who is asking. Please let me know if you have input on this before I create a pull request on this feature. # Fixes I believe this will fix a number of previously brought up issues (at least to my understanding of the issues listed) + https://trello.com/c/Zo7FAXlM/286-24-add-ability-to-password-secure-tools + (I saw some solution where they were adding "_beta" to tool names which gave permissions to developers somewhere, but cannot find that now) Cheers, Eric Rasche -- Eric Rasche Programmer II Center for Phage Technology Texas A&M University College Station, TX 77843 404-692-2048 esr(a)tamu.edu rasche.eric(a)yandex.ru

3 6

New Galaxy won't start, SQLite (OperationalError) database is locked
by Peter Cock 20 Dec '13

20 Dec '13

Hello all, While attempting a fresh Galaxy install from both galaxy-central and galaxy-dist, I ran into a problem initialising the default SQLite database (database/universe.sqlite), $ hg clone https://bitbucket.org/galaxy/galaxy-dist ... $ cd galaxy-dist $ ./run.sh ... galaxy.model.migrate.check DEBUG 2013-10-31 10:28:26,143 pysqlite>=2 egg successfully loaded for sqlite dialect Traceback (most recent call last): ... OperationalError: (OperationalError) database is locked u'PRAGMA table_info("dataset")' () After some puzzlement, I realised this was down to the file system - I was trying this under my home directory mounted via a distributed file system (gluster I think). Repeating the experiment under /tmp on a local hard disk worked :) (I'm posting this message for future reference; hopefully Google and/or mailing list searches will help anyone else facing this error) Regards, Peter

2 5

Re: [galaxy-dev] Galaxy wrappers for CLC Assembly Cell (CLCbio)
by Eric Kuyt 12 Dec '13

12 Dec '13

Hi Peter, After installing the clc testing galaxy wrapper I noticed there are still some static paths in the wrapper. I'm in favor of expecting binaries in the system path, but maybe that's a matter of taste. Greets, Eric Kuijt On 30 October 2013 17:03, Peter Cock <p.j.a.cock(a)googlemail.com> wrote: > Hello all, > > This is just to announce I am working on a wrapper for > "CLC Assembly Cell" which is the CLCbio commercial > command line assembly tool suite. > http://www.clcbio.com/products/clc-assembly-cell/ > > Our institute bought a licence primarily for use on plant > genomes where other assemblers at the time required > too much RAM to complete. This assembler is both fast > and low memory, which can be very useful. > > Wrapper development here: > https://github.com/peterjc/pico_galaxy/tree/master/tools/clc_assembly_cell > > Prototype releases will be on the Test Tool Shed (soon): > http://testtoolshed.g2.bx.psu.edu/view/peterjc/clc_assembly_cell > > Stable Tool Shed releases will be here (later): > http://toolshed.g2.bx.psu.edu/view/peterjc/clc_assembly_cell > > I would be interested to hear from anyone else with access to > a licensed copy of the tool interested in using it from Galaxy. > e.g. Is it reasonable to assume the tools are on the $PATH, > or is using a specific environment variable more helpful? > > Regards, > > Peter > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ > > >

3 5

FTP Problem: Users cannot see their uploads files.
by Misharl mon 08 Dec '13

08 Dec '13

Hi everybody, I have managed to get proftpd to work, it can connect to the galaxy sql database, and users can log to upload files in their directory. But there is a problem, when a galaxy user logs in the galaxy web platform, the user can't see his upload files since Galaxy doesn't have the rights to open the directory. How can i change the permissions in proftpd so that galaxy can open the user directory? Thanks in advance to all. Mish

4 17

RFC: GALAXY_MEMORY similar to GALAXY_SLOTS
by Bjoern Gruening 05 Dec '13

05 Dec '13

Hi, if we want to include JAVA programs into Galaxy we are usually forced to hardcode -Xmx4g -Xms3g or similar into the wrapper. Would it make sense to have something like GALAXY_MEMORY, similar to GALAXY_SLOTS, to obtain the maximum of allowed memory from the scheduler. With some cheetah magic we would be able to create dynamic "-Xmx4g -Xms3g" entries. Ciao, Bjoern

2 1