March 2013 - galaxy-dev - lists.galaxyproject.org

Running with Multiple DRMs
by Ganote, Carrie L 28 Mar '13

28 Mar '13

Hi Galaxy-Dev, I have a new local instance that submits to cluster1 via pbs-drmaa-1.0.14. Galaxy is on a VM that is a submit node on cluster1, and there is a shared filesystem amongst all the clusters where the datasets are stored. The Galaxy VM is also a submit node on cluster2 and we have installed another Torque client on the VM in a different location to interface with that one. We have confirmed that using /usr/local/bin/qsub submits to cluster1 and using /N/softs/galaxy/torque/qsub submits to cluster2. The two clusters are using different versions of Torque, but our end goal is to find a way to make it possible to submit to condor and sge as well (as I understand, setting up a distributed Galaxy is a hot topic for multiple institutions). My question is then, how to go about submitting certain jobs (say, all blast jobs) to cluster2 from Galaxy? Here are my ideas so far: (1.) If I can change the path from the default Torque client to our new Torque client while a tool is being submitted, that should do the trick -- that way Galaxy and Drmaa don't need to know the details. I thought about doing this in 200_runners.py but I don't think it would work to just export the path. (2.) If I can get drmaa configured to submit using the correct qsub command given drmaa://-q queuename@host, that would work. Currently just trying the queue and hostname hasn't worked. (3.) If tool handlers can be configured to use different settings for running jobs, this might be workable. I looked into: http://wiki.galaxyproject.org/Admin/Config/Jobs That looks very promising but I don't see job_conf* anywhere in my new clone. (4.) Change the wrapper to include some other parameter returned by 200_runners.py as is suggested here: http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-June/010080.html Not sure if this would do what I want. (5.) I may need to make changes to execute() and DefaultToolAction as alluded to here: http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-June/009964.html This will require more time and energy to understand and modify than I have available at the moment. Are there any other insights into this sort of setup? Thanks! -Carrie Ganote

1 0

Trinity wrapper not working on local installation
by Raj Ayyampalayam 28 Mar '13

28 Mar '13

Hello, I am trying to get the trinity wrapper to work on our local galaxy installation using the latest trinity version. The main issue is that the Trinity.pl call has --SS_lib_type None in the script file. The data I am using is unstranded paired end reads and I am selecting None in the tool parameters. I looked at the trinity_all.xml file and it seems like the following code is not working: #if $inputs.library_type != 'None': --SS_lib_type $inputs.library_type #end if I am new to cheeta and python and I am not sure why this code is not working. Any suggestion on how to go about debugging it? Thanks, -Raj

3 3

Missing from tool shed - lost in migration or never uploaded?
by Peter Cock 28 Mar '13

28 Mar '13

Hi Greg, Is there anything left of the old (pre-hg) Tool Shed? I'm wondering if I ever uploaded this tool back in early 2011, in which case could it have got lost in the migration to the current hg-based Tool Shed?: https://bitbucket.org/peterjc/galaxy-central/src/tools/tools/primers/seq_pr… https://bitbucket.org/peterjc/galaxy-central/src/tools/tools/primers/seq_pr… https://bitbucket.org/peterjc/galaxy-central/src/tools/tools/primers/seq_pr… <tool id="seq_primer_clip" name="Primer clip sequences" version="0.0.7"> <description>Trim off 5' or 3' primers</description> <command interpreter="python"> seq_primer_clip.py $input_file $input_file.ext $primer_fasta $primer_type $mm $min_len $keep_negatives $output_file </command> ... If searching the archives is a hassle, don't worry - I'm just curious. Another likely explanation is simply that I never upload the tool - it was written for a Roche 454 dataset (to remove SMART primers) and could work directly on the SFF files. Unfortunately it is probably too slow to use on large Illumina FASTQ files. I might as well (re)post it to the Tool Shed now, perhaps with a warning that it is intended for smaller Roche/IonTorrent/Sanger sized datasets. Thanks, Peter

2 2

Circster Visualization
by Merella Stefania 28 Mar '13

28 Mar '13

Dear all, this is the first time that I write to this mailing list. I'm trying to setting up a local instance of galaxy. I enabled Trackster visualization and it works fine and I can see the icon button of circster visualization. But if I click on it I receive an error. Obviously I get this error because I didn't change any file to activate it. The problem is that I don't known how to activate it. Googling "galaxy circster" I didn't find anything that helped as well as in the past mails of this mailing list… Can someone help me and tell me where I can find a wiki or something similar? Thanks in advance, Stefania -------------------------------------------------------------------------- LA TUA CURA E' SCRITTA NEL TUO DNA. AL SAN RAFFAELE LA STIAMO REALIZZANDO. AIUTA LA RICERCA, DAI IL TUO 5XMILLE - CF: 07636600962 info:www.5xmille@hsr.it - www.5xmille.org Disclaimer added by CodeTwo Exchange Rules 2007 http://www.codetwo.com

3 4

Re: [galaxy-dev] FTP upload - symlink to uploaded data
by Rob Hooft 28 Mar '13

28 Mar '13

Joachim, Nate, Leon Mei pointed me to a mailing list post of August 2012 where you two discussed a problem with uploads to Galaxy filling up /tmp. I think I have traced this down now after we suffered from this too several times. There are a number of places where temporary files are configurable in galaxy, but there is (at least) one place that uses the Python default directory (can be set with TMPDIR or some other envvars, but if you don't it is often /tmp). The "unconfigurable" place is tools/data_source/upload.py, where the code reads: if dataset.type == 'url': try: page = urllib.urlopen( dataset.path ) #page will be .close()ed by sniff methods temp_name, dataset.is_multi_byte = sniff.stream_to_file( page, prefix='url_paste', source_encoding=util.get_charset_from_http_headers( page.headers ) ) except Exception, e: file_err( 'Unable to fetch %s\n%s' % ( dataset.path, str( e ) ), dataset, json_file ) return dataset.path = temp_name sniff.stream_to_file uses the tempfile module, and since there is no "dir=" in the argument list to this call, the temporary file is made in /tmp. The central solution for the main galaxy code is in lib/galaxy/config.py: self.new_file_path = resolve_path( kwargs.get( "new_file_path", "database/tmp" ), self.root ) tempfile.tempdir = self.new_file_path But this assignment to "tempdir" does not help in this case because upload.py is a tool? It would be nice to fix this, which we can obviously do ourselves for our andromeda deployment, but it would be better to do it centrally. Regards, Rob -- Rob W.W. Hooft Chief Technology Officer BioAssist, Netherlands Bioinformatics Centre http://www.nbic.nl/ Skype: robhooft GSM: +31 6 27034319

2 1

Tool run on cluster returns error state on clean install
by Ganote, Carrie L 27 Mar '13

27 Mar '13

Hi all, I am trying to upgrade our instance to the newest dist but I get the following in stderr when I attempt to run Bowtie or BWA: PREFIX/subs/galaxy_31.sh: command substitution: line 7: syntax error near unexpected token `(' PREFIX/subs/galaxy_31.sh: command substitution: line 7: `diff -n <(printf "%s\n" "${LIST_BEFORE[@]}") <(printf "%s\n" "${LIST_AFTER[@]}") | tail -n +2' PREFIX/subs/galaxy_31.sh: command substitution: line 7: syntax error near unexpected token `(' PREFIX/subs/galaxy_31.sh: command substitution: line 7: `diff -n <(printf "%s\n" "${LIST_BEFORE[@]}") <(printf "%s\n" "${LIST_AFTER[@]}") | tail -n +2' The job is going to the cluster and coming back. The tool appears to run even with the weird error. Sometimes the results are even correct, but I can't use them due to the error state. Has anyone seen this error before? Sincerely, Carrie Ganote

2 2

alphabetically sorted references from admin tool -> numerically sorted?
by Langhorst, Brad 27 Mar '13

27 Mar '13

GATK really does not like alphabetically sorted references. http://gatkforums.broadinstitute.org/discussion/1204/what-input-files-does-… Does it make sense to rearrange genomes to be sorted in [chr1,chr2, chr3,...] order instead of [chr1,chr10,chr11,...] order before indexing? Lots of tools care about sorting… are there any that must have alphabetical order and not numerical order? Brad -- Brad Langhorst langhorst(a)neb.com<mailto:langhorst@neb.com>

1 0

Question about use of <options> tag and from_dataset with non-text data files
by Marc F Paterno 27 Mar '13

27 Mar '13

Hello, We are working on an astrophysics project, and making use of a currently private Galaxy instance. We have introduced a new datatype, which we call 'db'; this datatype corresponds to SQLite3 database files. We would like to be able to use the <options> tag, with the from_dataset attribute, to allow us to form a selection of options determined by data available in the SQLite3 file. However, looking at the code in dynamic_options.py, we don't see how to do what we want. It appears that the code associated with 'from_dataset' only supports obtaining information from textual files. When we tried to use the options tag, we obtain a failure that looks like the system has read the first megabyte of data from the SQLite3 file as a string, which of course can't be interpreted. Is there an way to make our 'db' data type have the right interface to allow the option selection to work with it? thanks Marc and Jim

1 0

galaxy not picking up updates from test toolshed
by jorrit poelen 26 Mar '13

26 Mar '13

Hey y'all, I noticed that my local galaxy instance doens't pickup latest updates from test toolshed. Steps to reproduce: 1. install obotools from test toolshed using hg clone http://jorrit@testtoolshed.g2.bx.psu.edu/repos/jorrit/obotools , revision 45:bb20766df19c 2. hg push new change of obotools of galaxy tool to testtoolshed , creating revision 46:532808477800 3. update obotools in galaxy admin expected result: galaxy admin detects new version and give an option to update. actual result: galaxy admin does not detect new version. I've attached some galaxy logging below. One thing I noticed that the reported commit time in testtoolshed seems 7 hours off - just minutes after I pushed the change, testtoolshed says the change was commit "~ 7 hours ago". Workaround I am using to do update is to deactivate and remove the tool all together, and re-install straight from test-toolshed. What should I do to make the tool version update work? I am probably missing something. Thanks! -jorrit Galaxy logging on looking for updates in test toolshed: --- 24.130.122.217 - - [26/Mar/2013:01:12:46 +0000] "GET /admin_toolshed/browse_repositories HTTP/1.1" 200 - "http://galaxy:7474/admin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10" 24.130.122.217 - - [26/Mar/2013:01:12:51 +0000] "GET /admin_toolshed/browse_repositories?operation=get+updates&id=a799d38679e985db HTTP/1.1" 302 - "http://galaxy:7474/admin_toolshed/browse_repositories" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10" 24.130.122.217 - - [26/Mar/2013:01:12:51 +0000] "GET /admin_toolshed/check_for_updates?id=a799d38679e985db HTTP/1.1" 302 - "http://galaxy:7474/admin_toolshed/browse_repositories" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10" 24.130.122.217 - - [26/Mar/2013:01:12:52 +0000] "GET /admin_toolshed/update_to_changeset_revision?tool_shed_url=http://testtoolshed.g2.bx.psu.edu/&name=obotools&owner=jorrit&changeset_revision=bb20766df19c&latest_changeset_revision=bb20766df19c&latest_ctx_rev=45 HTTP/1.1" 302 - "http://galaxy:7474/admin_toolshed/browse_repositories" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10" 24.130.122.217 - - [26/Mar/2013:01:12:52 +0000] "GET /admin_toolshed/manage_repository?status=done&message=The+installed+repository+named+%27obotools%27+is+current%2C+there+are+no+updates+available.++&id=a799d38679e985db HTTP/1.1" 200 - "http://galaxy:7474/admin_toolshed/browse_repositories" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10"

2 3

best practice to replace the hashing for galaxy-user passwords
by Vipin TS 26 Mar '13

26 Mar '13

Dear dev-team, I am interested to implement the secure hashed password specifically SHA-256 or greater to the galaxy users using publicly available python module PBKDF2. I have a piece of python code which will do the job but not quite sure how I can integrate this to my local galaxy code repository. I saw the current implementation in galaxy/util/hash_util.py any suggestions, thanks, --/Vipin

1 0