February 2014 - galaxy-dev - lists.galaxyproject.org

a question regarding galaxy 'workflow'
by ruiwang.sz 11 Feb '14

11 Feb '14

Hi All, A quick question on the indices. Here is the information provided by the sample file: #Your bowtie_indices.loc file should include an entry per line for each #index set you have stored. The "file" in the path does not actually #exist, but it is the prefix for the actual index files. For example: #hg18canon hg18 hg18 Canonical /depot/data2/galaxy/bowtie/hg18/hg18canon #hg18full hg18 hg18 Full /depot/data2/galaxy/bowtie/hg18/hg18full #/orig/path/hg19 hg19 hg19 /depot/data2/galaxy/bowtie/hg19/hg19 #Note that for backwards compatibility with workflows, the unique ID of #an entry must be the path that was in the original loc file, because that #is the value stored in the workflow for that parameter. That is why the #hg19 entry above looks odd. New genomes can be better-looking. What exactly is "path that was in the original loc file"? Where do I find this "original loc file"? Are we using a newer version of loc file now? I got the loc file from the main galaxy server, and I could see things like following: ce5 ce5 C. elegans (WS180): ce5 /galaxy/data/ce5/bowtie_index/ce5 /galaxy/data/ce6/bowtie_index/ce6 ce6 C. elegans (WS190): ce6 /galaxy/data/ce6/bowtie_index/ce6 Why would ce5 and ce6 appear differently? for ce6, seems the "original loc file path" is the same as the "filebasepath"? galGal3canon galGal3 Chicken (Gallus gallus): galGal3 Canonical /galaxy/data/galGal3/galGal3canon/bowtie_index/galGal3canon /galaxy/data/galGal3/bowtie_index/galGal3 galGal3 Chicken (Gallus gallus): galGal3 Full /galaxy/data/galGal3/galGal3full/bowtie_index/galGal3full but as we could see, for this galGal3, the "original loc file path" is different from the "filebasepath". same for mm9: mm9canon mm9 Mouse (Mus musculus): mm9 Canonical /galaxy/data/mm9/mm9canon/bowtie_index/mm9canon mm9female mm9 Mouse (Mus musculus): mm9 Canonical Female /galaxy/data/mm9/mm9female/bowtie_index/mm9female /galaxy/data/mm9/bowtie_index/mm9 mm9 Mouse (Mus musculus): mm9 Full /galaxy/data/mm9/mm9full/bowtie_index/mm9full How did this get determined? My conjecture is that in the older version of galaxy we don't have mm9canon and mm9female, for example, but we had mm9, and the entry was like: mm9 mm9 mm9 /galaxy/data/mm9/bowtie_index/mm9 but now we are using a newer version of galaxy and mm9 genome was placed in /galaxy/data/mm9/mm9full/bowtie_index/mm9full so we need to modify this loc file to add the mapping so that we don't have to modify the older workflow that uses mm9? is it just because data is organized differently? that's why for all the newer genomes we don't have that? I'd appreciate that if someone could please elaborate a bit. Best Regards, Rui

1 0

Import Datasets to History problem
by Pandori n 11 Feb '14

11 Feb '14

Hi, Using Bioblend API to communicate with Galaxy Try to import datasets to history, using command: gi.histories.upload_dataset_from_library(historyID, datasetID) But I've got the error: bioblend.galaxy.client.ConnectionError: Unexpected response from galaxy: 400: "Please define the source ('library' or 'hda') and the content." What's the *source *and the content? How to get it and put as arguments into this function?

2 1

Twill comparisons ignore GALAXY_TEST_NO_CLEANUP
by Peter Cock 11 Feb '14

11 Feb '14

Hi all, Something very handy for tool developers running functional tests is if the environment variable GALAXY_TEST_NO_CLEANUP is set, run_functional_tests.sh doesn't remove the test output. This is great when debugging why a test is failing. However, I was having trouble with failures like this: base.twilltestcase: INFO: ## files diff on /mnt/galaxy/galaxy-central/test-data/ecoli.mira4_de_novo.fasta and /tmp/tmpTvyobN/tmpYsEezt/new_files_path_pk7YKe/tmpXr9NYeecoli.mira4_de_novo.fasta lines_diff=0, found diff = 3010 Sadly the temp file here was being deleted - the patch at the end of the email fixes the diff code to respect the configuration setting. Having done that and looked at the preserved file, somehow the wrong output file was being compared to the FASTA output file (it was looking at my log file instead). Because the diff was really between two different files, showing the first forty lines wasn't informative. Instead I've made it show the first 25 and last 25 lines of the diff. I find this a great deal more informative :) Meanwhile this points at a possible bug in the test framework, if anyone wants to attempt to reproduce it I used this version of my MIRA4 wrapper: https://github.com/peterjc/pico_galaxy/commit/3f095eac3f5609b2890440b851029… $ export GALAXY_TEST_NO_CLEANUP=on $ ./run_functional_tests.sh -id mira_4_0_de_novo ... $ hg summary parent: 13513:f3dc213a5773 tip Backout of 28d43f4. branch: default commit: 7 modified, 229 unknown update: (current) Regards, Peter -- $ hg diff test/base/twilltestcase.py diff -r f3dc213a5773 test/base/twilltestcase.py --- a/test/base/twilltestcase.py Mon Feb 10 22:13:35 2014 -0600 +++ b/test/base/twilltestcase.py Tue Feb 11 14:09:30 2014 +0000 @@ -95,7 +95,10 @@ diff = list( difflib.unified_diff( local_file, history_data, "local_file", "history_data" ) ) diff_lines = get_lines_diff( diff ) if diff_lines > allowed_diff_count: - diff_slice = diff[0:40] + if len(diff) < 60: + diff_slice = diff[0:40] + else: + diff_slice = diff[:25] + ["********\n", "* SNIP *\n", "********\n"] + diff[-25:] #FIXME: This pdf stuff is rather special cased and has not been updated to consider lines_diff #due to unknown desired behavior when used in conjunction with a non-zero lines_diff #PDF forgiveness can probably be handled better by not special casing by __extension__ here @@ -897,7 +900,8 @@ errmsg += str( err ) raise AssertionError( errmsg ) finally: - os.remove( temp_name ) + if 'GALAXY_TEST_NO_CLEANUP' not in os.environ: + os.remove( temp_name ) def __default_dataset_fetcher( self ): def fetcher( hda_id, filename=None ): @@ -971,7 +975,8 @@ errmsg += str( err ) raise AssertionError( errmsg ) finally: - os.remove( temp_name ) + if 'GALAXY_TEST_NO_CLEANUP' not in os.environ: + os.remove( temp_name ) def is_zipped( self, filename ): if not zipfile.is_zipfile( filename ):

2 1

Feb 10, 2014 Galaxy Distribution & News Brief
by Jennifer Jackson 11 Feb '14

11 Feb '14

Feb 10, 2014 Galaxy Distribution & News Brief <https://wiki.galaxyproject.org/News/2014_02_10_Galaxy_Distribution> // *CompleteNews Brief <https://wiki.galaxyproject.org/DevNewsBriefs/2014_02_10>* *Highlights:* * Visualization upgrades, including Trackster CSS styling * Multiple Tools migrated to the Tool Shed for a leaner distribution * Redesign of UI rendering: new icons, new font, history pane updates * API functionality upgrades featuring a new master admin API key and * Tool Shed updates a focus on repository metadata, displays, installs, and tests * Over 35 new community contributions added http://getgalaxy.org <http://getgalaxy.org/> http://bitbucket.org/galaxy/galaxy-dist http://galaxy-dist.readthedocs.org <http://galaxy-dist.readthedocs.org/> new: $ hg clone https://bitbucket.org/galaxy/galaxy-dist#stable upgrade: $ hg pull $ hg update release_2014.02.10 /Thanks for using Galaxy!/ The Galaxy Team <https://wiki.galaxyproject.org/Galaxy%20Team>

1 0

Shed_tools couldn't installed due to lack of proxy support. Change in lib/tool_shed/util/common_util.py solves the problem.
by Luca Toldo 10 Feb '14

10 Feb '14

Dear Galaxians, I am behind a proxy with authentication, and I think I will contribute a wiki page on how to install and configure galaxy for saving other people's time... In this mail however I would like to propose a change in the lib/tool_shed/util/common_util.py The problem I had was that not all tool "wrappers" could be installed using the web interface since I was always getting an error "host not found" generated by the urllib2.py call >From that error message (found in paster.log) it was clear that the problem was due to the fact I am behind a firewall with authentication, and the http_proxy variable in the run.sh was not sufficient. I then modified lib/tool_shed/util/common_util.py following the instructions that I have found in http://stackoverflow.com/questions/34079/how-to-specify-an-authenticated-pr… and then now finally it works. Hereafter is what I've modified and it is certainly NOT the appropriate way of doing it from a proper IT point of view (since is not good practice to encode passwords in a source code). Furthermore, with the next mercurial update my code changes will disappear ... I really hope that therefore someone of your smart people could modify the codebase accordingly, perhaps reusing the http_proxy ... proxy_user and proxy_password system variables :-? ------------------------------------------------------------------------------------------------------------------ def tool_shed_get( app, tool_shed_url, uri ): """Make contact with the tool shed via the uri provided.""" registry = app.tool_shed_registry #CHANGES FOR PROXY AUTH password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm() password_mgr.add_password(None, 'http://myproxy.domain.com:8080','MYDOMAIN\MYUSER' , 'MYPASS') proxy_handler=urllib2.ProxyHandler({'http': 'http://myproxy.domain.com:8080'}) proxy_auth_handler=urllib2.ProxyBasicAuthHandler(password_mgr) urlopener = urllib2.build_opener(proxy_handler,proxy_auth_handler) #PREVIOUS CODE # urlopener = urllib2.build_opener() # password_mgr = registry.password_manager_for_url( tool_shed_url ) # if password_mgr is not None: # auth_handler = urllib2.HTTPBasicAuthHandler( password_mgr ) # urlopener.add_handler( auth_handler ) response = urlopener.open( uri ) content = response.read() response.close() return content

1 0

Error introduced with Fastq Groomer
by Philippe Moncuquet 10 Feb '14

10 Feb '14

Hi, Some unexpected symbols were introduced while grooming my fastq file Before @DJTPB5M1:327:C3PC4ACXX:6:1104:9355:84986 1:N:0:GTCCGC GAGCCTTGCTAGGAGAGGGAAGGTGGAAGATCATCATTTCCAGGAGAGCACTGCTAGCAGGAAGCCACGTCTGCATTACACGCTTCATTAGGGACTTCCC + @@@FFFFFFFHHHE@=FDEGCCG2A7CDFHE>F<:B?BDEGGHGICHC9B@FGEHEGG;F=GHI==CE:;BBC>C>>@CC>;8=?=CA;AA>AA<AC<C< After @DJTPB5M1:327:C3PC4ACXX:6:1104:9355:84986 1:N:0:GTCCGC GAGCCTTGCTAGGAGAGGGAAGGTGGAAGATCATCATTTCCAGGAGAGCACTGCTAGCAGGAAGCCACG+1�CATTACACGCTTCATTAGGGACTTCCC + @@@FFFFFFFHHHE@=FDEGCCG2A7CDFHE>F<:B?BDEGGHGICHC9B@FGEHEGG;F=GHI==CE:;BBC>C>>@CC>;8=?=CA;AA>AA<AC<C< I relaunch this step without being able to reproduce the bug. Any ideas about this problem ? Have you guys came across the same problem before ? Regards, Philip

3 2

running tools within tool
by Ketan Maheshwari 10 Feb '14

10 Feb '14

Hi, This is a question I posted to galaxy user mailing list a while back and was redirected to dev for possible answers: Is it possible in Galaxy to design a tool whose sole purpose is to run other tools. This is motivated by our desire to enhance execution capabilities of existing tools via a generic tool which acts as a wrapper. Thanks, Ketan

6 7

SRA datatype
by Todd Oakley 07 Feb '14

07 Feb '14

Hello, I am seeking advice on how to add the SRA datatype to Galaxy. We want to download data from NCBI that is in SRA format, and can no longer have the NCBI site convert it to fastq. I am not experienced with adding file types, to rudimentary advice would be most appreciated. Thanks, Todd Oakley, UCSB

2 1

Running Import History from File as local job
by graham etherington (TSL) 07 Feb '14

07 Feb '14

Hi, I have two local instances of Galaxy running (both entirely independent of each other and running on separate clusters). I’m trying to export a history from Instance A to Instance B, but so far have not been able. I’ve attempted this in two ways. 1. Download the Galaxy_history.tar.gz file from Instance A (Options > Export to File) and place it in a path on the filesystem which Galaxy B can see. Then Options > Import from File and in the 'Archived History URL:’ field place: '/full/path/to/Galaxy_history.tar.gz' (I’ve also tried ‘file:///full/path/to/Galaxy_history.tar.gz') The job runs (submitted to LSF cluster) with no errors in paster.log, but the stdout file in the job_working_directory gives the error Exception getting file from URL: unknown url type:/full/path/to/Galaxy_history.tar.gz <open file '<stderr>', mode 'w' at 0x2ba2d4e9b1e0> Error unpacking tar/gz archive: nothing to open <open file '<stderr>', mode 'w' at 0x2ba2d4e9b1e0> 2. Copy the url link for the History in Instance A (Options > Share or Publish ), then in Instance B go Options > Import from File and in the 'Archived History URL:’ field place the url provided by Instance A. Again the job runs OK, but the stdout file has the error: Exception getting file from URL: <urlopen error [Errno 111] Connection refused> <open file '<stderr>', mode 'w' at 0x2b359790a1e0> Error unpacking tar/gz archive: nothing to open <open file '<stderr>', mode 'w' at 0x2b359790a1e0> I’m pretty sure method number 2 is failing because the Import History job is being run on the cluster which cannot see the outside world (and hence Instance A). I think I should be able to overcome this by specifying in job_conf.xml that the import tool be run locally. The problem is, that I’ve not been able to identify what the actual tool or process is that runs the Import Histories method. I know that it produces a bash script which runs lib/galaxy/tools/imp_exp/unpack_tar_gz_archive.py, but not being a standard tool (i.e. found in the ./tools/ directory with an xml wrapper), I can’t find an ID for it. Naively I tried the following in job_conf.xml: (I have ‘local’ defined in ‘destinations’ and ‘plugins’) <tools default="local">  <tool id="lib/galaxy/tools/imp_exp/unpack_tar_gz_archive.py" destination="local"/> </tools> But this made no difference. Does anyone know how I can either get the Import History tool to run locally or suggest another way to import my history? Many thanks, Graham Dr. Graham Etherington Bioinformatics Support Officer, The Sainsbury Laboratory, Norwich Research Park, Norwich NR4 7UH. UK Tel: +44 (0)1603 450601

2 5

Galaxy server user activity
by Fenglou Mao 07 Feb '14

07 Feb '14

Dear All, We installed a galaxy server here locally, is there any way to generate a report of the user activity? Thanks, Fenglou

2 1