- galaxy-dev - lists.galaxyproject.org

Twill comparisons ignore GALAXY_TEST_NO_CLEANUP
by Peter Cock 11 Feb '14

11 Feb '14

Hi all, Something very handy for tool developers running functional tests is if the environment variable GALAXY_TEST_NO_CLEANUP is set, run_functional_tests.sh doesn't remove the test output. This is great when debugging why a test is failing. However, I was having trouble with failures like this: base.twilltestcase: INFO: ## files diff on /mnt/galaxy/galaxy-central/test-data/ecoli.mira4_de_novo.fasta and /tmp/tmpTvyobN/tmpYsEezt/new_files_path_pk7YKe/tmpXr9NYeecoli.mira4_de_novo.fasta lines_diff=0, found diff = 3010 Sadly the temp file here was being deleted - the patch at the end of the email fixes the diff code to respect the configuration setting. Having done that and looked at the preserved file, somehow the wrong output file was being compared to the FASTA output file (it was looking at my log file instead). Because the diff was really between two different files, showing the first forty lines wasn't informative. Instead I've made it show the first 25 and last 25 lines of the diff. I find this a great deal more informative :) Meanwhile this points at a possible bug in the test framework, if anyone wants to attempt to reproduce it I used this version of my MIRA4 wrapper: https://github.com/peterjc/pico_galaxy/commit/3f095eac3f5609b2890440b851029… $ export GALAXY_TEST_NO_CLEANUP=on $ ./run_functional_tests.sh -id mira_4_0_de_novo ... $ hg summary parent: 13513:f3dc213a5773 tip Backout of 28d43f4. branch: default commit: 7 modified, 229 unknown update: (current) Regards, Peter -- $ hg diff test/base/twilltestcase.py diff -r f3dc213a5773 test/base/twilltestcase.py --- a/test/base/twilltestcase.py Mon Feb 10 22:13:35 2014 -0600 +++ b/test/base/twilltestcase.py Tue Feb 11 14:09:30 2014 +0000 @@ -95,7 +95,10 @@ diff = list( difflib.unified_diff( local_file, history_data, "local_file", "history_data" ) ) diff_lines = get_lines_diff( diff ) if diff_lines > allowed_diff_count: - diff_slice = diff[0:40] + if len(diff) < 60: + diff_slice = diff[0:40] + else: + diff_slice = diff[:25] + ["********\n", "* SNIP *\n", "********\n"] + diff[-25:] #FIXME: This pdf stuff is rather special cased and has not been updated to consider lines_diff #due to unknown desired behavior when used in conjunction with a non-zero lines_diff #PDF forgiveness can probably be handled better by not special casing by __extension__ here @@ -897,7 +900,8 @@ errmsg += str( err ) raise AssertionError( errmsg ) finally: - os.remove( temp_name ) + if 'GALAXY_TEST_NO_CLEANUP' not in os.environ: + os.remove( temp_name ) def __default_dataset_fetcher( self ): def fetcher( hda_id, filename=None ): @@ -971,7 +975,8 @@ errmsg += str( err ) raise AssertionError( errmsg ) finally: - os.remove( temp_name ) + if 'GALAXY_TEST_NO_CLEANUP' not in os.environ: + os.remove( temp_name ) def is_zipped( self, filename ): if not zipfile.is_zipfile( filename ):

2 1

Feb 10, 2014 Galaxy Distribution & News Brief
by Jennifer Jackson 11 Feb '14

11 Feb '14

Feb 10, 2014 Galaxy Distribution & News Brief <https://wiki.galaxyproject.org/News/2014_02_10_Galaxy_Distribution> // *CompleteNews Brief <https://wiki.galaxyproject.org/DevNewsBriefs/2014_02_10>* *Highlights:* * Visualization upgrades, including Trackster CSS styling * Multiple Tools migrated to the Tool Shed for a leaner distribution * Redesign of UI rendering: new icons, new font, history pane updates * API functionality upgrades featuring a new master admin API key and * Tool Shed updates a focus on repository metadata, displays, installs, and tests * Over 35 new community contributions added http://getgalaxy.org <http://getgalaxy.org/> http://bitbucket.org/galaxy/galaxy-dist http://galaxy-dist.readthedocs.org <http://galaxy-dist.readthedocs.org/> new: $ hg clone https://bitbucket.org/galaxy/galaxy-dist#stable upgrade: $ hg pull $ hg update release_2014.02.10 /Thanks for using Galaxy!/ The Galaxy Team <https://wiki.galaxyproject.org/Galaxy%20Team>

1 0

Shed_tools couldn't installed due to lack of proxy support. Change in lib/tool_shed/util/common_util.py solves the problem.
by Luca Toldo 10 Feb '14

10 Feb '14

Dear Galaxians, I am behind a proxy with authentication, and I think I will contribute a wiki page on how to install and configure galaxy for saving other people's time... In this mail however I would like to propose a change in the lib/tool_shed/util/common_util.py The problem I had was that not all tool "wrappers" could be installed using the web interface since I was always getting an error "host not found" generated by the urllib2.py call >From that error message (found in paster.log) it was clear that the problem was due to the fact I am behind a firewall with authentication, and the http_proxy variable in the run.sh was not sufficient. I then modified lib/tool_shed/util/common_util.py following the instructions that I have found in http://stackoverflow.com/questions/34079/how-to-specify-an-authenticated-pr… and then now finally it works. Hereafter is what I've modified and it is certainly NOT the appropriate way of doing it from a proper IT point of view (since is not good practice to encode passwords in a source code). Furthermore, with the next mercurial update my code changes will disappear ... I really hope that therefore someone of your smart people could modify the codebase accordingly, perhaps reusing the http_proxy ... proxy_user and proxy_password system variables :-? ------------------------------------------------------------------------------------------------------------------ def tool_shed_get( app, tool_shed_url, uri ): """Make contact with the tool shed via the uri provided.""" registry = app.tool_shed_registry #CHANGES FOR PROXY AUTH password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm() password_mgr.add_password(None, 'http://myproxy.domain.com:8080','MYDOMAIN\MYUSER' , 'MYPASS') proxy_handler=urllib2.ProxyHandler({'http': 'http://myproxy.domain.com:8080'}) proxy_auth_handler=urllib2.ProxyBasicAuthHandler(password_mgr) urlopener = urllib2.build_opener(proxy_handler,proxy_auth_handler) #PREVIOUS CODE # urlopener = urllib2.build_opener() # password_mgr = registry.password_manager_for_url( tool_shed_url ) # if password_mgr is not None: # auth_handler = urllib2.HTTPBasicAuthHandler( password_mgr ) # urlopener.add_handler( auth_handler ) response = urlopener.open( uri ) content = response.read() response.close() return content

1 0

Error introduced with Fastq Groomer
by Philippe Moncuquet 10 Feb '14

10 Feb '14

Hi, Some unexpected symbols were introduced while grooming my fastq file Before @DJTPB5M1:327:C3PC4ACXX:6:1104:9355:84986 1:N:0:GTCCGC GAGCCTTGCTAGGAGAGGGAAGGTGGAAGATCATCATTTCCAGGAGAGCACTGCTAGCAGGAAGCCACGTCTGCATTACACGCTTCATTAGGGACTTCCC + @@@FFFFFFFHHHE@=FDEGCCG2A7CDFHE>F<:B?BDEGGHGICHC9B@FGEHEGG;F=GHI==CE:;BBC>C>>@CC>;8=?=CA;AA>AA<AC<C< After @DJTPB5M1:327:C3PC4ACXX:6:1104:9355:84986 1:N:0:GTCCGC GAGCCTTGCTAGGAGAGGGAAGGTGGAAGATCATCATTTCCAGGAGAGCACTGCTAGCAGGAAGCCACG+1�CATTACACGCTTCATTAGGGACTTCCC + @@@FFFFFFFHHHE@=FDEGCCG2A7CDFHE>F<:B?BDEGGHGICHC9B@FGEHEGG;F=GHI==CE:;BBC>C>>@CC>;8=?=CA;AA>AA<AC<C< I relaunch this step without being able to reproduce the bug. Any ideas about this problem ? Have you guys came across the same problem before ? Regards, Philip

3 2

running tools within tool
by Ketan Maheshwari 10 Feb '14

10 Feb '14

Hi, This is a question I posted to galaxy user mailing list a while back and was redirected to dev for possible answers: Is it possible in Galaxy to design a tool whose sole purpose is to run other tools. This is motivated by our desire to enhance execution capabilities of existing tools via a generic tool which acts as a wrapper. Thanks, Ketan

6 7

SRA datatype
by Todd Oakley 08 Feb '14

08 Feb '14

Hello, I am seeking advice on how to add the SRA datatype to Galaxy. We want to download data from NCBI that is in SRA format, and can no longer have the NCBI site convert it to fastq. I am not experienced with adding file types, to rudimentary advice would be most appreciated. Thanks, Todd Oakley, UCSB

2 1

Running Import History from File as local job
by graham etherington (TSL) 07 Feb '14

07 Feb '14

Hi, I have two local instances of Galaxy running (both entirely independent of each other and running on separate clusters). I’m trying to export a history from Instance A to Instance B, but so far have not been able. I’ve attempted this in two ways. 1. Download the Galaxy_history.tar.gz file from Instance A (Options > Export to File) and place it in a path on the filesystem which Galaxy B can see. Then Options > Import from File and in the 'Archived History URL:’ field place: '/full/path/to/Galaxy_history.tar.gz' (I’ve also tried ‘file:///full/path/to/Galaxy_history.tar.gz') The job runs (submitted to LSF cluster) with no errors in paster.log, but the stdout file in the job_working_directory gives the error Exception getting file from URL: unknown url type:/full/path/to/Galaxy_history.tar.gz <open file '<stderr>', mode 'w' at 0x2ba2d4e9b1e0> Error unpacking tar/gz archive: nothing to open <open file '<stderr>', mode 'w' at 0x2ba2d4e9b1e0> 2. Copy the url link for the History in Instance A (Options > Share or Publish ), then in Instance B go Options > Import from File and in the 'Archived History URL:’ field place the url provided by Instance A. Again the job runs OK, but the stdout file has the error: Exception getting file from URL: <urlopen error [Errno 111] Connection refused> <open file '<stderr>', mode 'w' at 0x2b359790a1e0> Error unpacking tar/gz archive: nothing to open <open file '<stderr>', mode 'w' at 0x2b359790a1e0> I’m pretty sure method number 2 is failing because the Import History job is being run on the cluster which cannot see the outside world (and hence Instance A). I think I should be able to overcome this by specifying in job_conf.xml that the import tool be run locally. The problem is, that I’ve not been able to identify what the actual tool or process is that runs the Import Histories method. I know that it produces a bash script which runs lib/galaxy/tools/imp_exp/unpack_tar_gz_archive.py, but not being a standard tool (i.e. found in the ./tools/ directory with an xml wrapper), I can’t find an ID for it. Naively I tried the following in job_conf.xml: (I have ‘local’ defined in ‘destinations’ and ‘plugins’) <tools default="local">  <tool id="lib/galaxy/tools/imp_exp/unpack_tar_gz_archive.py" destination="local"/> </tools> But this made no difference. Does anyone know how I can either get the Import History tool to run locally or suggest another way to import my history? Many thanks, Graham Dr. Graham Etherington Bioinformatics Support Officer, The Sainsbury Laboratory, Norwich Research Park, Norwich NR4 7UH. UK Tel: +44 (0)1603 450601

2 5

Galaxy server user activity
by Fenglou Mao 07 Feb '14

07 Feb '14

Dear All, We installed a galaxy server here locally, is there any way to generate a report of the user activity? Thanks, Fenglou

2 1

mpileup -> ImportError: cannot import name stylesheet_link_tag
by Adhemar 07 Feb '14

07 Feb '14

Hi! I was running mpileup and it's showing the error bellow. The strange thing is that 'galaxy/web/framework/helpers/__init__.py' is trying to import 'stylesheet_link_tag' and it's not available in the webhelpers I've installed for both python versions 2.6.6 or 2.7.3 Could you help me with this? Thanx, Adhemar Traceback (most recent call last): File "./scripts/set_metadata.py", line 29, in <module> import galaxy.model.mapping #need to load this before we unpickle, in order to setup properties assigned by the mappers File "/opt/bioinformatics/share/galaxy20130410/lib/galaxy/model/__init__.py", line 23, in <module> import galaxy.datatypes.registry File "/opt/bioinformatics/share/galaxy20130410/lib/galaxy/datatypes/registry.py", line 5, in <module> import data, tabular, interval, images, sequence, qualityscore, genetics, xml, coverage, tracks, chrominfo, binary, assembly, ngsindex, graph File "/opt/bioinformatics/share/galaxy20130410/lib/galaxy/datatypes/data.py", line 2, in <module> import metadata File "/opt/bioinformatics/share/galaxy20130410/lib/galaxy/datatypes/metadata.py", line 24, in <module> from galaxy.web import form_builder File "/opt/bioinformatics/share/galaxy20130410/lib/galaxy/web/__init__.py", line 4, in <module> from framework import expose File "/opt/bioinformatics/share/galaxy20130410/lib/galaxy/web/framework/__init__.py", line 28, in <module> import helpers File "/opt/bioinformatics/share/galaxy20130410/lib/galaxy/web/framework/helpers/__init__.py", line 4, in <module> from webhelpers import date, stylesheet_link_tag, javascript_include_tag, url_for ImportError: cannot import name stylesheet_link_tag

2 1

sysadmin help
by Shrum, Donald C 06 Feb '14

06 Feb '14

I have a galaxy server set up, running off of our HPC cluster file system. I'd like to submit a simple job to test my settings with our moab scheduler. Could someone point me to the easiest possible example of uploading and running a job... possible an example a sysadmin with limited knowledge could follow ;) Thanks, Don FSU RCC

2 1