- galaxy-dev - lists.galaxyproject.org

Functional tests results in Invalid HTTP return code 404/500, allowed codes: 200
by Serrano Pereira 19 Oct '13

19 Oct '13

Dear Galaxy hackers, I'm having trouble with the tool test framework. I have a local installation of a Galaxy Tool Shed (latest stable from the repository) on my Linux server. I have an Apache proxy to my tool shed (per http://wiki.galaxyproject.org/HostingALocalToolShed) The Tool Shed seems to work fine. I'm able to create and maintain repositories and I'm able to install tools from the Tool Shed from remote Galaxy instances. Now I want to run the functional tests of the tools in the Tool Shed. I do that by following the instructions on the wiki (http://wiki.galaxyproject.org/AutomatedToolTests) It comes down to the following commands: ---------------------------------------------------------------------- cd /opt/galaxy/galaxy-dist/ export GALAXY_INSTALL_TEST_TOOL_SHED_API_KEY=2aeecfe2d9f28d0c5b6959a41886b5fd export GALAXY_INSTALL_TEST_TOOL_SHED_URL=http://toolshed.galaxy.domain.org/ python lib/tool_shed/scripts/check_repositories_for_functional_tests.py tool_shed_wsgi.ini sh install_and_test_tool_shed_repositories.sh ---------------------------------------------------------------------- The second script (install_and_test_tool_shed_repositories.sh) fails with the following error: ====================================================================== FAIL: install_repository_usearch (install_and_test_tool_shed_repositories.functional.test_install_repositories.TestInstallRepository_usearch) Install the repository usearch from http://toolshed.galaxy.domain.org/. ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/galaxy/galaxy-dist/test/install_and_test_tool_shed_repositories/functional/test_install_repositories.py", line 48, in test_install_repository self.do_installation( repository_dict ) File "/opt/galaxy/galaxy-dist/test/install_and_test_tool_shed_repositories/functional/test_install_repositories.py", line 17, in do_installation self.install_repository( repository_info_dict ) File "/opt/galaxy/galaxy-dist/test/install_and_test_tool_shed_repositories/base/twilltestcase.py", line 61, in install_repository self.visit_url( '%s/repository/preview_tools_in_changeset?%s' % ( tool_shed_url, preview_params ) ) File "/opt/galaxy/galaxy-dist/test/install_and_test_tool_shed_repositories/base/twilltestcase.py", line 112, in visit_url ( return_code, url, ', '.join( str( code ) for code in allowed_codes ) ) AssertionError: Invalid HTTP return code 404 for http://toolshed.galaxy.domain.org//repository/preview_tools_in_changeset?re…, allowed codes: 200 ---------------------------------------------------------------------- (I've changed twilltestcase.py to also print the URL that is being browsed.) This is were it get's weird. The HTTP 404 response code indicates that the page was not found. However, when I browse to that URL, it works fine! What's more, when I try to access that page with `wget` from the same machine the HTTP response code is 200: ---------------------------------------------------------------------- wget http://toolshed.galaxy.domain.org//repository/preview_tools_in_changeset?re… ... HTTP request sent, awaiting response... 200 OK ... ---------------------------------------------------------------------- When I use the Tool Shed URL http://localhost:9009/ instead, I get a similar error, only the return code is 500 instead of 404. I suspect that the HTTP return code returned by the Python module that returns it (Mechanize?) is invalid. I found a note about this error on the wiki (http://wiki.galaxyproject.org/Admin/Running%20Tests) "Note: If you have another version of paste installed in your PYTHONPATH, it can cause problems running the functional tests. Many tests will error (instead of passing/failing) because of an HTML code 500 where a 200 was expected. There are two workarounds for this: uninstall paste or use virtualenv. " The thing is, I do not have another version of Paste installed. What can I do to fix this error? Regards, Serrano Pereira

2 5

page editor problem when saving Chinese
by lardo 18 Oct '13

18 Oct '13

I run a local instance of galaxy and edit content of a page, error occurs when the content contains Chinese: URL: http://127.0.0.1:8080/page/save Module weberror.evalexception.middleware:364 in respond >> app_iter = self.application(environ, detect_start_response) Module paste.recursive:84 in __call__ >> return self.application(environ, start_response) Module paste.httpexceptions:633 in __call__ >> return self.application(environ, start_response) Module galaxy.web.framework.base:132 in __call__ >> return self.handle_request( environ, start_response ) Module galaxy.web.framework.base:190 in handle_request >> body = method( trans, **kwargs ) Module galaxy.web.framework:98 in decorator >> return func( self, trans, *args, **kwargs ) Module galaxy.webapps.galaxy.controllers.page:566 in save >> trans.sa_session.flush() Module sqlalchemy.orm.scoping:114 in do Module sqlalchemy.orm.session:1718 in flush Module sqlalchemy.orm.session:1789 in _flush Module sqlalchemy.orm.unitofwork:331 in execute Module sqlalchemy.orm.unitofwork:475 in execute Module sqlalchemy.orm.persistence:64 in save_obj Module sqlalchemy.orm.persistence:558 in _emit_insert_statements Module sqlalchemy.engine.base:1449 in execute Module sqlalchemy.engine.base:1584 in _execute_clauseelement Module sqlalchemy.engine.base:1698 in _execute_context Module sqlalchemy.engine.base:1691 in _execute_context Module sqlalchemy.engine.default:331 in do_execute ProgrammingError: (ProgrammingError) You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings. u'INSERT INTO page_revision (create_time, update_time, page_id, title, content) VALUES (?, ?, ?, ?, ?)' ('2013-10-18 17:09:01.226991', '2013-10-18 17:09:01.227000', 1, u'test', '<p>aavdx</p><p>\xe4\xb8\xad</p>') []

2 1

Zero padding corruption using galaxy with torque on AFS
by Renato Alves 18 Oct '13

18 Oct '13

Hi everyone, I'm currently setting up galaxy to run on top of AFS using torque for handling jobs. Everything is setup according to wiki documentation but I'm having a weird filesystem corruption problem. The setup is the following: machine A: runs galaxy. Galaxy home folder is on AFS. machine B: runs torque server and shares Galaxy's home folder. When I launch a process via galaxy everything works as expected but the output file becomes corrupted with 4kb of leading zero bytes (file NC_010473.tabular). This corruption is reproducible at all times, regardless of file size. In the attached example, the original file is NC_010473.faa. If I restart the openafs client or flush the AFS file cache the corruption goes away. However, if I re-run the same script created by galaxy through torque the corruption doesn't happen. Hence it only happens if launched via Galaxy. I also tried both the DRMAA and the PBS modules but the corruption remained. Does anyone know what could be the cause of this? Thanks, Renato

1 1

file download returns empty file or webpage error
by UMD Bioinformatics 17 Oct '13

17 Oct '13

Hello, I've looked through the archives and I see that this issue has been raised before (http://dev.list.galaxyproject.org/file-download-returns-empty-file-or-webpa…) but without a noted resolution. I am running Apache as proxy but I downloads from multiple machines and browsers are either empty or return an error after clicking the download button. I enabled Apache xsend file: <Location "/"> XSendFile on XSendFileAllowAbove on XSendFilePath /galaxy </Location> And configured galaxy to be aware of this in universe_wsgi.ini apache_xsendfile = True. I have checked the apache error logs and do not see any issues reported. Any help would be appreciated. I've ended up changing apache_xsendfile = false just so that i can download data. How do I get galaxy apache and xsendfile to work properly? Cheers, Ian

1 0

RFC: remove trailing semicolons from command line -> Broken bowtie2_wrapper on some SGE systems
by Björn Grüning 17 Oct '13

17 Oct '13

Dear all, some SGE instances will add automatically an semicolon add the end of each command, resulting in a disrupted job, because ';;' are not allowed. The latest changes to the Bowtie2 wrapper resulting in such a case and a crash in our instance. An easy solution would be to fix the wrapper and omitting the trailing ';' but maybe its better to fix it once for all in the corresponding tools code, patched attached. I'm not familiar with other Job schedulers so I'm seeking for comments ... it there any disadvantage of removing trailing ';' from the command line? Ciao, Bjoern

6 18

Re: [galaxy-dev] Job splitting
by Bjoern Gruening 17 Oct '13

17 Oct '13

Hi Peter, > <bjoern.gruening(a)gmail.com> wrote: > > Hi Peter, > > > > I saw your are working again on the job splitting features. I may have a > > features request: > > Well, not really working on it - just reporting a regression using it. Ah ok :) > > For some tools you have different file formats as input but only for one > > or two of them the split and merge function are defined. If the filetype > > is now one of these non splittable ones, Galaxy will crash. I think a > > better way would be to default to non-split mode? > > I guess that isn't supported. > > > Does that make sense for you? Any pointer where to look? Its not urgent, > > but it was on my todo list. Just remembered it, reading your mail. > > Yes, that makes perfect sense. Possibly here lib/galaxy/jobs/__init__.py > it needs to look at the datatype to see that supports splitting: > > def can_split( self ): > # Should the job handler split this job up? > return self.app.config.use_tasked_jobs and self.tool.parallelism > > We could/should take this discussion to galaxy-dev, Sure, I thought it would be more easier, some missing "return" or something. Will have a deeper look at it if I have some more time. Trello card is here: https://trello.com/c/lIKKwiC1 Thanks! Bjoern > Regards, > > Peter

1 0

Deploying LOC files for tool built-in data during a tool installation
by Jean-Frédéric Berthelot 17 Oct '13

17 Oct '13

Hi list, The tool I am currently wrapping has built-in data, which may be used by the tool users (through a relevant < from_data_table> + .LOC file configuration). They are .fasta databases which are rather small and are thus bundled in the tool distribution package. Thanks to the tool_dependencies.xml file, said distribution package is downloaded at install time, code is compiled, and since they are here, the data files are copied to $INSTAL L_DIR too , ready to be used. After that, the user still has to edit tool-data/my_fancy_data_files.loc ; but the thing is, during the install I know where these data files are (since I copied those there), so I would like to save the user the trouble and set up this file automagically. I would have two questions: 1/ Is it okay to have tool built-in data files in $INSTAL L_DIR, or would it be considered bad practice? 2/ Is there a way to set up the tool-data/my_fancy_data_files.loc during the install? Here are the options I though of: *shipping a “real” my_fancy_data_files.loc.sample with the good paths already set-up, which is going to be copied as the .loc file (a rather ugly hack) *using more <action type="shell_command"> during install to create my_fancy_data_files.loc (but deploying this file it is not part of the tool dependency install per se) *variant of the previous : shipping my_fancy_data_files.loc as part of the tool distribution package, and copy it through shell_command (same concern than above). Any thoughts? Cheers, -- Jean-Frédéric Bonsai Bioinformatics group

5 10

Problem configuring Galaxy with an Apache proxy
by Erwan Delage 17 Oct '13

17 Oct '13

Hello everyone, I'm having trouble setting up Galaxy with an Apache Proxy. I did edit the Apache conf file with the following lines : RewriteEngine on RewriteRule ^/static/style/(.*)/home/nate/galaxy-dist/static/june_2007_style/blue/$1 [L] RewriteRule ^/static/scripts/(.*)/home/nate/galaxy-dist/static/scripts/packed/$1 [L] RewriteRule ^/static/(.*)/home/nate/galaxy-dist/static/$1 [L] RewriteRule ^/favicon.ico/home/nate/galaxy-dist/static/favicon.ico [L] RewriteRule ^/robots.txt/home/nate/galaxy-dist/static/robots.txt [L] RewriteRule ^(.*) http://localhost:8080$1 [P] The redirection seems to "work" as I reach a webpage that contains some Galaxy's links but, as you can see in the attached snapshot, the webpage does not really looks like the usual Galaxy welcome page :) Did you already face that problem or do you have any idea to solve it ? Thanks, Erwan, -- Erwan Delage Ingénieur d'études - Equipe COMBI LINA - Université de Nantes Tel : 02 51 12 58 21

3 2

bug in handling sessions
by Nikolai Vazov 17 Oct '13

17 Oct '13

Hi, I have discovered the following weird Galaxy behaviour. If one sets log_actions = True in universe_wsgi.ini and tries to display the Saved Histories, then the Saved histories don't show up. The problem arises from ../lib/galaxy/web/framework/helpers/grids.py line (around 230) trans.log_action( trans.get_user(), unicode( "grid.view" ), context, params) Until this line, the Query object 'query' has queried the DB and fetched the necessary data. Yet, in the file ../lib/galaxy/web/framework/__init__.py in the function log_action self.sa_session.flush() seems to flush everything, including the data fetched by the 'query' object in grids.py Thus, the Saved Histories get "No Items" message. Commenting out the line log_actions = True does the job, but ... Any suggestions? Best regards Nikolay -- Nikolay Vazov, PhD Research Computing Centre - http://hpc.uio.no USIT, University of Oslo

2 2

Re: [galaxy-dev] Tabular data not displaying in main panel
by Ian Misner 17 Oct '13

17 Oct '13

Hello All, I'm having the same issue with tabular data not displaying,but I'm afraid I'm much newer to running a local galaxy instance. Apparently switching to https://bitbucket.org/galaxy/galaxy-central/src/a477486bf18eafdd14dd7ba1e91… but frankly I don't know how to do that. Any help would be appreciated. Here is my current branch information changeset: 10421:a477486bf18e branch: stable tag: tip user: Nate Coraor <nate(a)bx.psu.edu> date: Thu Sep 26 11:02:58 2013 -0400 summary: Bugfix for tool-to-destination mapping, tool ids are lowercased but the mapping id was not lowercased. changeset: 10411:c42567f43aa7 user: greg date: Mon Aug 19 13:19:56 2013 -0400 summary: Filter invalid objects when generating the list of repository_dependencies objects that are associated with a tool shed repository installed into Galaxy. Cheers Ian

3 2