August 2013 - galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] Galaxy Release Cycle Length
by Leon Mei 21 Aug '13

21 Aug '13

Hi Dave, I am one of the requesters on a less frequent release. Certainly this does not need to replace the current release scheme in the development branch: Galaxy system users who prefer keeping their code up to date can still go there. It is just nice to have a separate branch with a less frequent stable release cycle with the support of fixing major bugs as Hans-Rudolf pointed out. This fixing support is only needed on the latest stable release. Regarding the frequency of such a release, I would say it is a bit dependent on the speed of introducing new major features in Galaxy. I wouldn't mind to have a 4 or 6 month release cycle in the stable branch if they synchronize better with the major milestones in Galaxy. Thanks, Leon Message: 3 Date: Wed, 21 Aug 2013 08:57:22 +0200 From: Hans-Rudolf Hotz <hrh(a)fmi.ch> To: Dave Clements <clements(a)galaxyproject.org> Cc: Galaxy Dev List <galaxy-dev(a)bx.psu.edu> Subject: Re: [galaxy-dev] Galaxy Release Cycle Length Message-ID: <521464D2.5010807(a)fmi.ch> Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Hi Dave two months is a good time between releases. Much more important than the release cycle length is fixing identified bugs on the release branch as well. Regards, Hans-Rudolf On 08/20/2013 08:36 PM, Dave Clements wrote: > Hello all, > > At one of the GCC2013 Birds of a Feather sessions > <http://wiki.galaxyproject.org/Events/GCC2013/BoF/PublicGalaxyServers> the > group was very clear that they would like to see less frequent releases > of Galaxy. We're currently aiming to do a release every 2 months and > have been pretty successful at making that target. In the past, we have > tried doing releases more often and less often. > > Is there a sweet spot for the time between releases? > > Please reply to the group. We are interested in a discussion. > > Thanks, > > Dave C -- Hailiang (Leon) Mei Netherlands Bioinformatics Center BioAssist NGS Taskforce - http://ngs.nbic.nl<https://wiki.nbic.nl/index.php/Next_Generation_Sequencing> Skype: leon_mei Mobile: +31 6 41709231

1 0

Galaxy Release Cycle Length
by Dave Clements 21 Aug '13

21 Aug '13

Hello all, At one of the GCC2013 Birds of a Feather sessions<http://wiki.galaxyproject.org/Events/GCC2013/BoF/PublicGalaxyServers> the group was very clear that they would like to see less frequent releases of Galaxy. We're currently aiming to do a release every 2 months and have been pretty successful at making that target. In the past, we have tried doing releases more often and less often. Is there a sweet spot for the time between releases? Please reply to the group. We are interested in a discussion. Thanks, Dave C -- http://galaxyproject.org/GCC2013 http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ http://wiki.galaxyproject.org/

5 4

error installing dependency for tool shed repository.
by Curt Palm 21 Aug '13

21 Aug '13

I've just created a repository in the sandbox and have been trying to test the installation of a dependency my tool needs. The tool needs the python module "requests" and I've written code. I get an error: STDERR python: can't open file 'setup.py': [Errno 2] No such file or directory I've modeled my code on examples I've found in the tool shed. I've replaced the requests module with other python modules and these are installed ok. I can also download and manually install requests with "python setup.py install" Any suggestions as to what I may be doing wrong? tool_dependencies.xml: <?xml version="1.0"?> <tool_dependency> <package name="requests" version="1.2.3"> <install version="1.0"> <actions> <action type="download_by_url">http://pypi.python.org/packages/source/r/requests/requests-1.2.3.tar.gz</action> <action type="make_directory">$INSTALL_DIR/lib/python</action> <action type="make_directory">$INSTALL_DIR/bin</action> <action type="set_environment"> <environment_variable name="PYTHONPATH" action="append_to">$INSTALL_DIR/lib/python</environment_variable> <environment_variable name="PATH" action="prepend_to">$INSTALL_DIR/bin</environment_variable> </action> <action type="shell_command">python setup.py install --home $INSTALL_DIR --install-scripts $INSTALL_DIR/bin</action> </actions> </install> <readme> </readme> </package> </tool_dependency> ingTest.xml: <tool id="ingTest" name="ingTest" version="1.0"> <description>Testing Tool Shed</description> <requirements> <requirement type="package" version="1.2.3">requests</requirement> </requirements> <command interpreter="python"> </command> <inputs> </inputs> <outputs> </outputs> <help> .. class:: infomark </help> </tool> Thanks, -Curt Curt Palm Data Engineer ________________________________________ INGENUITY SYSTEMS | www.ingenuity.com 1700 Seaport Blvd., 3rd Floor | Redwood City, CA 94063 650.381.5084

3 4

Bismark issue
by Shane Sturrock 21 Aug '13

21 Aug '13

I've had a bit of a time trying to get Bismark working on our Galaxy instance. The install doesn't work because of the dependency on samtools 0.1.18 but in fact it requires samtools 0.1.19. I have a build of that installed as standard on our system so putting a symlink to that in the place of the samtools built for the bismark wrapper makes it work - I was seeing issues where the BAM file wasn't created and noticed in Bjoern's readme.rst that said it required samtools 0.1.19 or later. Shane -- For urgent cases, contact support(a)biomatters.com Dr Shane Sturrock shane.sturrock(a)biomatters.com Senior Scientist Tel: +64 (0) 9 379 5064 76 Anzac Ave Auckland New Zealand

3 3

Action type 'template_command' on galaxy-dist?
by Peter Cock 21 Aug '13

21 Aug '13

Hi all, I'm currently trying out installation of tools from the Tool Shed on a secondary Galaxy server (not our production server nor my development setup). This is running galaxy-dist, current latest stable I believe: $ sudo -u galaxy hg heads changeset: 10201:ebe87051fadf tag: tip parent: 10199:8bf64d933704 user: Dannon Baker <dannonbaker(a)me.com> date: Tue Jul 02 10:48:31 2013 -0400 summary: Fix two more downgrade invocations to accept the migrate_engine parameter changeset: 10200:fd4113962c32 branch: stable parent: 10198:0a06df7da177 user: Dannon Baker <dannonbaker(a)me.com> date: Tue Jul 02 10:48:31 2013 -0400 summary: Fix two more downgrade invocations to accept the migrate_engine parameter I'm having some problems, e.g. Tool shed repository 'package_numpy_1_7' tool dependencies Name Version Type Status Error atlas 3.10.1 package Error Error installing tool dependency atlas version 3.10.1: [Errno 2] No such file or directory: u'/opt/galaxy-dist-shed-tools/atlas/3.10.1/iuc/package_atlas_3_10/597c0e54eded/env.sh' lapack 3.4.2 package Error Error installing tool dependency lapack version 3.4.2: [Errno 2] No such file or directory: u'/opt/galaxy-dist-shed-tools/lapack/3.4.2/iuc/package_lapack_3_4/98a7d116c15f/env.sh' numpy 1.7.1 package Error Error installing tool dependency numpy version 1.7.1: Unsupported action type 'template_command' in tool dependency definition. Is this due to to galaxy-dist being too old for Tool Shed tools whose XML installation scripts use action template_command? e.g. http://toolshed.g2.bx.psu.edu/view/iuc/package_numpy_1_7/c75482be1d3a One way to catch this kind of error would be if the nightly Tool Shed tests included attempting to install package repositories (e.g. the new "Tool dependency definition") explicitly. (I think I've suggested this before - but it is worth reiterating) Thanks, Peter

4 28

Re: [galaxy-dev] Duplicated entries in "Manage installed tool shed repositories" table
by Peter Cock 21 Aug '13

21 Aug '13

On Wed, Aug 14, 2013 at 11:08 PM, Ross <ross.lazarus(a)gmail.com> wrote: > FWIW: I've managed to produce a few of these too - generally arose after > futzing with a failed dependency, reaching for the "nuke it" ( 'completely > remove' tool) button, then reinstalling it - sometimes there's a complaint > that the same version was previously installed (even though it had been > supposedly totally nuked), sometimes not - but occasionally a white 'new' > ghost will reappear in the manage installed tools list after reinstallation > - AFAIK those white ghosts are impossible to remove... > > Thanks Ross, It is good to know this isn't an isolated problem - and I too had been "nuking" failed installs, starting with the July 2013 galaxy-dist but also the August galaxy-dist. Maybe that will help Dave reproduce it? I like your name "white ghost" for this since the "new" entry is shown on with a white background (it didn't show up in the HTML email sadly). Peter

3 4

Embedding a help text in a Workflow?
by Peter Cock 21 Aug '13

21 Aug '13

Hello all, I've recently started sharing some sample workflows on the Galaxy Tool Shed, http://toolshed.g2.bx.psu.edu/view/peterjc/rxlr_venn_workflow http://toolshed.g2.bx.psu.edu/view/peterjc/secreted_protein_workflow When writing a Galaxy tool we have a lot of flexibility within the <help> tag to describe the tool, its output, and relevant references using some nice markup language for things like bold fonts and tables. I would very much like something similar for embedding README or help text into a workflow - which again should be shown to the user, both on the 'View workflow action' but also under the "Run workflow" / "Execute" button when actually running a workflow. Am I overlooking something or is the only workflow level description the plain text 'Annotation / Notes' field? Thanks, Peter P.S. I'm not talking about the repository level README text which I can use for a workflow repository - that is only really seen by the Galaxy administrator (not by the users within the Galaxy web-interface) and is not specific to each workflow.

4 8

JSON error trying to update Tool Shed workflow
by Peter Cock 21 Aug '13

21 Aug '13

Hi all, I was trying to update this workflow repository on the Test Tool Shed, http://testtoolshed.g2.bx.psu.edu/view/peterjc/secreted_protein_workflow Attempting to upload the tar-ball appeared to fail as follows: URL: http://testtoolshed.g2.bx.psu.edu/upload/upload?repository_id=eb13e537583cd… Module galaxy.web.framework.middleware.error:149 in __call__ >> app_iter = self.application(environ, sr_checker) Module paste.debug.prints:106 in __call__ >> environ, self.app) Module paste.wsgilib:543 in intercept_output >> app_iter = application(environ, replacement_start_response) Module paste.recursive:84 in __call__ >> return self.application(environ, start_response) Module paste.httpexceptions:633 in __call__ >> return self.application(environ, start_response) Module galaxy.web.framework.base:132 in __call__ >> return self.handle_request( environ, start_response ) Module galaxy.web.framework.base:190 in handle_request >> body = method( trans, **kwargs ) Module galaxy.web.framework:98 in decorator >> return func( self, trans, *args, **kwargs ) Module galaxy.webapps.tool_shed.controllers.upload:203 in upload >> metadata_util.set_repository_metadata_due_to_new_tip( trans, repository, content_alert_str=content_alert_str, **kwd ) Module tool_shed.util.metadata_util:1850 in set_repository_metadata_due_to_new_tip >> error_message, status = set_repository_metadata( trans, repository, content_alert_str=content_alert_str, **kwd ) Module tool_shed.util.metadata_util:1779 in set_repository_metadata >> persist=False ) Module tool_shed.util.metadata_util:698 in generate_metadata_for_changeset_revision >> exported_workflow_dict = json.from_json_string( workflow_text ) Module simplejson:384 in loads Module simplejson.decoder:402 in decode Module simplejson.decoder:420 in raw_decode JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0) Confusingly the changes seem to get uploaded anyway but seems to have generated a dummy file _secreted_protein_workflow.ga (note the leading underscore) http://testtoolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=36b2c2b… Not realising this, I recreated and reupload the tarball (which worked and removed the phantom file): http://testtoolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=f32581a… The tar-ball (re)created for the upload was created using: $ tar -cf secreted_protein_workflow_v0.0.2.tar.gz README.rst repository_dependencies.xml secreted_protein_workflow.ga I don't have the first tar-ball anymore, but I also don't have any file named _secreted_protein_workflow.ga with a leading underscore in my working directory so I am pretty sure this was created by the Tool Shed somehow. Peter

2 2

Non-ascii characters in password break Galaxy login
by Peter Cock 21 Aug '13

21 Aug '13

Hi all, I've seen several unicode issues reported recently, so here is another one: I just tried to log in and naively copy and pasted my password from one browser to another: •••••••• This upset Galaxy, 143.234.100.162 - - [21/Aug/2013:15:47:55 +0100] "POST /galaxy/user/login?use_panels=False HTTP/1.1" 500 - "http://xxx/galaxy/user/login" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; .NET4.0E)" Error - <type 'exceptions.UnicodeEncodeError'>: 'ascii' codec can't encode characters in position 0-7: ordinal not in range(128) URL: http://ppserver/galaxy/user/login?use_panels=False File '/mnt/galaxy/galaxy-dist/lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ app_iter = self.application(environ, sr_checker) File '/mnt/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/mnt/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/mnt/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 132 in __call__ return self.handle_request( environ, start_response ) File '/mnt/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 190 in handle_request body = method( trans, **kwargs ) File '/mnt/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/user.py', line 438 in login message, status, user, success = self.__validate_login( trans, **kwd ) File '/mnt/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/user.py', line 485 in __validate_login elif not user.check_password( password ): File '/mnt/galaxy/galaxy-dist/lib/galaxy/model/__init__.py', line 100 in check_password return galaxy.security.passwords.check_password( cleartext, self.password ) File '/mnt/galaxy/galaxy-dist/lib/galaxy/security/passwords.py', line 30 in check_password if hashlib.sha1( guess ).hexdigest() == hashed: UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-7: ordinal not in range(128) Peter

1 0

Workflow import is not working, KeyError: 'header_lines'
by Peter Cock 21 Aug '13

21 Aug '13

Hi Dannon, I'm seeing a problem which seems very similar to the one Bjoern had back in June with the same workflow (which appeared to have been fixed according to the old IUC thread "Workflow import is not working"). This server is running the current galaxy-dist release, $ sudo -u galaxy hg head changeset: 10411:c42567f43aa7 tag: tip user: greg date: Mon Aug 19 13:19:56 2013 -0400 summary: Filter invalid objects when generating the list of repository_dependencies objects that are associated with a tool shed repository installed into Galaxy. changeset: 10408:6822f41bc9bb branch: stable parent: 10393:d05bf67aefa6 user: Dave Bouvier <dave(a)bx.psu.edu> date: Mon Aug 19 13:06:17 2013 -0400 summary: Fix for case where running functional tests might overwrite certain files in database/files. I have just installed this workflow repository with its dependencies (some of where I had previously installed via the Tool Shed). Most of the tools were actually already manually installed but have been commented out in my tool_conf.xml file: http://toolshed.g2.bx.psu.edu/view/peterjc/secreted_protein_workflow/606da4… Installation appeared to be successful, so I wanted to import the workflow into Galaxy. I clicked on the workflow and expected to see an SVG of the workflow and the import action to be available. In the browser I saw: <quote> Find secreted proteins with TMHMM and SignalP (this page displays SVG graphics) Internal Server Error Galaxy was unable to sucessfully complete your request An error occurred. This may be an intermittent problem due to load or other unpredictable factors, reloading the page may address the problem. The error has been logged to our team. </quote> On the terminal I have: 143.234.97.120 - - [21/Aug/2013:11:14:18 +0100] "GET /galaxy/admin_toolshed/browse_repositories HTTP/1.1" 200 - "http://ppcollab/galaxy/admin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:22.0) Gecko/20100101 Firefox/22.0" 143.234.97.120 - - [21/Aug/2013:11:14:26 +0100] "GET /galaxy/admin_toolshed/browse_repositories?async=false&sort=name&page=1&show_item_checkboxes=false&operation=manage_repository&id=eede92f8a72bf85d&f-deleted=False HTTP/1.1" 302 - "http://ppcollab/galaxy/admin_toolshed/browse_repositories" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:22.0) Gecko/20100101 Firefox/22.0" 143.234.97.120 - - [21/Aug/2013:11:14:26 +0100] "GET /galaxy/admin_toolshed/manage_repository?sort=name&f-deleted=False&async=false&show_item_checkboxes=false&id=eede92f8a72bf85d&page=1 HTTP/1.1" 200 - "http://ppcollab/galaxy/admin_toolshed/browse_repositories" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:22.0) Gecko/20100101 Firefox/22.0" 143.234.97.120 - - [21/Aug/2013:11:14:40 +0100] "GET /galaxy/admin_toolshed/view_workflow?repository_id=eede92f8a72bf85d&workflow_name=eef2c509eb9672ca81f4797561cfd5804652c810%3A46696e642073656372657465642070726f7465696e73207769746820544d484d4d20616e64205369676e616c50 HTTP/1.1" 200 - "http://ppcollab/galaxy/admin_toolshed/manage_repository?sort=name&f-deleted…" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:22.0) Gecko/20100101 Firefox/22.0" 143.234.97.120 - - [21/Aug/2013:11:14:41 +0100] "GET /galaxy/admin_toolshed/generate_workflow_image?repository_id=eede92f8a72bf85d&workflow_name=eef2c509eb9672ca81f4797561cfd5804652c810%3A46696e642073656372657465642070726f7465696e73207769746820544d484d4d20616e64205369676e616c50 HTTP/1.1" 500 - "http://ppcollab/galaxy/admin_toolshed/view_workflow?repository_id=eede92f8a…" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:22.0) Gecko/20100101 Firefox/22.0" Error - <type 'exceptions.KeyError'>: 'header_lines' URL: http://ppcollab/galaxy/admin_toolshed/generate_workflow_image?repository_id… File '/opt/galaxy-dist/lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ app_iter = self.application(environ, sr_checker) File '/opt/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/opt/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/opt/galaxy-dist/lib/galaxy/web/framework/base.py', line 132 in __call__ return self.handle_request( environ, start_response ) File '/opt/galaxy-dist/lib/galaxy/web/framework/base.py', line 190 in handle_request body = method( trans, **kwargs ) File '/opt/galaxy-dist/lib/galaxy/web/framework/__init__.py', line 221 in decorator return func( self, trans, *args, **kwargs ) File '/opt/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py', line 310 in generate_workflow_image return workflow_util.generate_workflow_image( trans, workflow_name, repository_metadata_id=None, repository_id=repository_id ) File '/opt/galaxy-dist/lib/tool_shed/util/workflow_util.py', line 215 in generate_workflow_image module_data_inputs = get_workflow_data_inputs( step, module ) File '/opt/galaxy-dist/lib/tool_shed/util/workflow_util.py', line 317 in get_workflow_data_inputs return module.get_data_inputs() File '/opt/galaxy-dist/lib/tool_shed/util/workflow_util.py', line 111 in get_data_inputs galaxy.tools.parameters.visit_input_values( self.tool.inputs, self.state.inputs, callback ) File '/opt/galaxy-dist/lib/galaxy/tools/parameters/__init__.py', line 37 in visit_input_values input_values[input.name], KeyError: 'header_lines' CGI Variables ------------- CONTENT_LENGTH: '0' HTTP_ACCEPT: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' HTTP_ACCEPT_ENCODING: 'gzip, deflate' HTTP_ACCEPT_LANGUAGE: 'en-gb,en;q=0.7,en-us;q=0.3' HTTP_CONNECTION: 'Keep-Alive' HTTP_COOKIE: 'galaxysession=fed9265fae01dfd8cf547ef38763dabd7ff6f6411d862840f8437fd67320a093efca06cca1fc50e9' HTTP_DNT: '1' HTTP_HOST: 'ppcollab' HTTP_REFERER: 'http://ppcollab/galaxy/admin_toolshed/view_workflow?repository_id=eede92f8a…' HTTP_USER_AGENT: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:22.0) Gecko/20100101 Firefox/22.0' PATH_INFO: '/admin_toolshed/generate_workflow_image' QUERY_STRING: 'repository_id=eede92f8a72bf85d&workflow_name=eef2c509eb9672ca81f4797561cfd5804652c810%3A46696e642073656372657465642070726f7465696e73207769746820544d484d4d20616e64205369676e616c50' REMOTE_ADDR: '143.234.97.120' REQUEST_METHOD: 'GET' SCRIPT_NAME: '/galaxy' SERVER_NAME: 'XXX' SERVER_PORT: '8080' SERVER_PROTOCOL: 'HTTP/1.1' WSGI Variables -------------- application: <paste.recursive.RecursiveMiddleware object at 0xb3994d0> is_api_request: False paste.cookies: (<SimpleCookie: galaxysession='fed9265fae01dfd8cf547ef38763dabd7ff6f6411d862840f8437fd67320a093efca06cca1fc50e9'>, 'galaxysession=fed9265fae01dfd8cf547ef38763dabd7ff6f6411d862840f8437fd67320a093efca06cca1fc50e9') paste.expected_exceptions: [<class 'paste.httpexceptions.HTTPException'>] paste.httpexceptions: <paste.httpexceptions.HTTPExceptionHandler object at 0xb399490> paste.httpserver.thread_pool: <paste.httpserver.ThreadPool object at 0x53ce0d0> paste.parsed_querystring: ([('repository_id', 'eede92f8a72bf85d'), ('workflow_name', 'eef2c509eb9672ca81f4797561cfd5804652c810:46696e642073656372657465642070726f7465696e73207769746820544d484d4d20616e64205369676e616c50')], 'repository_id=eede92f8a72bf85d&workflow_name=eef2c509eb9672ca81f4797561cfd5804652c810%3A46696e642073656372657465642070726f7465696e73207769746820544d484d4d20616e64205369676e616c50') paste.recursive.forward: <paste.recursive.Forwarder from /galaxy> paste.recursive.include: <paste.recursive.Includer from /galaxy> paste.recursive.include_app_iter: <paste.recursive.IncluderAppIter from /galaxy> paste.recursive.script_name: '/galaxy' paste.throw_errors: True request_id: '7e2437a60a4a11e3b07b000c290c0d9f' webob._parsed_query_vars: (MultiDict([('repository_id', 'eede92f8a72bf85d'), ('workflow_name', 'eef2c509eb9672ca81f4797561cfd5804652c810:46696e642073656372657465642070726f7465696e73207769746820544d484d4d20616e64205369676e616c50')]), 'repository_id=eede92f8a72bf85d&workflow_name=eef2c509eb9672ca81f4797561cfd5804652c810%3A46696e642073656372657465642070726f7465696e73207769746820544d484d4d20616e64205369676e616c50') wsgi process: 'Multithreaded' ------------------------------------------------------------ (My intention was to update this workflow to use Tool Shed style tool identifiers, instead of the current Tool identifiers is has now from manually installed versions of the tools). Regards, Peter

2 6