- galaxy-dev - lists.galaxyproject.org

set_metadata.py
by Bram Slabbinck 08 Feb '12

08 Feb '12

Hi I am running into a problem with the scripts/set_metadata.py file since the last update of galaxy-dist. In this script the name of the universe file is hard-coded as 'universe_wsgi.ini', see line 44: config_file_name = 'universe_wsgi.ini' # Safe assumption? I renamed the universe file in our setting, due to different Galaxy installs. Running the set_metadata.py script crashes however with this coding as it is constantly trying to make a tmpfile in './database/tmp' which is the default tmp path in universe_wsgi.ini. Replacing 'universe_wsgi.ini' by the name of our universe file solved the problem. So, how can I dynamically retrieve the name of the launched universe file? Thx for any help regards Bram -- ========================================================== Bram Slabbinck, PhD Bioinformatics& Systems Biology Division VIB Department of Plant Systems Biology, UGent Technologiepark 927, 9052 Gent, BELGIUM Email: Bram.Slabbinck(a)psb.ugent.be WWW: http://bioinformatics.psb.ugent.be ========================================================== Please consider the environment before printing this email

2 1

Error when attempting to get_updates from toolshed
by Ira Cooke 08 Feb '12

08 Feb '12

Dear Greg, In the latest version of the galaxy-central code there seems to be a bug preventing me from updating my tools. When I go to update a tool I get the following error URL: http://cookie.latrobe.edu.au:8300/admin_toolshed/browse_repositories?sort=n… File '/var/www/galaxy/galaxyapp/eggs/Paste-1.6-py2.7.egg/paste/exceptions/errormiddleware.py', line 143 in __call__ app_iter = self.application(environ, start_response) File '/var/www/galaxy/galaxyapp/eggs/Paste-1.6-py2.7.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/var/www/galaxy/galaxyapp/eggs/Paste-1.6-py2.7.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/var/www/galaxy/galaxyapp/lib/galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/var/www/galaxy/galaxyapp/lib/galaxy/web/framework/__init__.py', line 173 in decorator return func( self, trans, *args, **kwargs ) File '/var/www/galaxy/galaxyapp/lib/galaxy/web/controllers/admin_toolshed.py', line 96 in browse_repositories return self.check_for_updates( trans, **kwd ) File '/var/www/galaxy/galaxyapp/lib/galaxy/web/framework/__init__.py', line 173 in decorator return func( self, trans, *args, **kwargs ) File '/var/www/galaxy/galaxyapp/lib/galaxy/web/controllers/admin_toolshed.py', line 654 in check_for_updates ( tool_shed_url, url_for( '/', qualified=True ), repository.name, repository.owner, repository.changeset_revision ) ValueError: unsupported format character '/' (0x2f) at index 1 Looking at the code there seems to be a typo on line 702 of admin_toolshed.py It reads url = '%/srepository/check_for_updates?galaxy_url=%s&name=%s&owner=%s&changeset_revision=%s&webapp=galaxy' % \ But I think it should be url = '%s/repository/check_for_updates?galaxy_url=%s&name=%s&owner=%s&changeset_revision=%s&webapp=galaxy' % \ When I change this though I still get further errors ... seems like a query to the database is not finding my toolshedrepository The new error is URL: http://127.0.0.1:8300/admin_toolshed/update_to_changeset_revision?tool_shed… File '/Users/iracooke/Sources/galaxy-proteomics/eggs/WebError-0.8a-py2.7.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/Users/iracooke/Sources/galaxy-proteomics/eggs/Paste-1.6-py2.7.egg/paste/debug/prints.py', line 98 in __call__ environ, self.app) File '/Users/iracooke/Sources/galaxy-proteomics/eggs/Paste-1.6-py2.7.egg/paste/wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/Users/iracooke/Sources/galaxy-proteomics/eggs/Paste-1.6-py2.7.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/Users/iracooke/Sources/galaxy-proteomics/eggs/Paste-1.6-py2.7.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/Users/iracooke/Sources/galaxy-proteomics/lib/galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/Users/iracooke/Sources/galaxy-proteomics/lib/galaxy/web/framework/__init__.py', line 173 in decorator return func( self, trans, *args, **kwargs ) File '/Users/iracooke/Sources/galaxy-proteomics/lib/galaxy/web/controllers/admin_toolshed.py', line 772 in update_to_changeset_revision id=trans.security.encode_id( repository.id ), AttributeError: 'NoneType' object has no attribute 'id' If I browse my database I can find an entry in the tool_shed_repositories table which seems to have all the right data. I'm getting these errors on the latest version of the galaxy-central code. Ira

2 2

Galaxy Main Server Tool Repository
by Li Charlie Xia 07 Feb '12

07 Feb '12

Dear all, New bee here. Wonder if this is the right place to post following questions: We are interested in setting up a server like galaxy main server. Wonder which Linux Platform is the main server running on. Also is there a method for batch installing tools that now are available on the main galaxy server? If not, wonder if can point me to the documentation of the main galaxy server installation. Please excuse me if this is not the right list! Thanks a lot! Charlie -- Li Charlie Xia PhD Candidate Computational Biology and Bioinformatics Program at University of Southern California 1050 Childs Way, RRI 416C Los Angeles, CA 90089-2910, USA OP: 213-740-5168

3 3

Problem with latest database upgrade
by Thon Deboer 07 Feb '12

07 Feb '12

Hi, I updated to the latest galaxy-central version but got this error when trying to upgrade... Any ideas? It seems the table had already been created (this was the second time I run upgrade) but the JSON error was one that showed up the first time...) Thanks Thon sh manage_db.sh upgrade 90 -> 91... Migration script to create the tool_version and tool_version_association tables and drop the tool_id_guid_map table. 0091_add_tool_version_tables DEBUG 2012-02-06 10:23:39,847 Creating tool_version table failed: (ProgrammingError) relation "tool_version" already exists '\nCREATE TABLE tool_version (\n\tid SERIAL NOT NULL, \n\tcreate_time TIMESTAMP WITHOUT TIME ZONE, \n\tupdate_time TIMESTAMP WITHOUT TIME ZONE, \n\ttool_id VARCHAR(255), \n\ttool_shed_repository_id INTEGER, \n\tPRIMARY KEY (id), \n\t FOREIGN KEY(tool_shed_repository_id) REFERENCES tool_shed_repository (id)\n)\n\n' {} 0091_add_tool_version_tables DEBUG 2012-02-06 10:23:39,847 Creating tool_version table failed: (ProgrammingError) relation "tool_version" already exists '\nCREATE TABLE tool_version (\n\tid SERIAL NOT NULL, \n\tcreate_time TIMESTAMP WITHOUT TIME ZONE, \n\tupdate_time TIMESTAMP WITHOUT TIME ZONE, \n\ttool_id VARCHAR(255), \n\ttool_shed_repository_id INTEGER, \n\tPRIMARY KEY (id), \n\t FOREIGN KEY(tool_shed_repository_id) REFERENCES tool_shed_repository (id)\n)\n\n' {} 0091_add_tool_version_tables DEBUG 2012-02-06 10:23:39,863 Creating tool_version_association table failed: (ProgrammingError) relation "tool_version_association" already exists '\nCREATE TABLE tool_version_association (\n\tid SERIAL NOT NULL, \n\ttool_id INTEGER NOT NULL, \n\tparent_id INTEGER NOT NULL, \n\tPRIMARY KEY (id), \n\t FOREIGN KEY(tool_id) REFERENCES tool_version (id), \n\t FOREIGN KEY(parent_id) REFERENCES tool_version (id)\n)\n\n' {} 0091_add_tool_version_tables DEBUG 2012-02-06 10:23:39,863 Creating tool_version_association table failed: (ProgrammingError) relation "tool_version_association" already exists '\nCREATE TABLE tool_version_association (\n\tid SERIAL NOT NULL, \n\ttool_id INTEGER NOT NULL, \n\tparent_id INTEGER NOT NULL, \n\tPRIMARY KEY (id), \n\t FOREIGN KEY(tool_id) REFERENCES tool_version (id), \n\t FOREIGN KEY(parent_id) REFERENCES tool_version (id)\n)\n\n' {} Traceback (most recent call last): File "./scripts/manage_db.py", line 63, in <module> main( repository=repo, url=db_url ) File "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/shell.py", line 150, in main ret = command_func(**kwargs) File "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/api.py", line 221, in upgrade return _migrate(url, repository, version, upgrade=True, err=err, **opts) File "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/api.py", line 349, in _migrate schema.runchange(ver, change, changeset.step) File "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/schema.py", line 184, in runchange change.run(self.engine, step) File "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/script/py.py", line 101, in run func() File "lib/galaxy/model/migrate/versions/0091_add_tool_version_tables.py", line 78, in upgrade repository_metadata = from_json_string( str( row[1] ) ) File "/home/tdeboer/code/galaxy-central/eggs/simplejson-2.1.1-py2.6-linux-x86_64-ucs2.egg/simplejson/__init__.py", line 384, in loads File "/home/tdeboer/code/galaxy-central/eggs/simplejson-2.1.1-py2.6-linux-x86_64-ucs2.egg/simplejson/decoder.py", line 402, in decode File "/home/tdeboer/code/galaxy-central/eggs/simplejson-2.1.1-py2.6-linux-x86_64-ucs2.egg/simplejson/decoder.py", line 420, in raw_decode simplejson.decoder.JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0) Regards, Thon de Boer, Ph.D. Bioinformatics Guru +1-650-799-6839 thondeboer(a)me.com LinkedIn Profile

3 10

01/27 galaxy-dist startup problem (toolshed/pgsql related)
by Oleksandr Moskalenko 07 Feb '12

07 Feb '12

I've upgraded my developmental Galaxy instance to the latest galaxy-dist release with a transplanted the drmaa.py patch from galaxy-central. All .conf files have been updated. However, I can't start the instance now The error listed below is generated. I tried enabling/disabling the toolshed configs, but that didn't resolve the issue. The database was upgraded from version 85 to 90 cleanly. Regards, Alex galaxy.datatypes.registry DEBUG 2012-02-07 12:32:12,345 Loaded converter: pbed2ldindepconvert galaxy.tools.imp_exp DEBUG 2012-02-07 12:32:12,360 Loaded history export tool: __EXPORT_HISTORY__ galaxy.tools.imp_exp DEBUG 2012-02-07 12:32:12,367 Loaded history import tool: __IMPORT_HISTORY__ galaxy.datatypes.registry DEBUG 2012-02-07 12:32:12,383 Loaded external metadata tool: __SET_METADATA__ <Galaxy finished loading all tools> sqlalchemy.pool.QueuePool.0x...1290 WARNING 2012-02-07 12:32:12,831 Error closing cursor: current transaction is aborted, commands ignored until end of transaction block Traceback (most recent call last): File "/galaxy/run/dev/lib/galaxy/web/buildapp.py", line 82, in app_factory app = UniverseApplication( global_conf = global_conf, **kwargs ) File "/galaxy/run/dev/lib/galaxy/app.py", line 83, in __init__ self.installed_repository_manager.load_proprietary_datatypes() File "/galaxy/run/dev/lib/galaxy/tool_shed/__init__.py", line 19, in load_proprietary_datatypes .order_by( self.model.ToolShedRepository.table.c.id ): File "/galaxy/run/dev/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/query.py", line 1361, in __iter__ return self._execute_and_instances(context) File "/galaxy/run/dev/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/query.py", line 1364, in _execute_and_instances result = self.session.execute(querycontext.statement, params=self._params, mapper=self._mapper_zero_or_none()) File "/galaxy/run/dev/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/session.py", line 755, in execute clause, params or {}) File "/galaxy/run/dev/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 824, in execute return Connection.executors[c](self, object, multiparams, params) File "/galaxy/run/dev/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 874, in _execute_clauseelement return self.__execute_context(context) File "/galaxy/run/dev/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 896, in __execute_context self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context) File "/galaxy/run/dev/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 950, in _cursor_execute self._handle_dbapi_exception(e, statement, parameters, cursor, context) File "/galaxy/run/dev/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 931, in _handle_dbapi_exception raise exc.DBAPIError.instance(statement, parameters, e, connection_invalidated=is_disconnect) ProgrammingError: (ProgrammingError) column tool_shed_repository.uninstalled does not exist LINE 1: ...pository.deleted AS tool_shed_repository_deleted, tool_shed_... ^ 'SELECT tool_shed_repository.id AS tool_shed_repository_id, tool_shed_repository.create_time AS tool_shed_repository_create_time, tool_shed_repository.update_time AS tool_shed_repository_update_time, tool_shed_repository.tool_shed AS tool_shed_repository_tool_shed, tool_shed_repository.name AS tool_shed_repository_name, tool_shed_repository.description AS tool_shed_repository_description, tool_shed_repository.owner AS tool_shed_repository_owner, tool_shed_repository.installed_changeset_revision AS tool_shed_repository_installed_changeset_revision, tool_shed_repository.changeset_revision AS tool_shed_repository_changeset_revision, tool_shed_repository.metadata AS tool_shed_repository_metadata, tool_shed_repository.includes_datatypes AS tool_shed_repository_includes_datatypes, tool_shed_repository.update_available AS tool_shed_repository_update_available, tool_shed_repository.deleted AS tool_shed_repository_deleted, tool_shed_repository.uninstalled AS tool_shed_repository_uninstalled, tool_shed_repository.dist_to_shed AS tool_shed_repository_dist_to_shed \nFROM tool_shed_repository \nWHERE tool_shed_repository.includes_datatypes = %(includes_datatypes_1)s AND tool_shed_repository.deleted = %(deleted_1)s ORDER BY tool_shed_repository.id' {'includes_datatypes_1': True, 'deleted_1': False}

2 4

Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy?
by Jim Johnson 07 Feb '12

07 Feb '12

It is easiest to generate tools for galaxy when the applications or scripts can take arbitrarily named input files and generate output to given path names. Input directories, output directories are very convenient on the command line, but more of a challenge when crafting a galaxy tool. That said, many applications require a wrapper script to work with in galaxy. Thank you for the consistent script_info[] help/usage syntax in the qiime scripts, which enabled me to generate a skeleton galaxy tool_config file for each qiime script. I had some time last spring to work on integrating qiime into galaxy. Unfortunately, I haven't had any time since to work on this. I put those partial results on the Galaxy Tool Shed: http://toolshed.g2.bx.psu.edu/ There's a continuing effort at George Mason University to incorporate qiime into galaxy tools, so you may want to ask them what they need. I started by generating galaxy tool_config files, e.g. align_seqs.xml, by using python to get the script_info[] from the qiime script: $ cat generate_tool_config.bash #!/usr/bin/env bash python $1 > ${1%.*}.help cat tool_template.txt | sed "s/__TOOL_BINARY__/${1}/" | python -i $1 -h > ${1%.*}.log (I'll attach tool_template.txt ) This generated skeleton tool_config .xml files that I could then edit as needed. ( http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax ) I originally was calling all qiime scripts from a tool wrapper: qiime_wrapper.py But, if a script can be called with any input filepaths and write its results to any filepaths, and only writes to STDERR when it fails, then you could call that script directly. When should you use a tool_wrapper or call the qiime script directly? Many of the qiime scripts could probably be called directly, especially if it can be called with arbitary input/output file pathnames. The reasons for using a tool wrapper may be if input/output needs to be manipulated, moved, renamed in order to be used by the qiime script. You'll also need a tool wrapper if the names or number of the output files can not be determined from the parameter settings. ( http://wiki.g2.bx.psu.edu/Admin/Tools/Multiple%20Output%20Files ) If your tool relies on a file ext to determine a format, you'll have to rename the input. ( Galaxy dataset pathnames will look something like: /<your_galaxy_file_path>/072/dataset_72931.dat ) The format/type of a dataset is stored in its metadata, so the tool_config can use that information, especially if a script can take muliple alternative input formats. A tool_wrapper can also be used to manage the stdout or stderr from a tool. Galaxy currently interprets any output on stderr as a failure. A couple changes in galaxy should make somethings easier than when I first attempted this: - galaxy now accepts dataset requests with sub directories. ( https://bitbucket.org/galaxy/galaxy-central/issue/494/support-sub-dirs-in-e… ) That means that output HTML files with links into sub directories can be left intact, with the html copied to the output dataset and the linked files to its "extra_files_path". - if you know the pathname of an output relative to the working directory, galaxy can copy it automatically to the output dataset using the from_work_dir attribute. ( see example in: https://bitbucket.org/galaxy/galaxy-central/src/21b645303c02/tools/ngs_rna/… ) Datatypes You may want to create new datatypes to make it easier for the user to correctly select inputs to a tool from previous outputs. For example, the qiime mapping file is a tabular file with specific requirements. I put a 'qiimemapping' datatype in lib/galaxy/datatypes/metagenomics.py and datatypes_conf.xml so an input could generate a select list containing only qiimemapping datasets rather than all tabular ones. Generating a configfile You can generate configfiles in the galaxy tool_config .xml file. The configfile is generated by the Cheetah interpreter just as the commandline is. see: alpha_rarefaction.xml The qiime_wrapper.py was patterned after the mothur_wrapper.py with some of the same wrapper params to handle run time determined output (perhaps not needed): --galaxy_datasets a comma separated list of regex:output_dataset the wrapper searches the working_dir and copies the file that matches the regex to the outout dataset if the exact pathname is known, use the "from_work_dir" attribute instead --galaxy_datasetid would be an output dataset id that would be used to dynamically create additional new datasets at job termination ( http://wiki.g2.bx.psu.edu/Admin/Tools/Multiple%20Output%20Files "Number of Output datasets cannot be determined until tool run") --galaxy_new_datasets a comma separated list of regex:datatype used to dynamically create additional new datasets at job termination --galaxy_new_files_path the galaxy dir for dynamically generated output datasets On 12/14/11 12:11 PM, Jose Carlos Clemente wrote: > Hi Jim, > > we just had a meeting to discuss ideas for a QIIME GUI, and were wondering how far did you manage to get with your plan to develop a QIIME wrapper for Galaxy. Do you have something working that we could take a look at? Were there any particular issues with the integration? We might have some bandwidth to work on this over next year, but thought it would be good to check with you first. > > Thanks, > Jose > > > On Mon, Feb 7, 2011 at 20:24, Rob Knight <Rob.Knight(a)colorado.edu <mailto:Rob.Knight@colorado.edu>> wrote: > > Hi Jim, > > Just to echo Greg's comments; it would be excellent to connect QIIME to Galaxy: let me know what is necessary from my end to ensure that this happens. > > Thanks, > Rob > > On Feb 7, 2011, at 12:30 PM, Greg Caporaso wrote: > >> Hi Jim, >> This is great! We're very enthusiastic about hooking up QIIME with Galaxy, and definitely encourage you to work on it. It's something we've discussed doing, but no one in our lab is actively working on it. One particular use case that I've been thinking about would be to set up Galaxy/QIIME in our QIIME EC2 images so our EC2 users could access their images via the Galaxy web server. Are you planning to release all of the xml files under an open source/free distribution model? >> >> I'm definitely interested in progress updates. Let me know if there are ways that we can help out, for example by giving you a list of what we believe are the most commonly used QIIME scripts so you can focus your efforts on those. I'm also cc'ing some of the folks in the lab here who are interested in Galaxy integration. >> >> Greg >> >> On Mon, Feb 7, 2011 at 10:36 AM, Jim Johnson <johns198(a)umn.edu <mailto:johns198@umn.edu>> wrote: >> >> Greg, >> >> Tim te Beek let me know that you have an interest in incorporating Qiime into Galaxy. I'm currently working on that in support of a grant at the University of Minnesota. I'll keep you apprised of my progress if you are interested. >> >> Thanks, >> >> JJ >> >> James Johnson >> University of Minnesota Supercomputing Institute >> 117 Pleasant St. SE >> Minneapolis, MN 55455 >> Email: jj(a)umn.edu <mailto:jj@umn.edu> >> >> >> >> -------- Original Message -------- >> Subject: Re: [galaxy-dev] Existing efforts to convert the QIIME pipeline to Galaxy? >> Date: Mon, 07 Feb 2011 11:15:38 -0600 >> From: Jim Johnson <johns198(a)umn.edu> <mailto:johns198@umn.edu> >> Reply-To: jj(a)umn.edu <mailto:jj@umn.edu> >> Organization: MSI University of Minnesota >> To: Tim te Beek <tim.te.beek(a)nbic.nl> <mailto:tim.te.beek@nbic.nl> >> >> >> >> Tim, >> >> Sorry to hear we won't be doing this together. I'm finding some sporadic time to work on this. I also put my Mothur metagenomics wrapper code on http://community.g2.bx.psu.edu/ >> >> I looked at the initial work you did, and combined that with the script design ideas I used with Mothur. I'm writing a single script to call all the Qiime scripts, and generating a tool config (.xml) for each qiime script. Since the qiime scripts all contain a rather consistent script_info dictionary in the code, I wrote a script to generate an initial tool config from that script_info. I'm now starting the tedious process of editing those for each script. I'll attach the few I've started on. If you have any users that would able to beta test this when I have this more complete, please let me know. >> >> Thanks, >> >> JJ >> >> >> On 2/7/11 10:15 AM, Tim te Beek wrote: >>> Hi Jim, >>> >>> Sorry for the lack of feedback / updates on my part in converting QIIME to Galaxy. Following the initial difficulties with the conversion, the support project was put on hold pending further discussion. This morning the clients decided to forgo the Galaxy implementation, as their implementation would only ever be used by a single (technical) user, reducing the benefit of having a user friendly graphical interface. >>> >>> So I'm sorry to say I won't be able to contribute to the conversion, although your wrapper script did look like a viable solution to work around the unknown output file problems. Should you be able to complete the conversion yourself and be willing to share your implementation, I would still very much like to know. >>> >>> Perhaps the following could still be of some use to you: I contacted the QIIME development group about adding output file name arguments to the scripts, as can be seen here: >>> http://sourceforge.net/tracker/?func=detail&atid=1157167&aid=3164813&group_… <http://sourceforge.net/tracker/?func=detail&atid=1157167&aid=3164813&group_…> >>> Although it appears they wont be able to add output file name script parameters, they did say they're interested in Galaxy, and would be willing to help if there's a need for that. >>> >>> Additionally, we at NBIC do have a team that's more experienced at using Galaxy and converting tools to work in Galaxy, and if you want I could put you in contact with them about supporting this effort. In informal discussions they did seem to look favorable upon your solution, so perhaps that could be extended into a collaboration. >>> >>> Best regards, >>> Tim >>> >>> On Fri, Jan 14, 2011 at 20:01, Tim te Beek <tim.te.beek(a)nbic.nl <mailto:tim.te.beek@nbic.nl>> wrote: >>> >>> Ah ok, that's still fine. Thanks for the archive! I'll see what I can learn from how you've handled running command and handling in- & output for the non-trivial scripts, and apply that to the QIIME scripts. >>> Should you wish so I could get back to you if there's any significant new developments on my part in moving the QIIME scripts to Galaxy, other than the Trac / SVN changes. >>> >>> Have a good weekend! >>> Tim >>> >>> >>> On Fri, Jan 14, 2011 at 19:12, Jim Johnson <johns198(a)umn.edu <mailto:johns198@umn.edu>> wrote: >>> >>> I should probably be more clear, in that mothur and qiime overlap in functionality and in the algorithms they use. >>> They do use different data formats. >>> >>> >>> >>> On 1/14/11 12:02 PM, Tim te Beek wrote: >>>> Hi Jim, >>>> >>>> If you could send me the Mothur package, that would be great. If there's indeed as much overlap then it could be a great way to kick-start the QIIME pipeline conversion development. I can look into it starting monday, with hopefully nothing else to distract me the whole week. >>>> >>>> Best regards, >>>> Tim >>>> >>>> >>>> On Fri, Jan 14, 2011 at 18:49, Jim Johnson <johns198(a)umn.edu <mailto:johns198@umn.edu>> wrote: >>>> >>>> Tim, >>>> >>>> I just started looking at qiime myself. We are hoping to release a galaxy server to the University of Minnesota researchers in the next month, >>>> so there has been much to do in preparation. I'll take a look at what you've done. Thanks. >>>> >>>> I finished a galaxy interface to the Mothur metagenomics package http://www.mothur.org/wiki/Main_Page. It appears to have a lot of overlap with Qiime. >>>> I generated a number of datatypes, and made use of many of the tool config concepts that are available in galaxy. >>>> If you are interested, I can send that to you: mothur.tar.gz (55108 bytes). I hope to submit it tohttp://community.g2.bx.psu.edu/ in the next month. >>>> >>>> Regards, >>>> >>>> JJ >>>> >>>> >>>> On 1/14/11 9:52 AM, Tim te Beek wrote: >>>>> Hi Jim, >>>>> >>>>> Sorry to only respond just now: I've been tied up in other projects, so have not been able to get any work done on this up to today. >>>>> >>>>> What I currently have is not much, since my progress has been slowed by my unfamiliarity with both Galaxy and the QIIME tools (I'm working on this as an outsider for a small support project). But what I do have I've made available through the following Trac / SVN instance: >>>>> _ >>>>> _ >>>>> >>>>> https://trac.nbic.nl/brs2010p26/browser/trunk/galaxy_dist/tools/qiime >>>>> >>>>> >>>>> These configuration files so far can run: >>>>> >>>>> * check_id_map >>>>> * split_libraries >>>>> * pick_otus_through_otu_table >>>>> >>>>> For the first two the output is correctly directed to the Galaxy history, the last one generates quite a large set of output files which might have to be made available otherwise (the documentation on this gives a few options: https://bitbucket.org/galaxy/galaxy-central/wiki/ToolsMultipleOutput)) >>>>> >>>>> I had some trouble working around the fact that most QIIME scripts do not allow one to specify output filenames on the command line, which seems to be a requirement for Galaxy to import files into the history pane. What I've done is written a bash file to accompany each XML file, that expects as first command line argument the command that should be evaluated, and as additional arguments the output filenames expected by Galaxy. After running a tool I then move whatever output files were created to the paths expected by Galaxy. I must say I'm not (at all) sure if this current approach is something to recommend, or if better alternatives are available. Any feedback on this would be highly appreciated. >>>>> An additional source of pain was that the split_libraries script requires specific file extensions for it's fasta & quality files, which required passing them separately and symbolic linking them as .fna & .qual files. >>>>> >>>>> So far I've not taken the time to introduce new QIIME specific data types, but this could be something to consider given the large amount of 'tabular' input and output files. >>>>> >>>>> I hope this email and the scripts linked above can still be of some use to you (or vice-versa), but in either case any feedback or collaboration would be much appreciated. >>>>> >>>>> Best regards, >>>>> Tim >>>>> >>>>> >>>>> On Wed, Nov 24, 2010 at 15:48, Jim Johnson <ctoftey(a)comcast.net <mailto:ctoftey@comcast.net>> wrote: >>>>> > Tim, >>>>> > >>>>> > I hope to also look at incorporating QIIME into our local Galaxy instance at >>>>> > the University of Minnesota, but probably won't be able to start for a >>>>> > couple weeks. It would be good to develop that in coordination with others. >>>>> > >>>>> > I just finished incorporating "Mothur" metagenomics suite >>>>> > http://www.mothur.org/ (Dr. Patrick Schloss, Department of Microbiology & >>>>> > Immunology at The University of Michigan) into our Galaxy server at the >>>>> > University of Minnesota. I hope to contribute that to >>>>> > http://community.g2.bx.psu.edu/ after some testing by our researchers. If >>>>> > the Galaxy wrappings for Mothur are of any interest to you, I can send you a >>>>> > copy any time. >>>>> > >>>>> > Thanks, >>>>> > >>>>> > JJ >>>>> > >>>>> > James E Johnson >>>>> > Minnesota Supercomputing Institute >>>>> > University of Minnesota >>>>> > >>>>> > >>>>> > On Nov 23, 2010 at 07:22 AM, Tim te Beek wrote: >>>>> > >>>>> >> Hello all, >>>>> >> >>>>> >> Is anyone aware of any existing efforts to port the QIIME sequencing >>>>> >> pipeline (http://qiime.sourceforge.net/) to Galaxy? I would like to run >>>>> >> QIIME analyses through Galaxy to get better control of >>>>> >> intermediate processing steps, but before I start to convert (a subset of) >>>>> >> some 90 scripts, I'd first like to make sure this has not been done before >>>>> >> by anyone willing to share their work. >>>>> >> >>>>> >> So: has anyone converted the QIIME pipeline to Galaxy before, and would >>>>> >> they >>>>> >> be willing to share their scripts? >>>>> >> >>>>> >> Best regards, >>>>> >> Tim >>>>> > >>>>> > _______________________________________________ >>>>> > galaxy-dev mailing list >>>>> > galaxy-dev(a)lists.bx.psu.edu <mailto:galaxy-dev@lists.bx.psu.edu> >>>>> > http://lists.bx.psu.edu/listinfo/galaxy-dev >>>>> > >>>>> >>>> >>> >>> >>> >> >> >> > >

8 13

More helpful cluster job names
by Peter Cock 07 Feb '12

07 Feb '12

Hello all, We're not planning to try the new "run cluster jobs as user" functionality (at least, not just yet), so for now all our SGE jobs show on the cluster via qstat or any other monitoring tool as belonging to the user "galaxy". Currently the jobs are submitted as shell scripts "galaxy_NNN.sh" where NNN is the Galaxy job number. This is almost no help at all for a cluster administrator to use for monitoring or diagnostics. Given qstat truncates the command to 10 letters anyway, I typically see things like "galaxy_42." for "galaxy_42.sh" so once job numbers exceed three digits it becomes ambiguous. To me there are three things that could be done with the script name which would make it far more instructive in this regard. First, move the job number nearer the start. I'd find just "gNNN.sh" more useful than "galaxy_NNN.sh" simply because of the truncation in qstat. Second, unless configured to "run jobs as the user", then include the user name. I would suggest the Galaxy user's "Public Name" (which Galaxy should ensure avoids potential problem characters in filenames) or perhaps the pre- domain part of their email address? e.g. "gNNN_USER.sh" Third, the Galaxy tool ID (which will contain dashes and underscores, but no spaces etc). e.g. "gNNN_TOOL.sh" Even with truncation, this would make it easy to tell NCBI BLAST jobs apart from TopHat or whatever. Obviously using both, e.g. "gNNN_USER_TOOL.sh" wouldn't work so well with the truncation in qstat, but you should still see enough of the username to make it informative. Does this make sense? It would seem to be a small tweak needed in lib/galaxy/jobs/runners/drmma.py where there is currently also a hard coded path bug: https://bitbucket.org/galaxy/galaxy-central/issue/715/ Regards, Peter

3 5

Re: [galaxy-dev] Postgresql To Galaxy
by Nate Coraor 07 Feb '12

07 Feb '12

On Feb 7, 2012, at 6:28 AM, Rehan Saleem wrote: > Hi nate , > thanks for reply , actually i want to know how can i add my own tool in galaxy , how can i show the output of my postgreSQL function in galaxy just asume my postgresql database is connected to galaxy . i need the python and xml code for that . > thanks Hi Rehan, Your database shouldn't be connected to Galaxy, it should be connected to your tool, and then your tool will write the results as the tool's output, which will then be available in Galaxy. The documentation on how to write the tool config XML is in the wiki at: http://wiki.g2.bx.psu.edu/Admin/Tools/Add%20Tool%20Tutorial http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax I'd suggest using one of the existing tool config files in Galaxy as a reference. As for the script which queries the database, that would be a new tool to write. Two of our tools use sqlite, which may be useful as a reference when writing your tool: tools/next_gen_conversion/solid2fastq.py tools/taxonomy/find_diag_hits.py You are not limited to using Python to write the tool, however, it can be any language which you can run on the command line. --nate > > From: Nate Coraor <nate(a)bx.psu.edu> > To: Rehan Saleem <pk_rehan(a)yahoo.com> > Cc: "galaxy-dev(a)lists.bx.psu.edu" <galaxy-dev(a)bx.psu.edu> > Sent: Monday, February 6, 2012 10:19 PM > Subject: Re: [galaxy-dev] Postgresql To Galaxy > > On Feb 6, 2012, at 10:56 AM, Rehan Saleem wrote: > > > Hi Nate, > > well i don’t want to use galaxy default database , i want to set my own tools for example, i have created a function totaloverlapcount and its work perfectly fine in my separate postgresql database called ova , now i want to add a tool in galaxy with a name Total Overlap Count , as we know whenever we want to add a tool we configure it in tool_config file. > > Now if a user click on Total Overlap Count then my postgresql function totaloverlapcount should be called , and that function will perform query on my own created database(ova) and then return the result into galaxy interface > > Hi Rehan, > > Please use 'reply all' to keep replies on the mailing list. > > Since your tool already works with a separate postgres database outside of Galaxy, you should be able to add the tool to Galaxy without modification. I presume the tool has some means of knowing what host, user and port to connect to, and the setup of this information is the same as it would be when running it on the command line (via a config file, environment, hardcoded in the tool...?). > > --nate > > > > > From: Nate Coraor <nate(a)bx.psu.edu> > > To: Rehan Saleem <pk_rehan(a)yahoo.com> > > Cc: "galaxy-dev(a)lists.bx.psu.edu" <galaxy-dev(a)lists.bx.psu.edu> > > Sent: Monday, February 6, 2012 8:07 PM > > Subject: Re: [galaxy-dev] Postgresql To Galaxy > > > > On Feb 6, 2012, at 6:37 AM, Rehan Saleem wrote: > > > > > Hi , > > > i have created postgreSQL database where i have created all functions and views which performs annotations and some other functions , how can i show my postgresql database table to galaxyas output . i have connected my database from universe_wsgi.ini to my linux host where i am running galaxy and my database is on windows host, i need the python script which take values from table and show them back to galaxy. > > > THANKS > > > > Hi Rehan, > > > > I am a little unsure what you're trying to do. Do you want to display contents of this database in some sort of display within Galaxy, use them as tool inputs, or outputs? > > > > I would not suggest using this custom database as Galaxy's database (set in database_connection) if it contains separate data from the Galaxy application. Galaxy should live in its own database away from any other tables you use for other purposes. > > > > --nate > > > > > ___________________________________________________________ > > > Please keep all replies on the list by using "reply all" > > > in your mail client. To manage your subscriptions to this > > > and other Galaxy lists, please use the interface at: > > > > > > http://lists.bx.psu.edu/ > > > > > > > > >

1 0

manage_db.sh error
by Zachary Charlop-Powers 07 Feb '12

07 Feb '12

Hi Galaxy Team, I am running a small local instance of Galaxy and I've run into a database error following an upgrade to the Jan27th galaxy-dist release. The last line when I run run.sh tells me to upgrade my database: Exception: Your database has version '85' but this code expects version '90'. Please backup your database and then migrate the schema by running 'sh manage_db.sh upgrade'. But when I run the manage_db.sh I run into the following errors: $ sh manage_db.sh Traceback (most recent call last): File "./scripts/manage_db.py", line 12, in <module> from migrate.versioning.shell import main File "/home/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/shell.py", line 7, in <module> from migrate.versioning.base import * File "/home/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/base/__init__.py", line 5, in <module> from const import databases,operations File "/home/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.5.4-py2.7.egg/migrate/versioning/base/const.py", line 7, in <module> from sqlalchemy.util import OrderedDict ImportError: No module named sqlalchemy.util The first thing that pops out is that I the sqlalchemy egg is part of py2.7 while I am using python2.6. Any suggestions on how to troubleshoot this? thanks, zach cp

2 1

Adding a new python tool
by Sarah Maman 07 Feb '12

07 Feb '12

Hello, I managed to add a new tool coded in bash but I am having some difficulty adding a tool coded in python. Here are the contents of the test files: * Content of test.xml file in galaxy/galaxy-dist/tools/test/test.xml : <tool id="test" name="Test name"> <description>Test description</description> <command interpreter="python">test.py</command> </tool> * Content of test.py file in galaxy/galaxy-dist/tools/test/test.py : #!/usr/bin/python import os for param in os.environ.keys(): print "%20s %s" % (param,os.environ[param]) * Content of tool_conf_xml file in galaxy/galaxy-dist/ : <?xml version="1.0"?> <toolbox> <section name="Get Data" id="getext"> <tool file="test/test.xml" /> <tool file="uploadFile/uploadFile.xml" /> <tool file="data_source/upload.xml"/> <tool file="data_source/ucsc_tablebrowser.xml" /> <tool file="data_source/ucsc_tablebrowser_test.xml" />...... Could you, please, explain me why the new tool "test" isn't avalaible in Tools/Get Data windows (on the left, before uploadFile and upload tools ) ? Thanks in advance, Sarah

2 1